TXR --- Text Extractor.

A practical extract and report language has finally arived on the scene.

Written in C with a Lispy library, a lex and yacc parser, stack-scanning
garbage collector, and a from-scratch regex implementation based on
constructing and simulating an NFA.

Demo time!

Here are the contents of "procps.txr", which is written against /proc
on Linux.

@(next)!ls /proc | sort -n
@(collect)
@{process /[0-9]+/}
@  (next)/proc/@process/status
Name:	@name
State:	@state (@state_desc)
SleepAVG:	@sleep_avg%
Tgid:	@tgid
Pid:	@pd
PPid:	@ppid
@(bind pid pd)
@(skip)
Uid:	@uid	@/.*/
Gid:	@gid	@/.*/
@  (next)!ls /proc/@process/task
@  (collect)
@thr
@  (end)
@  (bind thread thr)
@  (some)
@    (next)/etc/passwd
@    (skip)
@user:@pw:@uid:@/.*/
@  (and)
@    (bind user uid)
@  (end)
@(end)
@(output)
USER       PID S NAME             THREADS
@  (repeat)
@{user 8} @{pd -5} @state @{name 16} @(rep)@thr, @(last)@thr@(end)
@  (end)
@(end)


Now, let's run it:

$ txr procps.txr
USER       PID S NAME             THREADS
root         1 S init             1
root         2 S ksoftirqd/0      2
root         3 S events/0         3

[ ... snip ... ]

root      1791 S lockd            1791
root      1821 S ypbind           1821, 1822, 1826
root      1839 S acpid            1839

[ ... snip ... ]


Let's bind a variable. How about constraining to process 1791?

$ txr -Dpid=1791 procps.txr
USER       PID S NAME             THREADS
root      1791 S lockd            1791


How about constraining to two processes?

$ txr -Dpid=1791,1 procps.txr
USER       PID S NAME             THREADS
root         1 S init             1
root      1791 S lockd            1791


Which process contains thread 1822?

$ txr -Dthread=1822 procps.txr
USER       PID S NAME             THREADS
root      1821 S ypbind           1821, 1822, 1826

:P


Without an @(output) section which sends something to stdout, txr outputs
bindings in a form that can be evaled by your shell, to become shell variables.

$ txr <(echo "@a @b @c") <(echo "1 2 3")
a="1"
b="2"
c="3"

In most cases, a txr query to pull out some information will be clearer and
easier to develop than some clumsy combination of grep, sed, perl, awk,
not to mention more robust.

On Sep 18, 4:25=A0pm, Kaz Kylheku <kkylh...@gmail.com> wrote:
> TXR --- Text Extractor.
>
> A practical extract and report language has finally arived on the scene.
>
> Written in C with a Lispy library, a lex and yacc parser, stack-scanning
> garbage collector, and a from-scratch regex implementation based on
> constructing and simulating an NFA.
>
> Demo time!
>
> Here are the contents of "procps.txr", which is written against /proc
> on Linux.
>

Where do you download this program?

On 2009-09-20, DMCC <dean.menezes@gmail.com> wrote:
> On Sep 18, 4:25�pm, Kaz Kylheku <kkylh...@gmail.com> wrote:
>> TXR --- Text Extractor.
>>
>> A practical extract and report language has finally arived on the scene.
>>
>> Written in C with a Lispy library, a lex and yacc parser, stack-scanning
>> garbage collector, and a from-scratch regex implementation based on
>> constructing and simulating an NFA.
>>
>> Demo time!
>>
>> Here are the contents of "procps.txr", which is written against /proc
>> on Linux.
>>
>
> Where do you download this program?

Nowhere, yet. I'm looking at project hosting options. I can mail you an
advance copy. :)

Kaz Kylheku <kkylheku@gmail.com> writes:

> On 2009-09-20, DMCC <dean.menezes@gmail.com> wrote:
>> On Sep 18, 4:25�pm, Kaz Kylheku <kkylh...@gmail.com> wrote:
>>> TXR --- Text Extractor.
>>>
>>> A practical extract and report language has finally arived on the scene.
>>>
>>> Written in C with a Lispy library, a lex and yacc parser, stack-scanning
>>> garbage collector, and a from-scratch regex implementation based on
>>> constructing and simulating an NFA.
>>>
>>> Demo time!
>>>
>>> Here are the contents of "procps.txr", which is written against /proc
>>> on Linux.
>>>
>>
>> Where do you download this program?
>
> Nowhere, yet. I'm looking at project hosting options. I can mail you an
> advance copy. :)

No, not you!  You didn't serriously commited such a monstruosity?

-- 
__Pascal Bourguignon__

On 2009-09-20, DMCC <dean.menezes@gmail.com> wrote:
> On Sep 18, 4:25?pm, Kaz Kylheku <kkylh...@gmail.com> wrote:
>> TXR --- Text Extractor.
>>
>> A practical extract and report language has finally arived on the scene.
>>
>> Written in C with a Lispy library, a lex and yacc parser, stack-scanning
>> garbage collector, and a from-scratch regex implementation based on
>> constructing and simulating an NFA.
>>
>> Demo time!
>>
>> Here are the contents of "procps.txr", which is written against /proc
>> on Linux.
>>
>
> Where do you download this program?

http://www.nongnu.org/txr/

On Sun, 2009-09-27 at 19:38 +0000, Kaz Kylheku wrote:
> On 2009-09-20, DMCC <dean.menezes@gmail.com> wrote:
> > On Sep 18, 4:25?pm, Kaz Kylheku <kkylh...@gmail.com> wrote:
> >> TXR --- Text Extractor.
> >>
> >> A practical extract and report language has finally arived on the scen=
e.
> >>
> >> Written in C with a Lispy library, a lex and yacc parser, stack-scanni=
ng
> >> garbage collector, and a from-scratch regex implementation based on
> >> constructing and simulating an NFA.
> >>
> >> Demo time!
> >>
> >> Here are the contents of "procps.txr", which is written against /proc
> >> on Linux.
> >>
> >
> > Where do you download this program?
>=20
> http://www.nongnu.org/txr/

Brilliant concept. Fantastic implementation. Indigestible
documentation :/

Please, for God's sake, move the explanation of the meaning of the code
NEARER to the code. My scroll bar is going to accuse me of improper
handling pretty soon.

On Sep 28, 6:01=A0am, Anti Vigilante <antivigila...@pyrabang.com> wrote:
> On Sun, 2009-09-27 at 19:38 +0000, Kaz Kylheku wrote:
> >http://www.nongnu.org/txr/
>
> Brilliant concept. Fantastic implementation. Indigestible
> documentation :/
The documentation is fine http://www.nongnu.org/txr/txr-manpage.html

> Please, for God's sake, move the explanation of the meaning of the code
> NEARER to the code. My scroll bar is going to accuse me of improper
> handling pretty soon.
Move the explanation before the results of the code?

On Mon, 2009-09-28 at 01:06 -0700, vippstar wrote:
> On Sep 28, 6:01 am, Anti Vigilante <antivigila...@pyrabang.com> wrote:
> > On Sun, 2009-09-27 at 19:38 +0000, Kaz Kylheku wrote:
> > >http://www.nongnu.org/txr/
> >
> > Brilliant concept. Fantastic implementation. Indigestible
> > documentation :/
> The documentation is fine http://www.nongnu.org/txr/txr-manpage.html
>=20
> > Please, for God's sake, move the explanation of the meaning of the code
> > NEARER to the code. My scroll bar is going to accuse me of improper
> > handling pretty soon.
> Move the explanation before the results of the code?

Well yes actually. Show us the code. Show what it means. And then let us
see the results knowing where they came from.

On 2009-09-28, Anti Vigilante <antivigilante@pyrabang.com> wrote:
> On Mon, 2009-09-28 at 01:06 -0700, vippstar wrote:
>> On Sep 28, 6:01 am, Anti Vigilante <antivigila...@pyrabang.com> wrote:
>> > On Sun, 2009-09-27 at 19:38 +0000, Kaz Kylheku wrote:
>> > >http://www.nongnu.org/txr/
>> >
>> > Brilliant concept. Fantastic implementation. Indigestible
>> > documentation :/
>> The documentation is fine http://www.nongnu.org/txr/txr-manpage.html
>> 
>> > Please, for God's sake, move the explanation of the meaning of the code
>> > NEARER to the code. My scroll bar is going to accuse me of improper
>> > handling pretty soon.
>> Move the explanation before the results of the code?
>
> Well yes actually. Show us the code. Show what it means. And then let us
> see the results knowing where they came from.

That's fine in a reference manual, but psychologically poor in an introductory
page. People are result-oriented.   ``You run this, and, look, this happens''.
Then explanations can follow.  Not everyone wants explanations, and not
everyone wants them right away, or upfront.

The purpose of the page is to captivate interest, not to teach.
That page serves the project, not the user.

People want the 30 second soundbite. The ``scoop'', if you will. Only those who
are drawn in by the scoop will dig through the details.  If you present things
wrongly, you will lose some of even these otherwise persistent people!

(I do realize that in computing there are some personalities who want to
recursively descent into all of the details before seeing something in its
broader context. That's a personality quirk that I don't care to cater to in
documenting.)

On Mon, 28 Sep 2009 17:24:55 +0000, Kaz Kylheku wrote:

> The purpose of the page is to captivate interest, not to teach. That
> page serves the project, not the user.
> 
> People want the 30 second soundbite. The ``scoop'', if you will. Only
> those who are drawn in by the scoop will dig through the details.  If
> you present things wrongly, you will lose some of even these otherwise
> persistent people!
> 
> (I do realize that in computing there are some personalities who want to
> recursively descent into all of the details before seeing something in
> its broader context. That's a personality quirk that I don't care to
> cater to in documenting.)

I agree with what you said.  I read the intro page and liked it.
Nevertheless, starting with a less complicated but still powerful
example (~5-10 loc) would be even better.  There are a lot of incidental
details in that code that just come from trying to reproduce what ps
does, so the efficiency of expressing the novel features of txr is not
that high.

txr is a nice idea, thanks for writing it.

Tamas

On Mon, 2009-09-28 at 17:24 +0000, Kaz Kylheku wrote:
> On 2009-09-28, Anti Vigilante <antivigilante@pyrabang.com> wrote:
> > On Mon, 2009-09-28 at 01:06 -0700, vippstar wrote:
> >> On Sep 28, 6:01 am, Anti Vigilante <antivigila...@pyrabang.com> wrote:
> >> > On Sun, 2009-09-27 at 19:38 +0000, Kaz Kylheku wrote:
> >> > >http://www.nongnu.org/txr/
> >> >
> >> > Brilliant concept. Fantastic implementation. Indigestible
> >> > documentation :/
> >> The documentation is fine http://www.nongnu.org/txr/txr-manpage.html
> >>=20
> >> > Please, for God's sake, move the explanation of the meaning of the c=
ode
> >> > NEARER to the code. My scroll bar is going to accuse me of improper
> >> > handling pretty soon.
> >> Move the explanation before the results of the code?
> >
> > Well yes actually. Show us the code. Show what it means. And then let u=
s
> > see the results knowing where they came from.
>=20
> That's fine in a reference manual, but psychologically poor in an introdu=
ctory
> page. People are result-oriented.   ``You run this, and, look, this happe=
ns''.
> Then explanations can follow.  Not everyone wants explanations, and not
> everyone wants them right away, or upfront.

Well see there's the problem. It isn't a psychological issue at all.
It's an ergonomics issue. You cannot expect people to flip back and
forth between the bottom of the page and the top.

> The purpose of the page is to captivate interest, not to teach.
> That page serves the project, not the user.

That's fine. But with a result dump sitting in the middle of it, it's
not doing a good job.

> People want the 30 second soundbite. The ``scoop'', if you will. Only tho=
se who
> are drawn in by the scoop will dig through the details.  If you present t=
hings
> wrongly, you will lose some of even these otherwise persistent people!

A result dump that contains many redundant rows and fills up the screen
does not a soundbite make. It just obscures contextually significant
material by shoving it off the page.

For example this is like when someone does not put documentation
comments in-line as is common in assembly or near the relevant
functions.

> (I do realize that in computing there are some personalities who want to
> recursively descent into all of the details before seeing something in it=
s
> broader context. That's a personality quirk that I don't care to cater to=
 in
> documenting.)

That's not the case. I wore out my scroll bar on that page.

Either split the documentation into frames so I can refer to the
explanation while I'm reading the results easily or reorder or even
shorten the example output.