I would like to let programs write log files to a pipe, while at the
same time a script reads from it filtering the data stream, and
outputs itself to a final result. Usually, one has to switch the log
file and apply the filter script to the rotated log file.

The problem with my idea is that if the script won't run for any
reason, the program that writes the log file will get stuck. So I came
to the idea of a "buffering" pipe, i.e., a pipe (or something) that
would accept a data stream into it even when it can't pass it further.

Is this a too crazy idea or what?

If not, your turn.

Rick Denoire

In article <7d13qv88rfpahum6555i1dljv1f7kaamia@4ax.com>,
Rick Denoire  <100.17706@germanynet.de> wrote:
>I would like to let programs write log files to a pipe, while at the
>same time a script reads from it filtering the data stream, and
>outputs itself to a final result. Usually, one has to switch the log
>file and apply the filter script to the rotated log file.
>
>The problem with my idea is that if the script won't run for any
>reason, the program that writes the log file will get stuck. So I came
>to the idea of a "buffering" pipe, i.e., a pipe (or something) that
>would accept a data stream into it even when it can't pass it further.
>
>Is this a too crazy idea or what?

Where would this buffer be kept?  The kernel intentionally provides minimal
buffering in pipes, to prevent processes from using too much kernel memory
(in many operating systems this memory isn't paged, so it locks up physical
memory).

There are several ways to accomplish what you want in user-mode code.  One
way is to use two pipes, with a process in the middle.  It reads from one
pipe and writes to the other; if the process that reads from the second
pipe doesn't run, this middle process can buffer the data that it's reading
from the first pipe.  If the buffer gets too large it could even be dumped
to a file, which it would read when the second pipe finally clears.

Another way is to use threads in the process that writes to the pipe.  The
main thread writes to a buffer, and a pipe-writing thread reads from the
buffer and writes to the pipe.  If the reading process stops, the
pipe-writing thread will block, but the main thread can continue writing to
the buffer.  It may need to realloc() it to make it larger, or could dump
it to a file as in the above method.

-- 
Barry Margolin, barry.margolin@level3.com
Level(3), Woburn, MA
*** DON'T SEND TECHNICAL QUESTIONS DIRECTLY TO ME, post them to newsgroups.
Please DON'T copy followups to me -- I'll assume it wasn't posted to the group.

Rick Denoire wrote:

> I would like to let programs write log files to a pipe, while at the
> same time a script reads from it filtering the data stream, and
> outputs itself to a final result. Usually, one has to switch the log
> file and apply the filter script to the rotated log file.
> 
> The problem with my idea is that if the script won't run for any
> reason, the program that writes the log file will get stuck. So I came
> to the idea of a "buffering" pipe, i.e., a pipe (or something) that
> would accept a data stream into it even when it can't pass it further.

Sooner or later the buffer will fill up.  Then you will have the
same problem you had before.  Adding a buffer just delays the onset
of the symptoms.

   - Logan