Tail -f with filter question - comp.unix.programmer

Hi folks, When testing I have a log file that I tail with: tail -f log.txt and watch the output. However there's some log entries that start with INFO that I don't want to see (the rest are error messages that I am interested in seeing). Is there way of filtering those out? I've tried this: tail -f log.txt | grep -v INFO but I think that's buffering and I don't get to see the error messages when they are produced (in real time). Obviously I code write a C program to do this, but wasn't sure if there was a better/easier way. Cheers, Matt

Matt Charles wrote: > Hi folks, > > When testing I have a log file that I tail with: > > tail -f log.txt > > and watch the output. > > However there's some log entries that start with INFO that I don't want > to see (the rest are error messages that I am interested in seeing). Is > there way of filtering those out? > > I've tried this: > > tail -f log.txt | grep -v INFO > > but I think that's buffering and I don't get to see the error messages > when they are produced (in real time). > > Obviously I code write a C program to do this, but wasn't sure if there > was a better/easier way. See if you have the "stdbuf" command installed.

On 2011-05-17, Matt Charles <mcharles48@gmail.com> wrote: > Hi folks, > > When testing I have a log file that I tail with: > > tail -f log.txt > > and watch the output. > > However there's some log entries that start with INFO that I don't > want to see (the rest are error messages that I am interested in > seeing). Is there way of filtering those out? > > I've tried this: > > tail -f log.txt | grep -v INFO > > but I think that's buffering and I don't get to see the error messages > when they are produced (in real time). > > Obviously I code write a C program to do this, but wasn't sure if > there was a better/easier way. > Do you use the syslog facility? If you do you could just change the rules in the syslog configuration file. It would also allow the users of your program to setup their own rules if they wish to do so. If you don't want to use syslog I found the following option for grep > --line-buffered Use line buffering on output. This can cause a > performance penalty. I guess grep uses block buffering by default, and AFAIU this should be sufficient for your error messages

Mc wrote: >> >> See if you have the "stdbuf" command installed. >> > Doesn't look like I do unfortunately. > > I should of mentioned that I'm running on a Solaris 10 Sparc host. Ah that makes it a bit more complicated. You can either install it (GNU coreutils), or maybe you can install expect, which comes with a sample script called "unbuffer" which should also do what you want.

Tobias Blass <tobiasblass@gmx.net> writes: >I guess grep uses block buffering by default, so line buffering should solve >your problem stderr is line buffered by default. stdout is line buffered if isatty() is true for the file descriptor associated with stdout, otherwise it is full buffered (512 to 4096+ byte buffer, depending on libc). Note that isatty() is false for a pipe. scott

* Matt Charles <mcharles48@gmail.com> in comp.unix.programmer: > tail -f log.txt | grep -v INFO > but I think that's buffering and I don't get to see the error messages > when they are produced (in real time). A page detailing and comparing the solutions already posted can be found here: http://mywiki.wooledge.org/BashFAQ/009 Best, -- DW

On Tue, 17 May 2011 16:36:05 +0200, Damien Wyart wrote: > * Matt Charles <mcharles48@gmail.com> in comp.unix.programmer: >> tail -f log.txt | grep -v INFO > >> but I think that's buffering and I don't get to see the error messages >> when they are produced (in real time). > > A page detailing and comparing the solutions already posted can be found > here: http://mywiki.wooledge.org/BashFAQ/009 In the first bullet of the 'less' section near the end of that page, perhaps you could list less -p pattern as an alternative to starting less and then entering /, pattern. -- jiw

> > A page detailing and comparing the solutions already posted can be > > found here: http://mywiki.wooledge.org/BashFAQ/009 * James Waldby <not@valid.invalid> in comp.unix.programmer: > In the first bullet of the 'less' section near the end of that page, > perhaps you could list less -p pattern as an alternative to starting > less and then entering /, pattern. Done, thanks for the suggestion. Btw, I am not affiliated in any way to the site I quoted, so as edition is not restricted to authenticated users (as I learnt while trying to add your suggestion), you could have done this modification by yourself. -- DW

On Tue, 17 May 2011 14:08:56 +0000, Scott Lurndal wrote: >>I guess grep uses block buffering by default, so line buffering should >>solve your problem > > > stderr is line buffered by default. On most Unices, stderr is initially *unbuffered*. > stdout is line buffered if isatty() is true for the file descriptor > associated with stdout, otherwise it is full buffered This is the situation in practice for most Unices. However, both the ISO C standard and POSIX have weaker requirements. The ISO C standard (§7.19.3p7) says: As initially opened, the standard error stream is not fully buffered; the standard input and standard output streams are fully buffered if and only if the stream can be determined not to refer to an interactive device. POSIX (§2.5) is practically identical: When opened, the standard error stream is not fully buffered; the standard input and standard output streams are fully buffered if and only if the stream can be determined not to refer to an interactive device. <http://pubs.opengroup.org/onlinepubs/9699919799/functions/V2_chap02.html#tag_15_05>

Tobias Blass wrote: > On 2011-05-17, Matt Charles <mcharles48@gmail.com> wrote: >> >> I've tried this: >> >> tail -f log.txt | grep -v INFO >> >> but I think that's buffering and I don't get to see the error messages >> when they are produced (in real time). > > I found the following option for grep >> --line-buffered Use line buffering on output. This can cause a >> performance penalty. > I guess grep uses block buffering by default, so line buffering should solve > your problem The buffering is being done in tail, not grep. The solutions given by others involve running tail with its output going to a pseudo-terminal, using a tool that will read that output from the master side of the pseudo-terminal and write it (without buffering) to the pipe. -- Geoff Clare <netnews@gclare.org.uk>