f



broken pipe when reading and writing a pipe

hi

I am trying to do a simple task: I have a buffer, I want to write this
buffer as the stdin of a command, say "tac", and capture the output
from the stdout and save to another buffer.

after searching examples and playing a little bit, I wrote the a
function (see below). However, it gave me a "Broken pipe" error when I
ran the code.

Can someone point out to me what was wrong? your help is greatly
appreciated!

Qianqian


int pipemydata(void *inbuf, long inlen, void *outbuf, long *outlen){
        int fd[2];
        int childid;
        *outlen=0;
        if(pipe(fd)==-1){
                *outlen=-1;
                return *outlen;
        }
        if((childid=fork())){  /*write data into the pipe*/   /*parent
thread*/
                close(fd[0]);
                FILE *f=fdopen(fd[1],"wb");
                fwrite(inbuf,inlen,1,f);
                fclose(f);
        }else{  /*child thread to read output from tac*/
                dup2(fd[0],fileno(stdin));
                close(fd[1]);
                *outlen=0;
                FILE *f=popen("tac","rb");
                while(!feof(f)){
                        *outlen+=fread(outbuf+(*outlen),1,1024,f);
                }
                pclose(f);
        }
        return *outlen;
}
0
fangqq (29)
9/18/2009 8:49:30 PM
comp.unix.programmer 10848 articles. 0 followers. kokososo56 (350) is leader. Post Follow

6 Replies
1171 Views

Similar Articles

[PageSpeed] 35

In article <6cec3c7f-94f4-4f59-9844-30ae2c9ff055@v2g2000vbb.googlegroups.com>,
FangQ  <fangqq@gmail.com> wrote:
>hi
>
>I am trying to do a simple task: I have a buffer, I want to write this
>buffer as the stdin of a command, say "tac", and capture the output
>from the stdout and save to another buffer.
>
>after searching examples and playing a little bit, I wrote the a
>function (see below). However, it gave me a "Broken pipe" error when I
>ran the code.
>
>Can someone point out to me what was wrong? your help is greatly
>appreciated!

Well there are a few things wrong, but the one you need to fix first is than
popen() doesn't do "rb" mode. In unix there are no "text files" and "binary
files", there are just files. popen() is a unix function, not an ANSI C
function. I'd say the same for fdopen() but for some reason it does accept
the extra "b" (which has no effect).

You should add more error checking. If you had done this:

  FILE *f=popen("tac","rb");
  if(!f)
    perror("popen");

you would have got a "popen: Invalid argument" message to give you a clue.

Change that "rb" to "r" and you'll make a little bit of progress.

[...]
>        if((childid=fork())){  /*write data into the pipe*/   /*parent
>thread*/
>                close(fd[0]);
>                FILE *f=fdopen(fd[1],"wb");
>                fwrite(inbuf,inlen,1,f);
>                fclose(f);
>        }else{  /*child thread to read output from tac*/
>                dup2(fd[0],fileno(stdin));
>                close(fd[1]);
>                *outlen=0;
>                FILE *f=popen("tac","rb");
>                while(!feof(f)){
>                        *outlen+=fread(outbuf+(*outlen),1,1024,f);
>                }
>                pclose(f);
>        }
>        return *outlen;
>}

The next major thing is why are you using popen() at all? You already created
a pipe and a child process, you don't need yet another pipe and yet another
process (which is what popen gives you).

And then there's the problem that your child doesn't exit when it's done, so
you have both processes reaching the end of the function and returning to the
caller, which is probably not what you wanted.

-- 
Alan Curry
0
pacman5 (289)
9/19/2009 2:13:37 AM
Am Freitag 18 September 2009 22:49, FangQ a écrit :

> Can someone point out to me what was wrong?

Look here for a description of popen,
http://www.opengroup.org/onlinepubs/9699919799/functions/popen.html
and fork,
http://www.opengroup.org/onlinepubs/9699919799/functions/fork.html

Do not randomly mix  all the different types; but most of all,
select /appropriate/ types (see manual pages).
Close descriptors when you no longer need them.


#include <sys/types.h>
#include <unistd.h>
#include <stdio.h>
#include <err.h>

ssize_t
pipemydata(const void *ib, size_t il, char *ob)
{
  int d[2]; pid_t p; size_t ol = 0;

  if (pipe(d)) return (ssize_t)-1;
  if ((p = fork()) < 0) {
    close(d[1]); close(d[0]);
    return (ssize_t)-1;
  }
  if (p) { /* parent */
    FILE *f; int c;
    dup2(STDIN_FILENO, d[1]);
    dup2(d[0], STDIN_FILENO); close(d[0]);
    f = popen("tac", "r");
    dup2(d[1], STDIN_FILENO); close(d[1]);
    if (!f) {
      warn("popen tac");
      return (ssize_t)-1;
    }
    while ((c = getc(f)) != EOF) {
      *ob++ = c; ol++;
    }
    if (pclose(f)) {
      warn("pclose tac");
      return (ssize_t)-1;
    }
    return ol;
  }
  /* child */
  close(d[0]);
  write(d[1], ib, il);
  _exit(0);
}

const char ibuf[] = "\
I am trying to do a simple task: \n\
I have a buffer, I want to write \n\
this buffer as the stdin of a \n\
command, say “tac”, and capture \n\
the output from the stdout and \n\
save to another buffer.\n\
";
char obuf[9999];

int
main()
{
  ssize_t l = sizeof(ibuf) - 1;

  printf("*****\n");
  write(STDOUT_FILENO, ibuf, l);
  printf("*****\n");
  l = pipemydata(ibuf, l, obuf);
  if (l == (ssize_t)-1)
    return 1;
  write(STDOUT_FILENO, obuf, l);
  printf("*****\n");
  return 0;
}

-- 
printf -v email $(echo \ 155 141 162 143 145 154 142 162 165 151 \
156 163 155 141 100 171 141 150 157 157 056 143 157 155|tr \  \\\\)
#  Live every life as if it were your last!  #
0
mb148 (28)
9/19/2009 3:53:07 AM
In article <h91eoh$f1c$1@aioe.org>, I wrote:
>
>The next major thing is why are you using popen() at all? You already created
>a pipe and a child process, you don't need yet another pipe and yet another
>process (which is what popen gives you).

Correction: You do need another pipe (since you're piping in and out of the
"tac" process) but you don't really need the extra process.

-- 
Alan Curry
0
pacman5 (289)
9/19/2009 4:25:21 AM
In article <h91mfh$mm5$1@aioe.org>, pacman@kosh.dhis.org (Alan Curry) 
wrote:

> In article <h91eoh$f1c$1@aioe.org>, I wrote:
> >
> >The next major thing is why are you using popen() at all? You already created
> >a pipe and a child process, you don't need yet another pipe and yet another
> >process (which is what popen gives you).
> 
> Correction: You do need another pipe (since you're piping in and out of the
> "tac" process) but you don't really need the extra process.

You may need it to avoid deadlock.  If the writer tries to write 
everything to the send pipe before reading anything, the receive pipe's 
buffer may fill up, and then the tac process would block, and stop 
reading from the send pipe, so it will fill up and the writer will block.

I don't think this can happen with a command like "tac", since it can't 
start writing until it reads everything you send it.  But it can happen 
with other programs that produce output as they go (e.g. awk, grep, 
sed).  And even if you know the program produces one line of output for 
each line of input, you can't solve it by doing alternating writes and 
reads, because it's likely that the program uses stdio, which by default 
does full buffering when output is to a pipe.

Using separate processes (or threads) for writing and reading solves 
this problem.  You could also do it with select(), but that's a little 
more complicated to code.

-- 
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***
0
barmar (6125)
9/19/2009 5:22:44 AM
thank you very much for all of the responses. I found all your
comments very helpful and Marcel's example code worked beautifully for
me.

To answer the question about using popen and pipe together, I actually
don't know other alternatives to save the output into a buffer, except
running an external command via popen and read from it. Most of the
examples I saw invoke exec* to pipe the data to an external process,
but I don't have access (or know how to gain access) to the output.

For Marcel's example, I am trying to understand the 3 dup2.calls, can
some one commend on this?

dup2(STDIN_FILENO, d[1]);  /*make d[1] pipe be the stdin of the
parent ?*/
dup2(d[0], STDIN_FILENO); close(d[0]);
f =3D popen("tac", "r");
dup2(d[1], STDIN_FILENO); close(d[1]);

thank you again

Qianqian

On Sep 19, 1:22=A0am, Barry Margolin <bar...@alum.mit.edu> wrote:
> In article <h91mfh$mm...@aioe.org>, pac...@kosh.dhis.org (Alan Curry)
> wrote:
>
> > In article <h91eoh$f1...@aioe.org>, I wrote:
>
> > >The next major thing is why are you using popen() at all? You already =
created
> > >a pipe and a child process, you don't need yet another pipe and yet an=
other
> > >process (which is what popen gives you).
>
> > Correction: You do need another pipe (since you're piping in and out of=
 the
> > "tac" process) but you don't really need the extra process.
>
> You may need it to avoid deadlock. =A0If the writer tries to write
> everything to the send pipe before reading anything, the receive pipe's
> buffer may fill up, and then the tac process would block, and stop
> reading from the send pipe, so it will fill up and the writer will block.
>
> I don't think this can happen with a command like "tac", since it can't
> start writing until it reads everything you send it. =A0But it can happen
> with other programs that produce output as they go (e.g. awk, grep,
> sed). =A0And even if you know the program produces one line of output for
> each line of input, you can't solve it by doing alternating writes and
> reads, because it's likely that the program uses stdio, which by default
> does full buffering when output is to a pipe.
>
> Using separate processes (or threads) for writing and reading solves
> this problem. =A0You could also do it with select(), but that's a little
> more complicated to code.
>
> --
> Barry Margolin, bar...@alum.mit.edu
> Arlington, MA
> *** PLEASE post questions in newsgroups, not directly to me ***
> *** PLEASE don't copy me on replies, I'll read them in the group ***

0
fangqq (29)
9/19/2009 4:57:32 PM
Am Samstag 19 September 2009 18:57, FangQ a écrit :

> For Marcel's example, I am trying to understand the 3 dup2
> calls, can some one comment on this?
>
> dup2(STDIN_FILENO, d[1]);  /*make d[1] pipe be the stdin
> of the parent ?*/

Save the original stdin in d[1] (else it will be lost).

I should have added,
  fcntl(d[1], F_SETFD, fcntl(d[1], F_GETFD)|FD_CLOEXEC);
otherwise this copy of the original stdin will remain open
in the 'tac' process created by popen().

> dup2(d[0], STDIN_FILENO); close(d[0]);

Redirect stdin for the 'tac' command, which reads its input from
stdin; and after duplicating d[0] (the read end of the pipe) to
stdin, close d[0], because it is no longer needed in the parent.

> f = popen("tac", "r");

Open 'tac' with stdin coming from the read end of the pipe,
and stdout going to fileno(f).

> dup2(d[1], STDIN_FILENO); close(d[1]);

Restore the original stdin, which was saved in d[1], and then
release the d[1] descriptor.

-- 
printf -v email $(echo \ 155 141 162 143 145 154 142 162 165 151 \
156 163 155 141 100 171 141 150 157 157 056 143 157 155|tr \  \\\\)
#  Live every life as if it were your last!  #
0
mb148 (28)
9/19/2009 8:33:25 PM
Reply: