COMPGROUPS.NET | Search | Post Question | Groups | Stream | About | Register

### Flushing and multiple pipes

• Email
• Follow

Dear group,

happy to post my first message here! So, now the business.

I define two pipes which print both two parsed datastreams on the
_same_ postscript file. The data values are selected through 'commands'
by certain criteria in the following manner:

# shortened pseudocode
#!/usr/bin/perl

my $pipe1 = " commands >>$psfile";
my $pipe2 = " commands >>$psfile";

open(H1 ,"| $pipe1") or die "\n --- Error: Could not plot$pipe1: $!\n"; open(H2 ,"|$pipe2") or die "\n --- Error: Could not plot $pipe2:$!\n";

foreach my $ifile (@ifiles){ open(IFILE, "<$ifile");
while(<IFILE>){
$line = split(/\s+/,$_);
print H1 $line; print H2$line;
}
close(IFILE);
}
close(H1);
close(H2);
# end shortened pseudocode

I noticed that, depending on the datastream type, sometimes the
postscript is written and closed correctly, sometimes it does not. I
noticed also that on Ubuntu the script behaves differently than on
Leopard OSX. No fancy modules are loaded, just pure Perl. The
'commands' are a pipe of awk and gmt routines.

To be on the safe side, i duplicated the foreach loop, and now i open
one pipe at a time, guessing that the problem is how the OS flushes the
pipes' buffers and how the postscript file gets the values.

It ain't by any means the best solution, because i have to parse the
same files twice. Then the question: is there a way to open concurrent
pipes in a robust way?

Thanks.


 0
Reply lelli.luca (8) 7/20/2012 12:44:03 PM

See related articles to this posting

neurino <lelli.luca@googlemail.com> writes:

[...]

> #!/usr/bin/perl
>
> my $pipe1 = " commands >>$psfile";
> my $pipe2 = " commands >>$psfile";
>
> open(H1 ,"| $pipe1") or die "\n --- Error: Could not plot$pipe1: $!\n"; > open(H2 ,"|$pipe2") or die "\n --- Error: Could not plot $pipe2:$!\n";
>
> foreach my $ifile (@ifiles){ > open(IFILE, "<$ifile");
> 	while(<IFILE>){
> 			$line = split(/\s+/,$_);
> 			print H1 $line; > print H2$line;
> 	}
> 	close(IFILE);
> }
> close(H1);
> close(H2);
> # end shortened pseudocode
>
> I noticed that, depending on the datastream type, sometimes the
> postscript is written and closed correctly, sometimes it does not.

I assume 'correctly' means you get two differently processed lines for
each input line in the output file, in the order they were written in
perl. That's never going to work reliably in this way because not only
perl employs internal output buffering but the commands running as
part of your pipeline do this as well: If they use stdio, their output
will be 'fully buffered' when stdout is not connected to an
interactive device. Also, the processes in both of your pipelines
execute asynchronously with respect to the Perl control process and
the processes in the other pipeline. This means you may get higher
throughput in this way but the downside is that output reordering may
(and usually will) occur.

The simple but relatively inefficient solution to that is to create
two new pipelines for each input line and don't start the second
before the first has terminated (or the third before the second has
terminated and so on). Unless you're repeatedlyv dealing with large
inputs, this is probably good enough, though. If you wan't to process
the input asynchronously and concurrenly, you need to employ a final
'put it back together' filter which reads data from both pipelines
as it becomes available and puts the output back into the proper
order.

 0
Reply rweikusat (2830) 7/20/2012 1:08:14 PM

On 2012-07-20 15:08:14 +0200, Rainer Weikusat said:

> I assume 'correctly' means you get two differently processed lines for
> each input line in the output file, in the order they were written in
> perl.

Exactly.

> That's never going to work reliably in this way because [...]

Thanks for the explanation. So, it seems i was on the right track somehow.

> The simple but relatively inefficient solution to that is to create
> two new pipelines for each input line and don't start the second
> before the first has terminated (or the third before the second has
> terminated and so on).

This is the way i implemented it now. I could read the files at once
but isn't an option as well. And you are pointing out the issue: the
data records are large enough, 1-2 Gb.

> If you wan't to process the input asynchronously and concurrenly, you
> need to employ a final 'put it back together' filter which reads data
> from both pipelines as it becomes available and puts the output back
> into the proper
> order.

It sounds quite new to me. I found on perfaq5 pack/unpack. Is is right?

Thanks.


 0
Reply lelli.luca (8) 7/20/2012 1:45:57 PM

2 Replies
53 Views

Similar Articles

12/12/2013 6:15:01 AM
[PageSpeed]

Similar Artilces:

Piping Multiple Responses into Commands
Hi All, We are writing a script to automate the updating of our Oracle databases to apple patches. It seems that some of the Oracle programs, when run, ask questions that reqire a response of Y|N. So, it is easy to say: echo Y | opatch apply But what if the command ('opatch apply' in this case) asks multiple questions? Is there any way to pipe multiple responses into the command? Thanks. "amerar@iwc.net" <amerar@iwc.net> writes: > Hi All, > > We are writing a script to automate the updating of our Oracle > databases to apple patches. It seems tha...

multiple pipes and tail -f #2
Hi, I have been trying to do the following but it doesn't seem to work. tail -f <filename> | grep <some string> | grep <some other string> where <filename> is the name of any file to which new lines are added, something like a log file for example. Is there some restriction on the number of pipes which could be used while using tail -f ? I would like use sed instead of the last grep in the example above, but for simplicity I have used second grep. Thanks and Regards, Prashant On Wed, 22 Aug 2007 08:53:07 -0700, pvsnmp@yahoo.com <pvsnmp@yahoo.com> wrote:...

Multiple Tables vs Multiple Files
OK, so I am experimenting with FMP 7 and converting some of my old solutions which involve multiple file relationships. I just say "convert", FMP 7 does it, and the converted solution works fine. But I'm feeling kind if inadequate because I haven't used any of the new super-duper features of FMP 7. What is the advantage of converting these multiple files to multiple tables in one file? It would be a lot of work redefining all the relationships and layouts and I don't see how it would work any different when I was done. -- Nelson It would probably be worthwh...

perl PIPE
My question is about pips. I have the following code: EXAMPLE 1: #----------------------------------------------------------- #!/usr/bin/perl -w use strict; my $command = "ls -al"; my @failureMsg; open(OUTPUT, "$command |") or die "can't do the command\n"; my $lines = <OUTPUT>; @failureMsg = grep(/\ssp.*\.pl$/, <OUTPUT>); print "@failureMsg"; #----------------------------------------------------------- In this code I'm piping my the output of my command to a file handle(FH). I understand this and am happy with it. What I...

Multiple rendering contexts/multiple windows
Hi I have two windows (hWnd) and I would like to have a different rendering context(and different DC) associated with each one. I set the pixel format and create a rendering context for the first window and it works fine...No errors. I set the pixel format for the second window but when I attempt to wglCreateContext for the second window I get GetLastError = 8 (ERROR_NOT_ENOUGH_MEMORY). I have looked everywhere for a solution without any luck. My second window is used to render the image to a bitmap. Only some lines on the images appear when output to the bitmap. Nothing fails but I do get er...

how to get multiple=multiple values from an optgroup ?
hey there all, i have an html page that uses a select element with optgroups. I want multiple="multiple" but when i pass it to the php script, i do not know how to get the values out. like this <select name="people" multiple="multiple" > when this is submitted, are they passed as an array? thanks "nephish" <nephish@gmail.com> wrote in message news:1160764737.443297.251860@h48g2000cwc.googlegroups.com... > hey there all, > > i have an html page that uses a select element with optgroups. I want > multiple="multiple&quo...

Rip Routes Flushed or not flushed .. that is the question
Just wondering about something with regards to RIP. If I physically unplug a cable from my router, will the router flush the routes right away, or will it wait for the timer to expire ??? Thanks. Greetings, > If I physically unplug a cable from my router, will the router flush > the routes right away, or will it wait for the timer to expire ??? As soon as an interface goes DOWN, any IP address associated with that interface is removed from ALL Router tables. Cheers................pk. -- Peter from Auckland. ...

multiple emails in bcc vs multiple emails
if i wanted to send an email to, say, 1000 people, which would be faster - sending a single email with a really large bcc field or sending out one email for each person? one advantage of sending an email to each person, personally, is that it'd be easier to detect bounced emails and delete them from the database via variable return envelope paths. also, what other considerations, aside from speed, are there? "yawnmoth" <terra1024@yahoo.com> wrote in message news:1108758609.480742.199490@o13g2000cwo.googlegroups.com > if i wanted to send an email to, say, 1000 people...

Multiple ISPs and Multiple IP Ranges from Each ISP
I have Multiple ISP Internet links and Multiple Valid IP ranges from Each ISPs.. I have configured to use both ISP lines using Route Maps. For Example let my First Range be 20.10.10.0/24 from ISP-A let my Second Range be 30.10.10.0/24 from ISP-B. Traffic from First range will use ISP-A's Internet Link and others will use ISP-B's link. Now, I want to Enable Redundancy between these two links...If one fails let it use the other. My Question here is, Will My ISPs will Route the packet from the other ISP. Say Suppose, My Internet link to ISP-A is down. Now, all my Internet traffic shou...

Multiple OpenGL Cards and Multiple Rendering Contexts
I have a Windows XP machine (Intel Platform) with two PCI Express slots in it. I would like to have two OpenGL Cards (say Card A and Card B) in the system. The idea is to create two concurrent OpenGL rendering contexts (Rendering Context A attached to Card A and Rendering Context B attached to Card B) so that two scenes can be simultaneously rendered with hardware accleration each fueled by an OpenGL card. It is also important to mention that I am NOT planning to use NVidia's SLI technology. I have two specific questions in this regard: (1) Is there any specific technique in the creation...

Applying multiple functions to multiple sets of arguements
Hi all. I was hoping someone cold help me. I wanted a way to apply an arbitrary number of functions to an arbitrary number of sets of arguments. Meaning if I have n functions: {f1, f2, ...,fn} and m sets of arguments: {x1, x2, ..., xm}, how can I generate a list like: {{f1[x1], f1[x2], ...,f1[xm]},{f2[x1], f2[x2], ....,f2[xm]},...,{fn[x1], fn[x2], ...,fn[xm]}}? I am currently doing this by looping throgh both lists and populating each results one by one. As expecting this is horribly slow. I was hoping someone could recommend a faster, functional way to do this. I tried usin...

Map entity to multiple tables in multiple databases
Hi, I'm using Hibernate for O/R-mapping. I want to create an entity who's data comes from multiple tables. I already figured out that I have to use @SecondaryTable to achieve that. My question is: Is there a way to map an entity to different tables in different databases? Or do the @Table and the @SecondaryTables have to be in the same database? TIA simon The only way I see to solve this problem is descripbed here: http://java.sun.com/mailers/techtips/enterprise/2006/TechTips_July06.html#2 ...

Re: Multiple observations into multiple variables #3
On Mon, Oct 6, 2008 at 1:51 PM, Joseph Cormier <jcormier@brconline.com>wrote: > I have a dataset that looks like this: Each person has multiple injuries... > > Year Person Injury Type > 2005 1 1 A > 2005 1 2 A > 2005 1 3 B > 2005 1 4 B > 2005 2 1 D > 2005 2 2 D > 2005 2 3 E > 2005 2 4 F > 2005 2 5 A > 2004 1 1 B > 2004 1 2 C > 2004 1 3 D > > I would like to create multiple variables for a single person so that the > different injuries are aligned in a single observation like this: > > Year Person Injury Injury 1 Injury 2 Inju...

Outputting multiple SAS files to multiple worksheets
Hi, *********I have an output with an excel file with multiple worksheets, but i want to name the worksheets as ALPHA, BETA and GAMMA instead of DATA1, DATA2 and DATA3. Can somebody help me with this/ or is this possible to do? Thanks, Hari Following is sample code ... >>>>>>>>>>>>>>>>>>>>>>>>>>>>program<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< LIBNAME X1 odbc noprompt="DRIVER=Microsoft Excel Driver (*.xls);...

Multiple Routes + Multiple Interfaces to the same Router/Gateway.
How does Solaris handle a situation like follows: default ->router1 IP-A -> router1->Interface-A IP-B -> router1 -> interface-B IP-C -> router1 ->interface-C all IPs are Class-C and fall into same subnet. where router1 is the same in all of the 4 routes. Basically a Server with 4 cables going into the same Appliance. Suppose Interface-A( or default) goes down ( both a Link failure & an Unreachable target through that interface for any reasons). Does the system continue to try other interfaces (routes) ?? Does it require a manual intervention ? p.s. IPMP is not a...

What is in a pipe?
Hi all, I'm trying to work out how many messages are queued in a pipe and how long said messages are. Anyone got any ideas? So far, the best I can come up with is something silly like receiving DBMS_PIPE messages, unpacking them, counting and adding lengths and then repacking and requeuing the messages. What I would like is something like this: select * from DBA_ALL_ABOUT_PIPES; And DBA_ALL_ABOUT_PIPES has the name, type of pipe (public or private), size, number of queued messages and length of messages. Anyone have any ideas? Cheers Geoff Geoff, How about checking V\$db_pipe...

multiplication
I want to multiply two very long integers (say of 20 digits).I am storing them in character array.Then what mechanism should i follow to mutiply two such numbers.If anybody has any suggestions please post. On 30 Apr 2007 04:29:26 -0700, helPlease <devlinahello2@gmail.com> wrote: >I want to multiply two very long integers (say of 20 digits).I am >storing them in character array.Then what mechanism should i follow >to mutiply two such numbers.If anybody has any suggestions please post. Take a pencil and paper and multiply two four-digit numbers. Carefully show **every** step. ...

Aesthetics when spanning multiple columns and multiple rows in a table
I can make the cell of a table span multiple columns and multiple rows, but I end up with a yucky mess, like this: http://homepage.mac.com/andrew_j_stewart/table_yuk.jpg Here is the code that created that table: \documentclass[10pt,letterpaper]{article} \usepackage{multirow} \begin{document} \begin{tabular}{| c | c | c |} \hline 1 & 2 & 3 \\ \hline \multicolumn{2}{|c|}{\multirow{2}{*}{4,5,7,8}} & 6 \\ \hline & & 9 \\ \hline 10 & 11 & 12 \\ \hline \end{tabular} \end{document} I can trim the excess vertical lines by removing them from the tabular definitio...

Beginner Help: Joining Multiple classes in multiple files?
I've been doing quite alot of reading on C++ and classes, however, everything I read just talks about the code itself and not the location of the code. My question is, what if you want to seperate classes into their own CPP files, what changes are needed to the files and how do you actually compile and build an executable. For instance, let's say I have the following all in one file, it would compile and link into an executable fine: ------------START contents of main.cpp START--------------------- #include <iostream.h> class Animal { public: int GetAge() { return Age;} ...

TCLLib cmdline
I am using cmdline version 1.2 I expect my script to interpret file list just like ls. It does not. Here are outputs and below that is the example script. Notice last 2 are contrary to my expectations. Thanks for help, -Bhushit bhushit@jungle:/home/bhushit/support: ls *.tcl clifinal.tcl stty.tcl wrapper.tcl bhushit@jungle:/home/bhushit/support: tclsh clifinal.tcl -files clifinal.tcl Passed me -files clifinal.tcl x 0 files clifinal.tcl bhushit@jungle:/home/bhushit/support: tclsh clifinal.tcl -files "clifinal.tcl stty.tcl" Passed me -files {clifinal.tcl stty.tcl} x ...