Hi,

We have the following code to call a shell script:

    Runtime rt = Runtime.getRuntime();
    Process proc = rt.exec(scriptName );
    int exitVal = proc.waitFor();
    log.info("Returned Value is: " + exitVal);

How can get access to the output from the shell script (normally from
stdout)?
-- 
(\__/)  M.
(='.'=) Due to the amount of spam posted via googlegroups and
(")_(") their inaction to the problem. I am blocking some articles
posted from there.  If you wish your postings to be seen by
everyone you will need use a different method of posting.

On 03/15/2011 02:46 PM, Mark wrote:
> Hi,
>
> We have the following code to call a shell script:
>
>      Runtime rt = Runtime.getRuntime();
>      Process proc = rt.exec(scriptName );
>      int exitVal = proc.waitFor();
>      log.info("Returned Value is: " + exitVal);
>
> How can get access to the output from the shell script (normally from
> stdout)?

You not only can but should use proc.getInputStream (stdout) and 
proc.getErrorStream (stderr) to prevent the very limited buffers from 
overrunning and blocking execution of the external process.

On Tue, 15 Mar 2011 15:42:14 +0100, Silvio <silvio@moc.com> wrote:

>On 03/15/2011 02:46 PM, Mark wrote:
>> Hi,
>>
>> We have the following code to call a shell script:
>>
>>      Runtime rt = Runtime.getRuntime();
>>      Process proc = rt.exec(scriptName );
>>      int exitVal = proc.waitFor();
>>      log.info("Returned Value is: " + exitVal);
>>
>> How can get access to the output from the shell script (normally from
>> stdout)?
>
>You not only can but should use proc.getInputStream (stdout) and 
>proc.getErrorStream (stderr) to prevent the very limited buffers from 
>overrunning and blocking execution of the external process.

Thanks.  I'm not sure how to fit this in with the waitFor() method
since this blocks until the script is finished.
-- 
(\__/)  M.
(='.'=) Due to the amount of spam posted via googlegroups and
(")_(") their inaction to the problem. I am blocking some articles
posted from there.  If you wish your postings to be seen by
everyone you will need use a different method of posting.

Mark wrote:
> On Tue, 15 Mar 2011 15:42:14 +0100, Silvio<silvio@moc.com>  wrote:
>
>> On 03/15/2011 02:46 PM, Mark wrote:
>>> Hi,
>>>
>>> We have the following code to call a shell script:
>>>
>>>       Runtime rt = Runtime.getRuntime();
>>>       Process proc = rt.exec(scriptName );
>>>       int exitVal = proc.waitFor();
>>>       log.info("Returned Value is: " + exitVal);
>>>
>>> How can get access to the output from the shell script (normally from
>>> stdout)?
>>
>> You not only can but should use proc.getInputStream (stdout) and
>> proc.getErrorStream (stderr) to prevent the very limited buffers from
>> overrunning and blocking execution of the external process.
>
> Thanks.  I'm not sure how to fit this in with the waitFor() method
> since this blocks until the script is finished.

Multi threaded software.

   BugBear

On 15/03/11 13:46, Mark wrote:
> Hi,
> 
> We have the following code to call a shell script:
> 
>     Runtime rt = Runtime.getRuntime();
>     Process proc = rt.exec(scriptName );
>     int exitVal = proc.waitFor();
>     log.info("Returned Value is: " + exitVal);
> 
> How can get access to the output from the shell script (normally from
> stdout)?

You may be better off these days using ProcessBuilder, and combining
stdout and stderr using ProcessBuilder.redirectErrorStream(). Then all
you need do is use the Process.getInputStream() to read both stdout and
stderr.

If you stick to using Runtime.exec() you'll need to use threads to read
both the stdout and stderr simultaneously to avoid potential deadlock
due to the buffered nature of the streams.

-- 
Nigel Wade

On 3/15/2011 10:23 AM, Nigel Wade wrote:
> On 15/03/11 13:46, Mark wrote:
>> Hi,
>>
>> We have the following code to call a shell script:
>>
>>      Runtime rt = Runtime.getRuntime();
>>      Process proc = rt.exec(scriptName );
>>      int exitVal = proc.waitFor();
>>      log.info("Returned Value is: " + exitVal);
>>
>> How can get access to the output from the shell script (normally from
>> stdout)?
>
> You may be better off these days using ProcessBuilder, and combining
> stdout and stderr using ProcessBuilder.redirectErrorStream(). Then all
> you need do is use the Process.getInputStream() to read both stdout and
> stderr.

Whether to combine them or not depends on what they contain, and how you
will be processing the output. Often, processing a mix of two types of
messages, one including error messages that can be difficult to predict
and test, is more difficult than processing the two types of messages
separately.

I often have two Runnable stream processors, each in its own thread,
doing different things. Typically, the one attached to standard out is
parsing it to extract a result. The one attached to standard error does
logging.

Patricia

In message <8u9lknFr4qU1@mid.individual.net>, Nigel Wade wrote:

> If you stick to using Runtime.exec() you'll need to use threads to read
> both the stdout and stderr simultaneously to avoid potential deadlock
> due to the buffered nature of the streams.

Don’t you have select(2) <http://docs.python.org/py3k/library/select.html>?

On Tue, 15 Mar 2011 13:46:03 +0000, Mark
<i@dontgetlotsofspamanymore.invalid> wrote, quoted or indirectly
quoted someone who said :

>How can get access to the output from the shell script (normally from
>stdout)?
>-- 

see http://mindprod.com/jgloss/exec.html
-- 
Roedy Green Canadian Mind Products
http://mindprod.com
Refactor early. If you procrastinate, you will have
even more code to adjust based on the faulty design.
..

Lawrence D'Oliveiro <ldo@geek-central.gen.new_zealand> wrote:
> In message <8u9lknFr4qU1@mid.individual.net>, Nigel Wade wrote:
>> If you stick to using Runtime.exec() you'll need to use threads to read
>> both the stdout and stderr simultaneously to avoid potential deadlock
>> due to the buffered nature of the streams.
> Don’t you have select(2) <http://docs.python.org/py3k/library/select.html>?

Iirc, with nio Java offers also the non-blocking approach to I/O.
Haven't yet used it myself, yet, as I do such tasks in Tcl ;-)

Based on how fragile thread-programming can be (forget one "synchronized"
and you end up reading a local CPU's cache instead of shared memory. Put
bad ones in and face bad performance or deadlocks), I understand this
wish for select-based IO - yet it *is* already available in Java.

Read up on java.nio.channels.Selector for a start.

PS: to be fair: event-based IO also has its pitfalls and hardships,
  just different ones.  ;)

On Tue, 15 Mar 2011 20:46:16 -0700, Roedy Green
<see_website@mindprod.com.invalid> wrote:

>On Tue, 15 Mar 2011 13:46:03 +0000, Mark
><i@dontgetlotsofspamanymore.invalid> wrote, quoted or indirectly
>quoted someone who said :
>
>>How can get access to the output from the shell script (normally from
>>stdout)?
>>-- 
>
>see http://mindprod.com/jgloss/exec.html

Thanks.  I'll pass this on.
-- 
(\__/)  M.
(='.'=) Due to the amount of spam posted via googlegroups and
(")_(") their inaction to the problem. I am blocking some articles
posted from there.  If you wish your postings to be seen by
everyone you will need use a different method of posting.

In message <slrnio0prt.nej.avl@gamma.logic.tuwien.ac.at>, Andreas Leitgeb 
wrote:

> Read up on java.nio.channels.Selector for a start.

<http://developer.android.com/reference/java/nio/channels/Selector.html>

Bloody hell. How many different layers of classes and calls do you need in 
Java to implement such a simple concept?

The underlying POSIX service is so simple 
<http://linux.die.net/man/2/select>: pass 3 bitmasks of file descriptors to 
wait for, plus an optional timeout. Python keeps this simplicity 
<http://docs.python.org/py3k/library/select.html>: pass 3 sequences of 
objects which can be anything you like, so long as they have valid “fileno” 
methods.

Lawrence D'Oliveiro <ldo@geek-central.gen.new_zealand> wrote:
> In message <slrnio0prt.nej.avl@gamma.logic.tuwien.ac.at>, Andreas Leitgeb wrote:
>> Read up on java.nio.channels.Selector for a start.
> <http://developer.android.com/reference/java/nio/channels/Selector.html>
> Bloody hell. How many different layers of classes and calls do you need in 
> Java to implement such a simple concept?

"simple". Yes, that describes it. Too simple in some cases, where lots of
filedescriptors are totally used by the app. (select scales badly with the
value of the latest filedescriptor, even if only a small number of fds is
really selected on.) There's also "poll" which is not that "simple", but
doesn't care if the fds you wait for are 3,4,5 or 1000,4042,4043.

I have no idea, which of select or poll is used with java's nio Selector,
but I believe that it is not user's concern, anyway.  Just because the user
doesn't get to build up bitsets, himself, its probably a good abstraction
that would allow even for a dynamic (or at least system-dependent) choice
between select and poll.

Tcl also uses select, internally, but the script-interface hides this away
so it could probably be rewritten for poll any time.  Also, it allows for
easier porting to systems, where such notifications for channels are handled
entirely different.

In message <slrnio3ja4.nej.avl@gamma.logic.tuwien.ac.at>, Andreas Leitgeb 
wrote:

> Lawrence D'Oliveiro <ldo@geek-central.gen.new_zealand> wrote:
>
>> <http://developer.android.com/reference/java/nio/channels/Selector.html>
>> Bloody hell. How many different layers of classes and calls do you need
>> in Java to implement such a simple concept?
> 
> "simple". Yes, that describes it. Too simple in some cases, where lots of
> filedescriptors are totally used by the app. (select scales badly with the
> value of the latest filedescriptor, even if only a small number of fds is
> really selected on.)

Really? I’ve written servers in Perl, C++ and Python using select, and its 
performance was never a problem; any bottleneck turned out to be elsewhere.

> I have no idea, which of select or poll is used with java's nio Selector,
> but I believe that it is not user's concern, anyway.  Just because the
> user doesn't get to build up bitsets, himself, its probably a good
> abstraction that would allow even for a dynamic (or at least
> system-dependent) choice between select and poll.

Python manages to avoid the bitsets in a much simpler way.

> Tcl also uses select, internally ...

Tcl is an antiquated language, where arrays are not even first-class 
objects.

Lawrence D'Oliveiro <ldo@geek-central.gen.new_zealand> wrote:
>> [Andreas Leitgeb wrote about limitations of select]
> Really? I’ve written servers in Perl, C++ and Python using select, and its 
> performance was never a problem; any bottleneck turned out to be elsewhere.
What were the access-rates on those servers? How many fds did it ever have
open at the same time? (including fds that were not incoming connections)

> Python manages to avoid the bitsets in a much simpler way.
That sounds good. Maybe it is closer to tcl, than I thought it was
after reading your mention of explicit "select" in python?

>> Tcl also uses select, internally ...
> Tcl is an antiquated language, where arrays are not even first-class 
> objects.
Apart from that it is a bogus statement, it also doesn't address
Tcl's approach to event-based programming, namely hiding that ugly 
select system-call beyond recognition, which was the only reason
I brought it up here.

Lawrence D'Oliveiro wrote:
> In message <slrnio3ja4.nej.avl@gamma.logic.tuwien.ac.at>, Andreas Leitgeb 
> wrote:
> 
> Really? I’ve written servers in Perl, C++ and Python using select, and its 
> performance was never a problem; any bottleneck turned out to be elsewhere.

The scalability problems with select are well-documented. See for
example Stevens, _UNIX Network Programming_ 2nd ed vol 1, 6.3; that
also refers you to Wright & Stevens, _TCP/IP Illustrated_ vol 2,
16.13, which shows an implementation of select that may be enlightening.

UNP 6.3 also notes that in some implementations attempting to enlarge
the select FD_SETSIZE does not work correctly because the sets were
still restricted in the kernel; the Standard Unix Specification (v3)
does not require that implementations permit changing FD_SETISZE.

UNP 6.10 notes that poll() does not suffer from the latter problem.

Another discussion may be found at [1], which notes that neither
select() nor poll() scale very well to thousands of descriptors;
that's why OS vendors started introducing other mechanisms, such as
/dev/poll, in the mid-1990s.

Another example is the "Heuristic 2" discussion in the Winsock FAQ's
discussion of I/O strategies.[2] Of course, select in Winsock uses an
array of handles rather than a bitmask, so it's closer to Unix's
poll() than to the traditional select() in its implementation.

Or see the discussion at [3], particularly where Barry Margolin points
out that there are different scaling tradeoffs between select() and
poll(). If the descriptor set is large and dense, select() is more
efficient in switching between user and kernel mode; if either of
those doesn't hold, poll() is more efficient at that point.

The scalability issues with select() ought to be obvious. There's the
limit of FD_SETSIZE, in implementations where that limit is in the
kernel as well. (If it isn't, you can work around that at the
application level.) There's the cost of copying FD_SETs between user
and kernel space. There's the cost of linear scans of the FD_SETs. The
latter two are particularly wasteful if the sets are sparse; that's
the problem that poll() fixes, but at the cost of using far more
storage per descriptor, so whether you benefit from poll() depends on
just which descriptors you're using.

On Windows, there's just select(), and it's effectively poll(), and
(as the Winsock FAQ explains) it's pretty much the worst of the
available choices.

For a handful of clients, even up to several hundred, it likely
doesn't matter; chances are good that I/O multiplexing won't dominate
processing time, even though it adds a lot of looping within the main
loop. When a server has to handle thousands of simultaneous clients
it's a different story. I have (proprietary) profiling data from real,
commercial servers showing significant performance issues with
select() at those kinds of loads.


[1] http://www.kegel.com/c10k.html
[2] http://tangentsoft.net/wskfaq/articles/io-strategies.html
[3]
https://groups.google.com/group/comp.protocols.tcp-ip/browse_thread/thread/24409068a4c51c62/e3cdcda1e6c216f1

-- 
Michael Wojcik
Micro Focus
Rhetoric & Writing, Michigan State University

In message <im0497017a8@news3.newsguy.com>, Michael Wojcik wrote:

> UNP 6.10 notes that poll() does not suffer from the latter problem.

Fine. I’ve never used poll, but Python supports that too if you want, with 
just one class and four methods 
<http://docs.python.org/py3k/library/select.html#poll-objects>.

Contrast that to the number of different classes that Java requires 
<http://developer.android.com/reference/java/nio/channels/Selector.html>: 
Selector (9 methods), SelectableChannel (8 methods), SelectionKey (another 
dozen methods), SelectorProvider ... I’m losing count.