f



Parallel processing in batches

What would be the cleanest way to execute a script in parallel, in
batches multiple time, keeping the same number of instances at all
times.

I have the following scenario:

- execute a script n times (first parameter), in parallel (second
parameter). So to interpret it in a better way, say Parm1 = 25, the
script needs to be executed 25times. The second parameter drives the
number of parallel instances that will be invoked (say Parm2 = 5).  At
any given time 5instances of the script should be invoked.  Total of 5
batches should run concurrently

I was thinking in the similar lines
(psuedo code)

parm1=25
partm2=5
for tot_exec_time from 1 to $parm1
   ctr=1
   while ctr <= $parm2
    do
        nohup exec_script 1>&  a.log.$parm1 &
        ctr++
        while (`ps -ef|grep exec_script` = $parm2)
        do
            sleep 2
        done
        ctr --
     done
done


Any pointers would help


TIA Rex
0
10/15/2009 12:23:29 PM
comp.unix.solaris 26025 articles. 2 followers. Post Follow

2 Replies
743 Views

Similar Articles

[PageSpeed] 10

Benz schrieb:
> What would be the cleanest way to execute a script in parallel, in
> batches multiple time, keeping the same number of instances at all
> times.
> 
> I have the following scenario:
> 
> - execute a script n times (first parameter), in parallel (second
> parameter). So to interpret it in a better way, say Parm1 = 25, the
> script needs to be executed 25times. The second parameter drives the
> number of parallel instances that will be invoked (say Parm2 = 5).  At
> any given time 5instances of the script should be invoked.  Total of 5
> batches should run concurrently
> 
> I was thinking in the similar lines
> (psuedo code)
> 
> parm1=25
> partm2=5
> for tot_exec_time from 1 to $parm1
>    ctr=1
>    while ctr <= $parm2
>     do
>         nohup exec_script 1>&  a.log.$parm1 &
>         ctr++
>         while (`ps -ef|grep exec_script` = $parm2)
>         do
>             sleep 2
>         done
>         ctr --
>      done
> done
> 
> 
> Any pointers would help
> 
> 
> TIA Rex

You might want to take a look at my xjobs utility. It makes
parallelizing multiple jobs easy, and runs on Solaris and other UNIXs.
Get it here:
http://www.maier-komor.de/xjobs.html

Cheers,
Thomas
0
Thomas
10/15/2009 12:52:38 PM
Thomas Maier-Komor  wrote:

> Benz schrieb:
>> What would be the cleanest way to execute a script in parallel, in
>> batches multiple time, keeping the same number of instances at all
>> times.
[...]
> You might want to take a look at my xjobs utility. It makes
> parallelizing multiple jobs easy, and runs on Solaris and other UNIXs.
> Get it here:
> http://www.maier-komor.de/xjobs.html

Otherwise you can convince some make versions (dmake or gmake for
instance) to do this kind of job control for you. The syntax would
probably not be so nice in this case though.
0
Marc
10/15/2009 3:14:24 PM
Reply: