f



help me solve a problem in converting a non-parallel program into parallel program...

Hi all,

I need your help.

I have a Matlab program, which uses the Matlab symbolic toolbox and does
exclusively symbolic computations. It calls Maple's library internally. My
program is currently non-parallel. I need to run it on a super computer. The
super computer does not have Matlab and it is Linux based and supports MPI.

I didn't figure out how to convert non-parallel program into parallel
exectuables on Linux with MPI.

Since I failed to get a nice solution of the above problem, now I am willing
to make a comprise, I am going to make it coarse-grained distrubuted
computing:

(1) It is easy to compile Matlab non-parallel program into non-parallel
exectuables;
(2) I am willing to launch 100 independent copies fo these exectuables on
100 CPUs;

Is there a mechanism in Linux, or MPI, or third party applications that can
facilitate luanching of such 100 programs and finally automatically collect
back the 100 results from these 100 runs?

I am thinking this is too hard to do well by me myself manually. I need some
kind of automation to manage this whole thing.

Please shed some lights on me! Thanks a lot!




0
housing2006 (283)
1/7/2007 5:33:27 AM
comp.os.linux.misc 33599 articles. 1 followers. amosa69 (78) is leader. Post Follow

15 Replies
1086 Views

Similar Articles

[PageSpeed] 32

Mike wrote:
> I have a Matlab program, which uses the Matlab symbolic toolbox and does
> exclusively symbolic computations. It calls Maple's library internally. My
> program is currently non-parallel. I need to run it on a super computer. The
> super computer does not have Matlab and it is Linux based and supports MPI.
> 
> I didn't figure out how to convert non-parallel program into parallel
> exectuables on Linux with MPI.
> 
> Since I failed to get a nice solution of the above problem, now I am willing
> to make a comprise, I am going to make it coarse-grained distrubuted
> computing:
> 
> (1) It is easy to compile Matlab non-parallel program into non-parallel
> exectuables;
> (2) I am willing to launch 100 independent copies fo these exectuables on
> 100 CPUs;
> 
> Is there a mechanism in Linux, or MPI, or third party applications that can
> facilitate luanching of such 100 programs

Yes, it's called a loop, and the shell can do it.  As in:

	for i in 1 2 3 4 5 6 7 8 9
	do
	   ./myprogram $i &
	done

Or:

	while read line
	do
	    ./myprogram $line &
	done < my_file_with_parameters_for_each_desired_instance

> and finally automatically collect
> back the 100 results from these 100 runs?

Sure, collecting the results from 100 programs is easy.  If they all
write everything to standard output, you'd just do something like
this:

	while read outputfile param1 param2
	do
	    ./myprogram $param1 $param2 > $outputfile &
	done < my_file_with_output_filenames_and_parameters

	# now wait for them *all* to finish
	wait

	echo "all of the programs are done now!"

Of course, you can put all that in a file and have the shell run it
as a script so you don't have to type it all interactively (and so
you can do it again if you make an error or something).

   - Logan
0
Logan
1/7/2007 6:19:08 AM
"Logan Shaw" <lshaw-usenet@austin.rr.com> wrote in message 
news:45a090dc$0$5175$4c368faf@roadrunner.com...
> Mike wrote:
>> I have a Matlab program, which uses the Matlab symbolic toolbox and does
>> exclusively symbolic computations. It calls Maple's library internally. 
>> My
>> program is currently non-parallel. I need to run it on a super computer. 
>> The
>> super computer does not have Matlab and it is Linux based and supports 
>> MPI.
>>
>> I didn't figure out how to convert non-parallel program into parallel
>> exectuables on Linux with MPI.
>>
>> Since I failed to get a nice solution of the above problem, now I am 
>> willing
>> to make a comprise, I am going to make it coarse-grained distrubuted
>> computing:
>>
>> (1) It is easy to compile Matlab non-parallel program into non-parallel
>> exectuables;
>> (2) I am willing to launch 100 independent copies fo these exectuables on
>> 100 CPUs;
>>
>> Is there a mechanism in Linux, or MPI, or third party applications that 
>> can
>> facilitate luanching of such 100 programs
>
> Yes, it's called a loop, and the shell can do it.  As in:
>
> for i in 1 2 3 4 5 6 7 8 9
> do
>    ./myprogram $i &
> done
>
> Or:
>
> while read line
> do
>     ./myprogram $line &
> done < my_file_with_parameters_for_each_desired_instance
>
>> and finally automatically collect
>> back the 100 results from these 100 runs?
>
> Sure, collecting the results from 100 programs is easy.  If they all
> write everything to standard output, you'd just do something like
> this:
>
> while read outputfile param1 param2
> do
>     ./myprogram $param1 $param2 > $outputfile &
> done < my_file_with_output_filenames_and_parameters
>
> # now wait for them *all* to finish
> wait
>
> echo "all of the programs are done now!"
>
> Of course, you can put all that in a file and have the shell run it
> as a script so you don't have to type it all interactively (and so
> you can do it again if you make an error or something).
>
>   - Logan

Logan, thanks for your help!

Do you mean I just need to launch the 100 programs using a loop and then the 
supercomputer will automatically dispatch the jobs to 100 CPUs?

Your shell script looks serial to me... (as you see, I've got no previous 
experience with parallel computing...)

Thanks! 


0
Mike
1/7/2007 8:16:51 AM
"Mike" <housing2006@gmail.com> wrote in message 
news:enq0n6$af8$1@news.Stanford.EDU...
> Hi all,
>
> I need your help.
>
> I have a Matlab program, which uses the Matlab symbolic toolbox and does
> exclusively symbolic computations. It calls Maple's library internally. My
> program is currently non-parallel. I need to run it on a super computer. 
> The
> super computer does not have Matlab and it is Linux based and supports 
> MPI.
>
> I didn't figure out how to convert non-parallel program into parallel
> exectuables on Linux with MPI.
>
> Since I failed to get a nice solution of the above problem, now I am 
> willing
> to make a comprise, I am going to make it coarse-grained distrubuted
> computing:
>
> (1) It is easy to compile Matlab non-parallel program into non-parallel
> exectuables;
> (2) I am willing to launch 100 independent copies fo these exectuables on
> 100 CPUs;
>
> Is there a mechanism in Linux, or MPI, or third party applications that 
> can
> facilitate luanching of such 100 programs and finally automatically 
> collect
> back the 100 results from these 100 runs?
>
> I am thinking this is too hard to do well by me myself manually. I need 
> some
> kind of automation to manage this whole thing.
>
> Please shed some lights on me! Thanks a lot!
>
Check my werbsite for the MPI tutorial link to learn about MPI itself.
Basically there are only 6 functions you need to know.

http://www.personal.leeds.ac.uk/~bgy1mm


Under MPI, each program is identical, but they send messages to each other, 
so the data and the execution statye can be different on each node. 
Generally, however, all nodes will communicate at the same point.

So your algorithm is

lauch program with all problems stored on node 0
divide problem into 100 chunks, send data to all nodes.
calculate chunk on each node
send solutions back to node 0
print out result.

step 2 may be easy or difficult, depending on your problem.

The problem is that the code is in Matlab whilst MPI provides C and Fortran. 
The easiest solution, conceptually, is to translate the code from Matlab to 
C. I am sure there will be some tool to enable you to this without rewriting 
everything manually, however it is likely to be difficult to use. 


0
Malcolm
1/7/2007 10:36:59 AM
"Mike" <housing2006@gmail.com> writes:

> "Logan Shaw" <lshaw-usenet@austin.rr.com> wrote in message 
> news:45a090dc$0$5175$4c368faf@roadrunner.com...
>> Mike wrote:
>>> I have a Matlab program, which uses the Matlab symbolic toolbox and does
>>> exclusively symbolic computations. It calls Maple's library internally. 
>>> My
>>> program is currently non-parallel. I need to run it on a super computer. 
>>> The
>>> super computer does not have Matlab and it is Linux based and supports 
>>> MPI.
>>>
>>> I didn't figure out how to convert non-parallel program into parallel
>>> exectuables on Linux with MPI.
>>>
>>> Since I failed to get a nice solution of the above problem, now I am 
>>> willing
>>> to make a comprise, I am going to make it coarse-grained distrubuted
>>> computing:
>>>
>>> (1) It is easy to compile Matlab non-parallel program into non-parallel
>>> exectuables;
>>> (2) I am willing to launch 100 independent copies fo these exectuables on
>>> 100 CPUs;
>>>
>>> Is there a mechanism in Linux, or MPI, or third party applications that 
>>> can
>>> facilitate luanching of such 100 programs
>>
>> Yes, it's called a loop, and the shell can do it.  As in:
>>
>> for i in 1 2 3 4 5 6 7 8 9
>> do
>>    ./myprogram $i &
>> done
>>
>> Or:
>>
>> while read line
>> do
>>     ./myprogram $line &
>> done < my_file_with_parameters_for_each_desired_instance
>>
>>> and finally automatically collect
>>> back the 100 results from these 100 runs?
>>
>> Sure, collecting the results from 100 programs is easy.  If they all
>> write everything to standard output, you'd just do something like
>> this:
>>
>> while read outputfile param1 param2
>> do
>>     ./myprogram $param1 $param2 > $outputfile &
>> done < my_file_with_output_filenames_and_parameters
>>
>> # now wait for them *all* to finish
>> wait
>>
>> echo "all of the programs are done now!"
>>
>> Of course, you can put all that in a file and have the shell run it
>> as a script so you don't have to type it all interactively (and so
>> you can do it again if you make an error or something).
>>
>>   - Logan
>
> Logan, thanks for your help!
>
> Do you mean I just need to launch the 100 programs using a loop and then the 
> supercomputer will automatically dispatch the jobs to 100 CPUs?
>
> Your shell script looks serial to me... (as you see, I've got no previous 
> experience with parallel computing...)

the '&' operator starts a process in the background and the shell
continues without waiting it to return.

look at these examples:

phix@igloo:~> { sleep 10 ; echo I am done } &
[1] 10674
phix@igloo:~> echo test
test
phix@igloo:~> I am done

[1]  + done       {; sleep 10; echo I am done; }
phix@igloo:~>             
phix@igloo:~> { { sleep 5; echo last } & echo first } > out ; wait ; cat out
[2] 10710
[2]  + done       {; sleep 5; echo later; }
first
last

-- 
Philippe Amarenco, aka Phix
epita 2007 - EpX - ex-{GISTR,ACU,LSE}
0
Philippe
1/7/2007 10:41:30 AM
"Mike =D0=B4=B5=C0=A3=BA
"
> Hi all,
>
> I need your help.
>
> I have a Matlab program, which uses the Matlab symbolic toolbox and does
> exclusively symbolic computations. It calls Maple's library internally. My
> program is currently non-parallel. I need to run it on a super computer. =
The
> super computer does not have Matlab and it is Linux based and supports MP=
I=2E
>
> I didn't figure out how to convert non-parallel program into parallel
> exectuables on Linux with MPI.
>
> Since I failed to get a nice solution of the above problem, now I am will=
ing
> to make a comprise, I am going to make it coarse-grained distrubuted
> computing:
>
> (1) It is easy to compile Matlab non-parallel program into non-parallel
> exectuables;
> (2) I am willing to launch 100 independent copies fo these exectuables on
> 100 CPUs;
>
> Is there a mechanism in Linux, or MPI, or third party applications that c=
an
> facilitate luanching of such 100 programs and finally automatically colle=
ct
> back the 100 results from these 100 runs?
>
> I am thinking this is too hard to do well by me myself manually. I need s=
ome
> kind of automation to manage this whole thing.
>
> Please shed some lights on me! Thanks a lot!


Hi,try this way:
oh,I'm not an English native speaker,so please ignore the spelling
errors if they occur :)
1,you must find a proper way to split your problem  into small pieces
(it has nothing to do with MPI. This is a key step and it's just a math
problem)
2, rewrite your matlab programm, make it can work with some parameters
to solve a part of your problem...(the parameters it received decide
which part it should to compute) I mean, after this step ,you should be
able to solve the problem by repeated calling the programme,each time
with a different parameter.
3=2E Compile your matlab programme into .EXE file. There are several ways
to do that. Maybe LCC  can work.
4=2EWrite a wrapper tool for your programme.The wrapper should use MPI to
interact with each other,and make it  call your .EXE programme with
parameters it received from the center control node.
5=2E Dispatch your wrappers and .EXE programmes

P=2ES  MPI itself can  dispatch and collect back results perfectly.No
third party application is needed.Google MPI_Broadcast  and MPI_Wait
for more.

0
PiggestPig
1/7/2007 1:04:14 PM
PiggestPig wrote:
> "Mike ���
> "
>> Hi all,
>>
>> I need your help.
>>
>> I have a Matlab program, which uses the Matlab symbolic toolbox and does
>> exclusively symbolic computations. It calls Maple's library internally. My
>> program is currently non-parallel. I need to run it on a super computer. The
>> super computer does not have Matlab and it is Linux based and supports MPI.
>>
>> I didn't figure out how to convert non-parallel program into parallel
>> exectuables on Linux with MPI.
>>
>> Since I failed to get a nice solution of the above problem, now I am willing
>> to make a comprise, I am going to make it coarse-grained distrubuted
>> computing:
>>
>> (1) It is easy to compile Matlab non-parallel program into non-parallel
>> exectuables;
>> (2) I am willing to launch 100 independent copies fo these exectuables on
>> 100 CPUs;
>>
>> Is there a mechanism in Linux, or MPI, or third party applications that can
>> facilitate luanching of such 100 programs and finally automatically collect
>> back the 100 results from these 100 runs?
>>
>> I am thinking this is too hard to do well by me myself manually. I need some
>> kind of automation to manage this whole thing.
>>
>> Please shed some lights on me! Thanks a lot!
> 
> 
> Hi,try this way:
> oh,I'm not an English native speaker,so please ignore the spelling
> errors if they occur :)
> 1,you must find a proper way to split your problem  into small pieces
> (it has nothing to do with MPI. This is a key step and it's just a math
> problem)
> 2, rewrite your matlab programm, make it can work with some parameters
> to solve a part of your problem...(the parameters it received decide
> which part it should to compute) I mean, after this step ,you should be
> able to solve the problem by repeated calling the programme,each time
> with a different parameter.
> 3. Compile your matlab programme into .EXE file. There are several ways
> to do that. Maybe LCC  can work.
> 4.Write a wrapper tool for your programme.The wrapper should use MPI to
> interact with each other,and make it  call your .EXE programme with
> parameters it received from the center control node.
> 5. Dispatch your wrappers and .EXE programmes
> 
> P.S  MPI itself can  dispatch and collect back results perfectly.No
> third party application is needed.Google MPI_Broadcast  and MPI_Wait
> for more.
> 
Of course, .EXE programs do not run in Linux.

-- 
  .~.  Jean-David Beyer          Registered Linux User 85642.
  /V\  PGP-Key: 9A2FC99A         Registered Machine   241939.
 /( )\ Shrewsbury, New Jersey    http://counter.li.org
 ^^-^^ 08:45:01 up 78 days, 11:18, 3 users, load average: 4.36, 4.36, 4.32
0
Jean
1/7/2007 1:46:16 PM
On Sat, 6 Jan 2007 21:33:27 -0800, "Mike" <housing2006@gmail.com> wrote:

>I have a Matlab program, which uses the Matlab symbolic toolbox and does
>exclusively symbolic computations. It calls Maple's library internally. My
>program is currently non-parallel. I need to run it on a super computer. The
>super computer does not have Matlab and it is Linux based and supports MPI.
>
>I didn't figure out how to convert non-parallel program into parallel
>exectuables on Linux with MPI.

>Is there a mechanism in Linux, or MPI, or third party applications that can
>facilitate luanching of such 100 programs and finally automatically collect
>back the 100 results from these 100 runs?
>
>I am thinking this is too hard to do well by me myself manually. I need some
>kind of automation to manage this whole thing.
>
>Please shed some lights on me! Thanks a lot!

You are probably looking for IPC, InterProcessCommunication.

Check out Chapter 5 in 
http://www.advancedlinuxprogramming.com/alp-folder

Personally I would do it with Perl, and Perl's IPC modules,
like IPC::Open3
Perl makes it easy to setup socket connections, the IPC, etc.
and combine them into a single program.

One of the things to consider is whether you can collect results
all at the end-of-run in a results-file, or whether you need to collect
the results in real-time as they are returned. Possibly you need an even
more complex system, where results are returned in realtime to the 
collecting system, decisions are made, and new commands are sent out
to the clients, so they can alter their computations.

In other words, it may be a bit more complex than shell scripts can
cleanly handle, and you might want to look into Perl, Python, etc.

I would ask this question at http://perlmonks.org for some advice
from a wide variety of systems programmers, it's free, and probably
would set you on the right track.




-- 
I'm not really a human, but I play one on earth.
http://zentara.net/japh.html
0
zentara
1/7/2007 3:01:26 PM
Mike wrote:
> "Logan Shaw" <lshaw-usenet@austin.rr.com> wrote in message 
> news:45a090dc$0$5175$4c368faf@roadrunner.com...
>> Mike wrote:

>>> Is there a mechanism in Linux, or MPI, or third party applications that 
>>> can
>>> facilitate luanching of such 100 programs

>> Yes, it's called a loop, and the shell can do it.  As in:
>>
>> for i in 1 2 3 4 5 6 7 8 9
>> do
>>    ./myprogram $i &
>> done

> Do you mean I just need to launch the 100 programs using a loop and then the 
> supercomputer will automatically dispatch the jobs to 100 CPUs?

Good point.  Obviously, it will depend on the specifics of the computer.

But in most cases, it probably will be good enough.  There is no guarantee
that the job will get a fixed assignment to one of the 100 CPUs, but it
very well may not matter if a job moves between one CPU and another.
The scheduler makes this decision for you.

Some (most?) schedulers in multiprocessor systems have a "processor
affinity" feature where they note which processor a job most recently
ran on and prefer to schedule it on that processor again in the future.
That does not create a fixed binding, but it does reduce the (already
fairly small) overhead of switching CPUs a bit.

The real answer to this question depends on what type of multiprocessor
architecture that system has.

Is it:

(1) an SMP machine (SMP stands for symmetric multiprocessor and basically
     means a machine where all processors are identical and all share the
     same physical memory; with SMP, there is very little penalty for
     switching around between processors)?
(2) a NUMA (NUMA is like SMP except that some processors have local memory
     they can access faster than others; this is harder to program for
     since it's very important which processor is running what, but it
     can get higher performance)?
(3) actually not truly a single multiprocessor machine but instead a
     cluster of completely separate machines (each with their own
     process, memory, and instance of the operating system) connected
     together by a network, i.e. a Beowulf cluster?
(4) something else?

The right thing to do will depend on the type of machine.  In fact,
if it is a cluster of machines, then launching all the jobs locally
with a simple loop is probably precisely the *wrong* thing to do.
I looked up MPI, and it seems there are MPI implementations that can
support different architectures, so the fact that you mentioned MPI
doesn't really seem to imply what kind of architecture you have.

So, I suggest you should talk to the support staff for the computer
you will be using.  They probably have a web page that you can use
to read up about how to do all this, and they are surely used to
questions from people in exactly your position, i.e. people who've
never run anything on a 100-processor system before, which would
be the majority of the people in the world.  :-)

   - Logan
0
Logan
1/7/2007 5:13:21 PM
On Sun, 07 Jan 2007 06:33:27 +0100, Mike <housing2006@gmail.com> wrote:

> Hi all,
>
> I need your help.
>
> I have a Matlab program, which uses the Matlab symbolic toolbox and does
> exclusively symbolic computations.

Did you consider to use "Matlab Distributed Computing Toolbox 3.0"?

-- 
This signature intentionally left almost blank.
0
Georg
1/7/2007 8:56:58 PM
"Georg Bisseling" <spam@bisseling.de> wrote in message 
news:op.tlsxs8h4z4pjzz@donner.fini...
> On Sun, 07 Jan 2007 06:33:27 +0100, Mike <housing2006@gmail.com> wrote:
>
>> Hi all,
>>
>> I need your help.
>>
>> I have a Matlab program, which uses the Matlab symbolic toolbox and does
>> exclusively symbolic computations.
>
> Did you consider to use "Matlab Distributed Computing Toolbox 3.0"?
>

Did you see I said the super-computer does not have Matlab installed? 


0
Mike
1/7/2007 9:03:44 PM
"Malcolm McLean" <regniztar@btinternet.com> wrote in message 
news:v-CdnQd0-KhxsTzYnZ2dnUVZ8tqqnZ2d@bt.com...
>
> "Mike" <housing2006@gmail.com> wrote in message 
> news:enq0n6$af8$1@news.Stanford.EDU...
>> Hi all,
>>
>> I need your help.
>>
>> I have a Matlab program, which uses the Matlab symbolic toolbox and does
>> exclusively symbolic computations. It calls Maple's library internally. 
>> My
>> program is currently non-parallel. I need to run it on a super computer. 
>> The
>> super computer does not have Matlab and it is Linux based and supports 
>> MPI.
>>
>> I didn't figure out how to convert non-parallel program into parallel
>> exectuables on Linux with MPI.
>>
>> Since I failed to get a nice solution of the above problem, now I am 
>> willing
>> to make a comprise, I am going to make it coarse-grained distrubuted
>> computing:
>>
>> (1) It is easy to compile Matlab non-parallel program into non-parallel
>> exectuables;
>> (2) I am willing to launch 100 independent copies fo these exectuables on
>> 100 CPUs;
>>
>> Is there a mechanism in Linux, or MPI, or third party applications that 
>> can
>> facilitate luanching of such 100 programs and finally automatically 
>> collect
>> back the 100 results from these 100 runs?
>>
>> I am thinking this is too hard to do well by me myself manually. I need 
>> some
>> kind of automation to manage this whole thing.
>>
>> Please shed some lights on me! Thanks a lot!
>>
> Check my werbsite for the MPI tutorial link to learn about MPI itself.
> Basically there are only 6 functions you need to know.
>
> http://www.personal.leeds.ac.uk/~bgy1mm
>


I like your suggestion - learning 6 functions is good. But in your tutorial 
you said MPI doesnot work with C++?

>
> Under MPI, each program is identical, but they send messages to each 
> other, so the data and the execution statye can be different on each node. 
> Generally, however, all nodes will communicate at the same point.
>
> So your algorithm is
>
> lauch program with all problems stored on node 0
> divide problem into 100 chunks, send data to all nodes.
> calculate chunk on each node
> send solutions back to node 0
> print out result.
>
> step 2 may be easy or difficult, depending on your problem.
>

I don't have to divide the program. Matlab allows me to compile a 
non-parallel Matlab program into non-parallel c program and executables.

So I need to dispatch 100 copy of such executables and hopefully they will 
run on 100 cpus simultaneously. 


0
Mike
1/7/2007 9:08:51 PM
On Sun, 07 Jan 2007 22:03:44 +0100, Mike <housing2006@gmail.com> wrote:

>
> "Georg Bisseling" <spam@bisseling.de> wrote in message
> news:op.tlsxs8h4z4pjzz@donner.fini...
>> On Sun, 07 Jan 2007 06:33:27 +0100, Mike <housing2006@gmail.com> wrote:
>>
>>> Hi all,
>>>
>>> I need your help.
>>>
>>> I have a Matlab program, which uses the Matlab symbolic toolbox and does
>>> exclusively symbolic computations.
>>
>> Did you consider to use "Matlab Distributed Computing Toolbox 3.0"?
>>
>
> Did you see I said the super-computer does not have Matlab installed?

That could be changed, couldn't it? Maybe you can bring your own Matlab
in libraries?

I interpret your "answer" as "No, I did not yet consider using it.".

If you see no way to use the parallelization features that come with
the programming environment of your choice then it can get very hard.

So let me ask me another question:

At the level of control that you currently have in your program, could you
split the problem in about 100 nearly equaly time consuming parts?
If the answer is yes then you could really let run 100 instances of your
program via sh/rsh/ssh (whatever suits the given super computer).

If the answer is no, then the question becomes: can you convert the program
to get more control? Answering this question is very tough without knowing
the details. If you do not provide the details, nobody can help.
Did you try to consult some newsgroups that are centered around Matlab like
comp.soft-sys.matlab or cern.matlab?

-- 
This signature intentionally left almost blank.
0
Georg
1/7/2007 9:48:18 PM
"Mike" <housing2006@gmail.com> wrote
>> lauch program with all problems stored on node 0
>> divide problem into 100 chunks, send data to all nodes.
>> calculate chunk on each node
>> send solutions back to node 0
>> print out result.
>>
>> step 2 may be easy or difficult, depending on your problem.
>>
>
> I don't have to divide the program. Matlab allows me to compile a 
> non-parallel Matlab program into non-parallel c program and executables.
>
> So I need to dispatch 100 copy of such executables and hopefully they will 
> run on 100 cpus simultaneously.
You've got to divide the problem, not the program.
If matlab will spit out a C program that will take as a parameter 1/100 th 
of your problem and return 1/100th of the answer, then you've got it made.
Simply compile the C under MPI, then in your main function write a little 
bit of code to send 1/100th of the problem to each node, collect the 
answers, and probaly you'll need a bit of code to collate them.

Eg you want to calcuate an average of 10000 numbers in parallel.

matlab spits out

double sum(double *x, int N)
{
  int i;
  doubel answer = 0;
  for(i=0;i<N;i++)
    answer += x[i];
  return answer;
}

pur MPI program goes something like

#include "mpi.h"

int main(int argc, char **argv)
{
  double array[10000];
  int rank;
  MPI_Init(argc, argv);
  MPI_Rank(MPI_COMM_WORLD, &rank);
  readinarray(array);
  if(rank == 0)
 {
    for(i=1;i<100;i++)
       /* haven't got the parameters quite right */
      MPI_Send(array + i * 100, MPI_DOUBLE, 100, i, MPI_COMM_WORLD):
 }
 else
 {
    MPI_Recv(array, MPI_DOUBLE,100, 0, MPI_COMM_WORLD);
 }
 /* calcuate a partial answer */
 answer = sum(array, 100);
 if(rank != 0)
  MPI_Send(&answer, MPI_DUBLE, 1, 0, MPI_COMM_WORLD):
 else
 {
   double total = answer;
   double partial;
    for(i=1;i<100;i++)
    {
       MPI_Recv(&partial, MPI_DOUBLE, 1, i, MPI_COMM_WORLD);
       total += partial;
    }
    printf("average is %g\n", total/10000);
  }

  MPI_Finalize();
}

This is only untested code, but it gives you the general idea. An average is 
extremely easy to break up. Other functions might be a lot harder. 


0
Malcolm
1/7/2007 9:59:38 PM
Mike wrote:

> I don't have to divide the program. Matlab allows me to compile a
> non-parallel Matlab program into non-parallel c program and
> executables.
>

Note that (a) Matlab is available for linux and (b) GNU Octave
is available for linux, if your code can run under octave (okay probably
not true in your case given maple usage, but hey), it can
help skirt the licensing issues that can crop up running
closed-source stuff like matlab on large clusters.
 
> So I need to dispatch 100 copy of such executables and hopefully they
> will run on 100 cpus simultaneously.

There are a variety of systems that both real supercomputers and
ordinary linux clusters run that manage such things (as previously
hinted by other posters). 

*** You'll just have to *ask the local systems guy* what they're running 
(or read the local documentation), if anything.  What you want to do
sounds like a typical use case for a mid-range linux batch cluster to
munch through, though - Don't be too proud to just ask the guys running
the system you have access to, rather than random folk on usenet, they
probably know the system in question far better than anyone out here!

[Following list is non-exhaustive]

(i) batch systems:

Many, perhaps most (at least in academic/research environments),
supercomputers and linux clusters run batch job processing software. 
Such software includes "Torque", "PBS", "Sun Grid Engine".  In this
model, one writes a job script, submits it, it's put in a queue, and
scheduled for execution (by a batch scheduler like "maui").  Batch
processing  allows fair sharing of a limited computational resource
while providing guaranteed run times of jobs and exclusive locks on
nodes.  

It feels a bit 1970s I guess, but unless users tolerate highly variable
run times due to users' processes slowing eachother down (and most
_don't_ at the high end), batch processing remains the choice. 

In your case, you might submit 100 jobs.  The batch+scheduler system,
usually implemented as a collection of daemon processes, looks after
spawning your processes on available nodes (or reserving nodes for your
use later if need be according to some priority scheme), tracking
stdio, staging in input data and staging back out outputs, and so on,
according to your specifications laid out in the job submission
script(s) for your jobs. (you can typically pass parameters or at least
env vars into the script, thus reusing the same basic script for 1000s
of similar but nonidentical runs)

[Note 1] Alternatively you might structure your run as a single 100-proc
parallel job, perhaps with the help of an MPI implementation or a ready
made intra-batch-job task farming script.

Here are the instructions for a pretty typical batch facility:
http://www.ichec.ie/user_documentation
(I'm not affiliated with them, I run a smaller 256-proc system elsewhere
in Ireland).


(ii) Process migrating linux clusters:

In these clusters, which typically involve a patched kernel, you just
spawn everything on the head node, and the system farms out (with
varying degrees of efficiency and post-migration performance issues -
check out "kerrighed" for a particularly interesting one) to worker
nodes.  These *may* _also_ be batch systems, but typically you see them
where a cluster is used by a single department or research group, where
squabbling over "hogging" of resources is less likely. The naive "kick
stuff off in a loop in shell script with &" may well work on such
clusters.

(To people in the field "beowulf" suggests a particular kind of process
migrating linux cluster, though laymen tend to use the word
in a much looser sense, thanks in no small part to trolls on /. ...)


(iii) bunch o' machines linux clusters:

In these clusters, there's no persistent management software or tight
integration, and users can just kick off stuff on whatever node for
however long whenever they like with ssh - But systems like
mpich2's "mpd" persistent daemons are often used to manage parallel
jobs more efficiently. This works fine so long as your users all like
and care about eachother and pay attention to eachother's needs. 
As if.



0
David
1/7/2007 10:11:32 PM
"Malcolm McLean" <regniztar@btinternet.com> wrote in message 
news:OOmdnR_K0LLS8DzYnZ2dnUVZ8s6gnZ2d@bt.com...
>
>
> Eg you want to calcuate an average of 10000 numbers in parallel.
>
> matlab spits out
>
> double sum(double *x, int N)
> {
>  int i;
>  doubel answer = 0;
>  for(i=0;i<N;i++)
>    answer += x[i];
>  return answer;
> }
>
> pur MPI program goes something like
>
> #include "mpi.h"
>
> int main(int argc, char **argv)
> {
>  double array[10000];
>  int rank;
>  MPI_Init(argc, argv);
>  MPI_Rank(MPI_COMM_WORLD, &rank);
>  readinarray(array);
>  if(rank == 0)
> {
>    for(i=1;i<100;i++)
>       /* haven't got the parameters quite right */
>      MPI_Send(array + i * 100, MPI_DOUBLE, 100, i, MPI_COMM_WORLD):
> }
> else
> {
>    MPI_Recv(array, MPI_DOUBLE,100, 0, MPI_COMM_WORLD);
> }
> /* calcuate a partial answer */
> answer = sum(array, 100);
> if(rank != 0)
>  MPI_Send(&answer, MPI_DUBLE, 1, 0, MPI_COMM_WORLD):
> else
> {
>   double total = answer;
>   double partial;
>    for(i=1;i<100;i++)
>    {
>       MPI_Recv(&partial, MPI_DOUBLE, 1, i, MPI_COMM_WORLD);
>       total += partial;
>    }
>    printf("average is %g\n", total/10000);
>  }
>
>  MPI_Finalize();
> }
>
> This is only untested code, but it gives you the general idea. An average 
> is extremely easy to break up. Other functions might be a lot harder.
>

Shame on me. I tried your tutorial examples, but there is no mpicc, or 
mpirun on the debugging computer that I am using. (I don't have access to 
the super computer yet and I would like to debug everything locally 
first...)

What's the best mpi implementation? I am not a Linux admin, will I be able 
to install an MPI and test debugging my program? 


0
Mike
1/8/2007 7:11:39 AM
Reply: