Hello,


Can anyone compare the pros and cons of threads vs MPI for parallel 
processing with Fortran? Or maybe you know a place where I can see a 
comparison? (pros and cons).

I've had difficulty finding information online. In fact, right now I'm 
not even sure that Fortran supports threads at all. I also tried a more 
general "threads vs MPI" search. The best I could find was a blog post, 
but it's not really a comparison:

http://kerneltrap.org/node/7105

This is not important, I'd just like to understand the subject better. 
Today I'm taking a break from my take-home exam and I'm curious about 
parallel programming.

Cheers,
Daniel.

In article <irbia7$j3p$1@dont-email.me>,
Daniel Carrera  <daniel@gmail.com> wrote:
>
>Can anyone compare the pros and cons of threads vs MPI for parallel 
>processing with Fortran? Or maybe you know a place where I can see a 
>comparison? (pros and cons).

Fortran doesn't.  You may be surprised to know that C doesn't,
either, and nor does current C++.

You can look at http://www-uxsup.csx.cam.ac.uk/courses/Parallel/,
and replace the Parallel by MultiApplics and MPI for other pages.
I am currently writing a course on OpenMP.

>I've had difficulty finding information online. In fact, right now I'm 
>not even sure that Fortran supports threads at all. I also tried a more 
>general "threads vs MPI" search. The best I could find was a blog post, 
>but it's not really a comparison:
>
>http://kerneltrap.org/node/7105
>
>This is not important, I'd just like to understand the subject better. 
>Today I'm taking a break from my take-home exam and I'm curious about 
>parallel programming.

Please note that my courses are on what is PRACTICALLY usable;
while they have a little of the theory and fundamentals, they
don't have a lot.  Also, I have let the online versions get a
little out of date.


Regards,
Nick Maclaren.

In article <irbia7$j3p$1@dont-email.me>,
 Daniel Carrera <daniel@gmail.com> wrote:

> I've had difficulty finding information online.

When I do a google search on "mpi programming", I get 4.8 million 
hits.  You might also look for "spmd programming", which results in 
243,000 hits; that is almost always the way mpi is used.

> In fact, right now I'm 
> not even sure that Fortran supports threads at all.

When I do a google search on "openmp fortran", I get 355,000 hits.

In fortran, you might also be interested in "coarray fortran", which 
results in 4190 hits in a google search.  That is now a part of 
standard fortran, although support is not yet widespread.

Until recently, OMP/threads was a way to achieve modest levels of 
parallel processing, say on 2 to 8 threads.  MPI in contrast has 
been applied on supercomputers with millions of cores.  More 
recently, mixed programming models have been developed where 
OMP/threads are used on each multicore node, and the SPMD/MPI model 
is used to communicate between nodes.  Given the current hardware 
trajectory, that must almost certainly be the important programming 
model in the near future.

$.02 -Ron Shepard

On 5/22/2011 1:48 PM, Daniel Carrera wrote:
> Hello,
>
>
> Can anyone compare the pros and cons of threads vs MPI for parallel
> processing with Fortran? Or maybe you know a place where I can see a
> comparison? (pros and cons).
>
> I've had difficulty finding information online. In fact, right now I'm
> not even sure that Fortran supports threads at all. I also tried a more
> general "threads vs MPI" search. The best I could find was a blog post,
> but it's not really a comparison:
>
Most current Fortran compilers support OpenMP threading, although it's 
not part of the Fortran standard.  This usually involves less 
development effort than MPI, and is sufficient for common single memory 
image platforms.  On linux and similar OS, the OpenMP implementation is 
likely to be built on pthreads, so offers a degree of compatibility with 
lower level threading.  In my experience, gfortran implementation of 
OpenMP on linux is excellent, not so much so on Windows.
Co-arrays, just recently added to Fortran standard, supported by few 
compilers, give a higher level programming model for MPI or OpenMP 
parallelism.  It's not necessarily compatible with direct use of MPI or 
OpenMP.
MPI supporting multiple ranks per node/memory image often requires 
significantly more RAM than threading by OpenMP or otherwise, with a 
performance advantage, at least on non-uniform memory access.
OpenMP is frequently used in combination with MPI, usually in accordance 
with the MPI_THREAD_FUNNELED model, in order to combine the advantages 
of both.
Needless to say, if you search on any of these topics, you should be 
busy for quite a while.

-- 
Tim Prince

On 05/22/2011 09:00 PM, Ron Shepard wrote:
> In article<irbia7$j3p$1@dont-email.me>,
>   Daniel Carrera<daniel@gmail.com>  wrote:
>
>> I've had difficulty finding information online.
>
> When I do a google search on "mpi programming", I get 4.8 million
> hits.  You might also look for "spmd programming", which results in
> 243,000 hits; that is almost always the way mpi is used.


Perhaps I wasn't clear. It was threads that I had trouble with. Search 
for "pthreads fortran" or "threads fortran" or "threads vs MPI" and the 
hits I get don't seem very useful.


> When I do a google search on "openmp fortran", I get 355,000 hits.

Is OpenMP related to either threads or MPI?


> In fortran, you might also be interested in "coarray fortran", which
> results in 4190 hits in a google search.  That is now a part of
> standard fortran, although support is not yet widespread.

I'm actually doing a project related to coarray fortran and MPI this 
summer. But I was curious about threads.


> Until recently, OMP/threads was a way to achieve modest levels of
> parallel processing, say on 2 to 8 threads.  MPI in contrast has
> been applied on supercomputers with millions of cores.

Thanks. And I guess I just learned that OpenMP means threads. So I was 
probably searching for the wrong term when I searched for "fortran threads".


> More
> recently, mixed programming models have been developed where
> OMP/threads are used on each multicore node, and the SPMD/MPI model
> is used to communicate between nodes.  Given the current hardware
> trajectory, that must almost certainly be the important programming
> model in the near future.

Ok.  And by "node" we basically mean "computer" or at least a CPU with 
dedicated memory (i.e. in a multi-CPU machine), right?

Daniel.

On 05/22/2011 09:35 PM, Tim Prince wrote:
> Most current Fortran compilers support OpenMP threading, although it's
> not part of the Fortran standard. This usually involves less development
> effort than MPI, and is sufficient for common single memory image
> platforms. On linux and similar OS, the OpenMP implementation is likely
> to be built on pthreads, so offers a degree of compatibility with lower
> level threading. In my experience, gfortran implementation of OpenMP on
> linux is excellent, not so much so on Windows.

Ok. Now I'm searching for "OpenMP vs MPI" and I'm getting some useful 
hits. I think I now have a rough understanding:


1) MPI is like having multiple processes communicating by TCP/IP. The 
TCP/IP stack creates overhead, making MPI slower for a single computer, 
but on the other hand, MPI is the only option for multiple nodes.


2) OpenMP is more "fine grain" than MPI. For example, you can use it to 
parallelize a loop, something which MPI doesn't do.



> MPI supporting multiple ranks per node/memory image often requires
> significantly more RAM than threading by OpenMP or otherwise, with a
> performance advantage, at least on non-uniform memory access.

This might be a stupid question, but what's a rank?

If MPI requires more memory, how can it have a performance advantage?


> Needless to say, if you search on any of these topics, you should be
> busy for quite a while.

Yeah. I think I just chose the wrong search term. Searching for "OpenMP 
vs MPI" gives several interesting hits.

Thanks,
Daniel.

Daniel Carrera wrote:
> 1) MPI is like having multiple processes communicating by TCP/IP. The
> TCP/IP stack creates overhead, making MPI slower for a single computer,
> but on the other hand, MPI is the only option for multiple nodes.

Well, MPI can have several backends: Threads, TCP/IP (or other protocol 
via ethernet), Infiniband, ... And with Threads it is not much slower 
than Theads/OpenMP.


> 2) OpenMP is more "fine grain" than MPI. For example, you can use it to
> parallelize a loop, something which MPI doesn't do.

Well, I think they follow different concepts. The general statement that 
one cannot use MPI to parallelize a loop is definitely wrong (we use MPI 
for that all the time.) However, quickly parallizing a loop with OpenMP 
is easier.

Roughly speaking with OpenMP one starts a single program which then 
forks/joins parallel section and all memory except of explicitly private 
memory is shared.

On the other hand, MPI (or coarrays): There, one starts several 
processes such that all variables are private and one explicitly needs 
to transfer the data if it has to be available on the other process.
(That's not completely true, especially with MPI_Alloc_Mem and 
single-sided communication, one can share memory to a certain extend.)

The consequence is that starting with OpenMP parallelization is easier: 
One just needs to add some "$omp parallel do" around a hot loop and one 
is done. (Well, one should spend some time to check whether it is 
parallelizable and which variables should be private or shared.) If one 
wants to improve the scaling (the ratio between speedup in wall time 
versus number of cores/processors used), it gets increasingly difficult 
with OpenMP - in particular, forking/joining threads does not come for 
free and with, e.g., 64 threads it might start to matter.

With MPI, one needs to parallelize the whole program at once, raising 
the starting barrier, but it then can be used also on distributed memory 
machines and it is - arguably - easier to tune the scaling than OpenMP. 
By construction, the memory is not shared and thus there is some data 
duplication - inceasing the required total memory compared with OpenMP. 
However, as the data is only partially duplicated, the memory per node 
can decrease if one goes to multiple nodes.

Both OpenMP and MPI are widely used (often also in the same program). 
Their specification is available at http://www.mpi-forum.org/ and at 
http://openmp.org/wp/.

Tobias

PS: For POSIX Threads (pthreads) there exists a C binding module at 
http://www.daniellnagle.com/pub/pthread.f03  (untested by me)

On 5/22/2011 4:06 PM, Daniel Carrera wrote:
> On 05/22/2011 09:35 PM, Tim Prince wrote:
>> Most current Fortran compilers support OpenMP threading, although it's
>> not part of the Fortran standard. This usually involves less development
>> effort than MPI, and is sufficient for common single memory image
>> platforms. On linux and similar OS, the OpenMP implementation is likely
>> to be built on pthreads, so offers a degree of compatibility with lower
>> level threading. In my experience, gfortran implementation of OpenMP on
>> linux is excellent, not so much so on Windows.
>
> Ok. Now I'm searching for "OpenMP vs MPI" and I'm getting some useful
> hits. I think I now have a rough understanding:
>
>
> 1) MPI is like having multiple processes communicating by TCP/IP. The
> TCP/IP stack creates overhead, making MPI slower for a single computer,
> but on the other hand, MPI is the only option for multiple nodes.
Message passing is usually via memcpy() within a single node, often by 
higher performance methods than TCP/IP when running on a cluster of 
significant size (e.g. InfiniBand/OFED and the like), but these are 
selectable at run time, so you could have some portability across 
cluster configurations.
>
>
> 2) OpenMP is more "fine grain" than MPI. For example, you can use it to
> parallelize a loop, something which MPI doesn't do.
>
>
>
>> MPI supporting multiple ranks per node/memory image often requires
>> significantly more RAM than threading by OpenMP or otherwise, with a
>> performance advantage, at least on non-uniform memory access.
>
> This might be a stupid question, but what's a rank?
rank is MPI term for number of MPI process, 0...n-1  I won't further 
embarrass myself trying to distinguish between rank and process. 
Colleagues who should know better don't distinguish between MPI rank and 
OpenMP thread.
>
> If MPI requires more memory, how can it have a performance advantage?
Each rank or process normally has a distinct address space, so, for 
example, no false sharing.  Insufficient RAM evidently would destroy the 
advantage.
You had an earlier response indicating the growing trend toward 
combination of OpenMP and MPI, and similar organization.
>
>
>> Needless to say, if you search on any of these topics, you should be
>> busy for quite a while.
>
> Yeah. I think I just chose the wrong search term. Searching for "OpenMP
> vs MPI" gives several interesting hits.
>
> Thanks,
> Daniel.


-- 
Tim Prince

On 05/22/2011 11:15 PM, Tobias Burnus wrote:
> Well, MPI can have several backends: Threads, TCP/IP (or other protocol
> via ethernet), Infiniband, ... And with Threads it is not much slower
> than Theads/OpenMP.

Cool.

After a Google search I could not confirm that OpenMPI has this, but I'm 
sure it does. I did confirm that it has Infiniband support on by default.


> Well, I think they follow different concepts...


On that topic, last year I experimented a little with OMP, but I had a 
hard time figuring it out. I didn't really know what I was doing. In 
comparison, I have found MPI much easier to understand and use. Writing 
a simple program was very easy once I had all my dependencies installed.



> The consequence is that starting with OpenMP parallelization is easier:
> One just needs to add some "$omp parallel do" around a hot loop and one
> is done. (Well, one should spend some time to check whether it is
> parallelizable and which variables should be private or shared.) If one
> wants to improve the scaling (the ratio between speedup in wall time
> versus number of cores/processors used), it gets increasingly difficult
> with OpenMP - in particular, forking/joining threads does not come for
> free and with, e.g., 64 threads it might start to matter.
>
> With MPI, one needs to parallelize the whole program at once, raising
> the starting barrier, but it then can be used also on distributed memory
> machines and it is - arguably - easier to tune the scaling than OpenMP.
> By construction, the memory is not shared and thus there is some data
> duplication - inceasing the required total memory compared with OpenMP.
> However, as the data is only partially duplicated, the memory per node
> can decrease if one goes to multiple nodes.


Very interesting.


Cheers,
Daniel.

On 22/05/2011 6:48 PM, Daniel Carrera wrote:
> Can anyone compare the pros and cons of threads vs MPI for parallel
> processing with Fortran? Or maybe you know a place where I can see a
> comparison? (pros and cons).
...

> This is not important, I'd just like to understand the subject better.
> Today I'm taking a break from my take-home exam and I'm curious about
> parallel programming.

You should also check out GPGPU - the use of graphics processing units 
(gaming cards) for general purpose computing.  The programming model is 
a bit like OpenMP but with thousands of threads rather than half a dozen 
or so.   It's mostly programmed in C-like languages, but PGI have 
extensions to support it in their Fortran compiler.  Other compiler 
vendors are also making efforts to support GPGPU.

Daniel Carrera <daniel@gmail.com> wrote:

> Perhaps I wasn't clear. It was threads that I had trouble with. Search
> for "pthreads fortran" or "threads fortran" or "threads vs MPI" and the
> hits I get don't seem very useful.

Pthreads and MPI are libraries. While I've never used pthreads with
Fortran, I know that using MPI with fortran is trivial, and I imagine
pthreads is too.

For the conceptual part, see chapter 2 of tinyurl.com/EijkhoutHPC

Basically, there is no x versus y. Threads are shared memory, MPI is
distributed. There is simply no way to do threads beyond a few dozen
cores (unless you have a UV or a Gemini).

Victor.

Victor.

-- 
Victor Eijkhout -- eijkhout at tacc utexas edu