I've a code in which the master keeps receiving data from slaves and 
writes them to a file. There is lot of bottleneck here as all slaves 
send to master and master has to do file o/p.

while(not finished) {
receive tiles from any slave (blocking)in buffer bufLHS
write tiles (bufLHS) to file
}

Now is it possible to spawn a thread/process which will handle writing 
the tiles concurrently?
There is only one rec'ving buffer, so I think threads may not do it.
But processes have there own addr. space so can this be done by forking 
a process?

while(not finished) {
receive tiles from any slave (blocking) in bufLHS
fork a process which writes tiles (bufLHS) to file
}

Please tell me which one is correct?
Pushkar Pradhan

I went ahead and implemented this by forking a process:
master:
while(not finished) {
receive tiles from any slave (blocking) in bufLHS
fork a process which writes tiles (bufLHS) to file
}
slave:
read tiles into buffer bufRGB
process tiles, put result in bufLHS
send buffer bufLHS

I run this using mpirun -np 6 ......
So I make sure I've one PE left on which it can fork a new process, by 
doing this
if(rank == 0) /*master*/ {....}
else if(rank != (numtasks-1)) /*slave*/ {....}
Thus 1 PE is guaranteed to be free to fork the process, even then I 
don't see any reduction in execution time. Why would this happen?

Pushkar Pradhan wrote:
> I've a code in which the master keeps receiving data from slaves and 
> writes them to a file. There is lot of bottleneck here as all slaves 
> send to master and master has to do file o/p.
> 
> while(not finished) {
> receive tiles from any slave (blocking)in buffer bufLHS
> write tiles (bufLHS) to file
> }
> 
> Now is it possible to spawn a thread/process which will handle writing 
> the tiles concurrently?
> There is only one rec'ving buffer, so I think threads may not do it.
> But processes have there own addr. space so can this be done by forking 
> a process?
> 
> while(not finished) {
> receive tiles from any slave (blocking) in bufLHS
> fork a process which writes tiles (bufLHS) to file
> }
> 
> Please tell me which one is correct?
> Pushkar Pradhan
> 

> while(not finished) {
> receive tiles from any slave (blocking) in bufLHS
> fork a process which writes tiles (bufLHS) to file
> }

If you really do a fork for each tile
then I'm not surprised to see that this
is slow.

Pushkar Pradhan wrote:
> 
> I've a code in which the master keeps receiving data from slaves and
> writes them to a file. There is lot of bottleneck here as all slaves
> send to master and master has to do file o/p.

Maybe the ideal solution would be to let the slaves 
write to the file. Either via MPI-I/O or via NFS 
directly.
NFS could work if the slaves would touch only non
overlapping regions in the output file, but I'm not 
sure.

Other possibilities:

Would it be possible to build the file in memory
and write it in a single step after completion?

Would it be possible to have one master doing the
assignment of work and another master to gather
all results?

Would it be possible to parallelize over more
than one Tiff? So each slave would touch only one
file?

Did you try to make the tiles as wide as the 
whole picture? That should make a tile contiguous
in the file.

I'm not sure if I have enough memory to store whole file.
> Maybe the ideal solution would be to let the slaves 
> write to the file. Either via MPI-I/O or via NFS 
> directly.
> NFS could work if the slaves would touch only non
> overlapping regions in the output file, but I'm not 
> sure.
Yes each slave works on separate parts of the file so I would try this.

> Would it be possible to parallelize over more
> than one Tiff? So each slave would touch only one
> file?
No, I think I will only work on one file at a time.
Thanks,
Pushkar Pradhan

George,
Can you specify what you mean by writing via NFS.
I tried to open the o/p file in each PE and write but the file doesn't 
contains o/p from only the 1st PE.

Georg Bisseling wrote:
> Pushkar Pradhan wrote:
> 
>>I've a code in which the master keeps receiving data from slaves and
>>writes them to a file. There is lot of bottleneck here as all slaves
>>send to master and master has to do file o/p.
> 
> 
> Maybe the ideal solution would be to let the slaves 
> write to the file. Either via MPI-I/O or via NFS 
> directly.
> NFS could work if the slaves would touch only non
> overlapping regions in the output file, but I'm not 
> sure.
> 
> Other possibilities:
> 
> Would it be possible to build the file in memory
> and write it in a single step after completion?
> 
> Would it be possible to have one master doing the
> assignment of work and another master to gather
> all results?
> 
> Would it be possible to parallelize over more
> than one Tiff? So each slave would touch only one
> file?
> 
> Did you try to make the tiles as wide as the 
> whole picture? That should make a tile contiguous
> in the file.

Pushkar Pradhan wrote:
> 
> George,
> Can you specify what you mean by writing via NFS.
> I tried to open the o/p file in each PE and write but the file doesn't
> contains o/p from only the 1st PE.

Of course I assume that all ranks have
access to the same filesystem via NFS. 

Did you arrange for that?

Pushkar Pradhan wrote:
> 
> George,
> Can you specify what you mean by writing via NFS.
> I tried to open the o/p file in each PE and write but the file doesn't
> contains o/p from only the 1st PE.

Maybe you TIFF library does some caching in memory
so that it cannot be used in concurrent processes.

You can use a raw image file format to get full
control over that.

I don't know is there something specific I need to do for that?
I just do an open for write in all the processes and try to write it, 
and close it in all the PEs.


Georg Bisseling wrote:
> Pushkar Pradhan wrote:
> 
>>George,
>>Can you specify what you mean by writing via NFS.
>>I tried to open the o/p file in each PE and write but the file doesn't
>>contains o/p from only the 1st PE.
> 
> 
> Of course I assume that all ranks have
> access to the same filesystem via NFS. 
> 
> Did you arrange for that?

Pushkar Pradhan wrote:
> 
> I don't know is there something specific I need to do for that?
> I just do an open for write in all the processes and try to write it,
> and close it in all the PEs.

Well and I can only guess what exactly
went wrong in your experiment :-)

First all processes have to "see" the same
files under the same path names. Probably
you or your sysadmin did that.

The problem is that any caching either in
the NFS (client or maybe even server) or 
any caching in your Tiff routines could 
spoil the file, because there is no instance 
to ensure the cache coherency between your 
processes.

Imagine that you have two processes 0,1 and
an image with two tiles a and b assigned to 
0 and 1:

aaabbb
aaabbb
aaabbb

Process 0 will transform a to A, and 1 will
transform b to B.

Now imagine that the whole file is cached
either in the NFS client or in your Tiff
routines.

You probably will not see

AAABBB
AAABBB
AAABBB

but either

aaaBBB
aaaBBB
aaaBBB

if process 1 writes last ("wins") and the other way 
round if process 0 writes last. Or even a mixture
of a, b, A and B depending on who wins which file-
system block or cache block.

So it is essential to have the write regions of
different processes non interleaving, not even
touching the same blocks, whatever the blocksize
is inbetween OR to use some form of file locking
to serialize the writes. I guess you cannot arrange
for that in a portable way.

Maybe you should try MPI-I/O for this because it
tries to solve all these problems for you.

But whatever piece of code really solves the problem, 
it will perform better if you could use tiles that are
as wide as the whole picture, so that each write
is one contiguous block.

ciao
Georg