I have two processes that need to communicate through shared memory.
They must be responsible for cleaning up after themselves without
outside influences (i.e. sysV ipcrm). The application is already
heavily disk IO bound, so the shared memory must not fetch/store
to/from disk unless absolutely necessary (paging). The shared memory
regions are generally quite large (i.e. typically 64MB-512MB). And
lastly, one of the processes is allowed to be unstable and may die. If
this happens, upon restart, the process MUST be able to reconnect to
the shared memory segment.

My current solution is to open a temporary file, immediately unlink it
(this is acceptable for cleanup), mmap it (MMAP_SHARED), send the file
descriptor to the related process, who then uses the file descriptor
for it's own mmap. If one of the processes dies, it can reconnect to
the shared memory segment using a named unix-domain socket to pass the
file descriptor.

Unfortunately, I've heard that many systems don't optimize away disk IO
for files that have no hard links (unlinked mmap'd files) - so now it's
quite likely that sync will be dumping all modified regions of this
(possibly) 512MB chunk of memory - extremely undesirable. Why don't
they? It seems quite silly to me not to optimize this away - just
because its not very common? From searching posts, it certainly seems
like this behavior is highly desireable and often needed.

I've read about kernel hacks that allow 'sync' to skip files with no
hard links. Can all (recent) open source kernels be hacked in this
manner - this could be left as an option to the end-user for a
significant performance improvement and well documented as application
tuning.

I've also read that some operating systems (Solaris?) allow reattaching
to a sysV shared memory segment even if it's been IPC_RMID'd, providing
at least one process is still attached to it. On what other operating
systems is this possible? If it is somewhat portable, this seems like a
the best implementation (falling back to mmap'd files on unsupported
platforms).

Skip this rant if there's an easy and portable solution I missed:
Unix-based operating systems were designed around IPC, yet they have
several brain-dead versions of shared memory. Heck, even Win32 does a
better job at this sort of thing - creating a memory-map fd (HANDLE)
without a file to back it (silently using the swap file) which can be
passed between processes and then mapped to the address space - and
Win32 isn't nearly as IPC-driven as Unix-based operating systems are.
It's shocking, IMO.

> I've read about kernel hacks that allow 'sync' to skip files with no
> hard links.

Forgot one question... do any operating systems do this by default or
by a build option (no hacking required)?

In article <1130895653.745811.40840@z14g2000cwz.googlegroups.com>,
 lindahlb@hotmail.com wrote:

> > I've read about kernel hacks that allow 'sync' to skip files with no
> > hard links.
> 
> Forgot one question... do any operating systems do this by default or
> by a build option (no hacking required)?

I think that versions of Unix that allow you to use a file as an 
additional swap partition have a combination of permission modes that 
tell the OS not to sync that file (since swap files don't need to be 
preserved across crashes).  But I don't remember the modes.

-- 
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***

> I think that versions of Unix that allow you to use a file as an
> additional swap partition have a combination of permission modes that
> tell the OS not to sync that file (since swap files don't need to be
> preserved across crashes).  But I don't remember the modes.

This sounds like exactly what I need - except for one problem, it
probably requires root access, right? I'll look into it though, thanks.

Anyone else have suggestions?

lindahlb@hotmail.com wrote:
> 
> I have two processes that need to communicate through shared memory.
> They must be responsible for cleaning up after themselves without
> outside influences (i.e. sysV ipcrm). The application is already
> heavily disk IO bound, so the shared memory must not fetch/store
> to/from disk unless absolutely necessary (paging). The shared memory
> regions are generally quite large (i.e. typically 64MB-512MB). And
> lastly, one of the processes is allowed to be unstable and may die. If
> this happens, upon restart, the process MUST be able to reconnect to
> the shared memory segment.
> 
> My current solution is to open a temporary file, immediately unlink it
> (this is acceptable for cleanup), mmap it (MMAP_SHARED), send the file
> descriptor to the related process, who then uses the file descriptor
> for it's own mmap. If one of the processes dies, it can reconnect to
> the shared memory segment using a named unix-domain socket to pass the
> file descriptor.

How are you planning this?  Is what you're saying is some one process
is the "master"; the restarted process will have to commumicate with
it to re-establish its relationship with this shared memory...?

If this is a parent-child, then the parent could just wait() on the
child and restart the child as needed.  Otherwise, you'll need some
protocol for _this_ process to communicate with process holding
this memory so that it can re-attach to it.

> 
> Unfortunately, I've heard that many systems don't optimize away disk IO
> for files that have no hard links (unlinked mmap'd files) - so now it's
> quite likely that sync will be dumping all modified regions of this
> (possibly) 512MB chunk of memory - extremely undesirable. Why don't
> they? It seems quite silly to me not to optimize this away - just
> because its not very common? From searching posts, it certainly seems
> like this behavior is highly desireable and often needed.

Paging algorithms have matured quite a bit since the 70's (which
systems did you hear 'bout?)  In simple terms, modern paging
algorithms won't load a page until it's touched, and won't write
a changed page until that page is needed (stolen) by another
process (or even itself, but you'd _really_ be low on memory then) -
so unless you're low on physical memory _anyway_, "sync"ing won't
be an issue for you.  BTW, 512 MByte is no big deal on some of these
platforms which have 64 Gig of physical ram...

> 
> I've read about kernel hacks that allow 'sync' to skip files with no
> hard links. Can all (recent) open source kernels be hacked in this
> manner - this could be left as an option to the end-user for a
> significant performance improvement and well documented as application
> tuning.
>

I think the concept of "sync" is much different now with filesystem
logging, etc., than it was some time ago...

> I've also read that some operating systems (Solaris?) allow reattaching
> to a sysV shared memory segment even if it's been IPC_RMID'd, providing
> at least one process is still attached to it. On what other operating
> systems is this possible? If it is somewhat portable, this seems like a
> the best implementation (falling back to mmap'd files on unsupported
> platforms).
>
> Skip this rant if there's an easy and portable solution I missed:
> Unix-based operating systems were designed around IPC, yet they have
> several brain-dead versions of shared memory. Heck, even Win32 does a
> better job at this sort of thing - creating a memory-map fd (HANDLE)
> without a file to back it (silently using the swap file) which can be
> passed between processes and then mapped to the address space - and
> Win32 isn't nearly as IPC-driven as Unix-based operating systems are.
> It's shocking, IMO.


-- 
Stephen

>> My current solution is to open a temporary file, immediately unlink it
>> (this is acceptable for cleanup), mmap it (MMAP_SHARED), send the file
>> descriptor to the related process, who then uses the file descriptor
>> for it's own mmap. If one of the processes dies, it can reconnect to
>> the shared memory segment using a named unix-domain socket to pass the
>> file descriptor.
> How are you planning this?  Is what you're saying is some one process
> is the "master"; the restarted process will have to commumicate with
> it to re-establish its relationship with this shared memory...?

One process is the master which uses a library which kicks off helper
processes to provide functionality. The helper processes, in part,
provide reliability for the master process. If the master process dies,
it has the capability to reconnect to the helper processes (reattaching
to shared memory, among other things). This ability to reconnect saves
extensive processing that would be required if the helper process just
restarted. This reconnection is done through the named unix domain
socket that I mentioned in my original post - so yes, a protocol for
this reconnection does exist.

> Paging algorithms have matured quite a bit since the 70's (which
> systems did you hear 'bout?)  In simple terms, modern paging
> algorithms won't load a page until it's touched, and won't write
> a changed page until that page is needed (stolen) by another
> process

Yes, paging algorithms have matured, but from what I've read, that is
only for virtual memory. I'm talking about a memory mapped file (that's
been unlinked) - which is being used for shared memory.

>From what I've read, most Unices (i.e. Linux) will NOT understand that
this unlinked memory mapped file is only being used as shared memory
and then treat it like memory, as opposed to a real file. Thus, the
sync process that flushes the disk buffers periodically (30 sec?) will
also flush this memory mapped file to disk - despite the fact that it
is pointless, as it's really just shared memory and doesn't need to
persist.

A somewhat recent post I read from the 1998 said that the best you can
hope for on Linux is to hack the kernel so the sync process skips files
that have no hard links:
http://groups.google.com/group/comp.os.linux.development.system/browse_thread/thread/59a33e8605ced5c/78739c0028025b32?lnk=st&q=mmap+sync+unlink&rnum=4#78739c0028025b32

> I think the concept of "sync" is much different now with filesystem
> logging, etc., than it was some time ago...

>From the above link, it appears not, unless we are not in sync (bad
pun, I know).

But maybe I am missing something - I'd appreciate any more information
you have to share on this topic.

> Paging algorithms have matured quite a bit since the 70's (which
> systems did you hear 'bout?)  In simple terms, modern paging
> algorithms won't load a page until it's touched, and won't write
> a changed page until that page is needed (stolen) by another
> process

I want to emphasize that I wish to avoid the extra IO of keeping a
memory mapped file synchronized (a 512 MB sync every so often
*shudder*) - not the extra IO of paging (which is unavoidable). There
may have been a miscommunication.