Hi,

I'm sending this post following the "The Hoard Scalable Memory
Allocator Options" discussion in which David Butenhof said:
> (And, no, POSIX TSD doesn't -- and couldn't -- work that way in any case
> because there's no association to a shared library. Each thread
> associates a thread-specific value with a shared key. Even deleting the
> key doesn't asynchronously remove values from existing threads; nor is
> operation of the per-thread destructor in any dependent on the location
> or existence of the pthread_key_t value.)

According to the above it seems that POSIX TSD can't be used safely in
shared libraries since there might be a race condition between thread
exit notification and pthread_key_delete called from library unload.

I'm trying to figure out a safe usage of TSD in shared libraries that
might be dynamically loaded/unloaded (e.g. COM DLL) and therefore
might have TSD in threads that outlives the shared library.

The basic usage that I thought about is:
1) Library load - create TSD key
2) Threads calling the library - set TSD on demand
3) Thread exit while library loaded - remove TSD
4) Library unload - delete the key and clear all TSD's assoc data

Step 4 requires that no thread exit notification will arrive *after*
TSD delete returns so it's possible to safely clear all the data
associated with outstanding threads. Obviously threads should not try
to access such TSD after unload started and that's the responsibility
of the caller that is not suppose to call the library once it started
unloading (i.e. not referencing dead object).

It seems that pthread_key_delete doesn't provide this guarantee and
therefore the destructor function can be called *after* the library
was unloaded causing a crash (i.e. destructor function unmapped).

In windows it seems possible to have safe usage of TSD since the
thread exist notifications and library unload are serialized via the
loader's lock (i.e. DllMain notifications). Notice that win32's
TlsAlloc (create TSD key), like pthread_key_create, guarantees that
TSDs are nullified which makes on demand usage easily safe.

I'm quite surprised that POSIX is not allowing safe usage of TSD in
shared libraries unless completely missed David Butenhof's point.

Thoughts?

Thanks,
Rani

On Oct 21, 12:43 pm, rani_shar...@hotmail.com wrote:

> I'm quite surprised that POSIX is not allowing safe usage of TSD in
> shared libraries unless completely missed David Butenhof's point.

Don't blame POSIX. Any program that uses non-system shared libraries
(that might be unloaded) should provide a sane way to use TSD. If it
doesn't, it's that program's fault.

One common way is to have a 'thread entry' and a 'thread exit'
function that is called by every thread that wants to use the library
or that is done using the library. Every thread that wants to use the
library must finish using the library before the library is unloaded.
This not only allows the TSD to be cleaned up before the library is
unloaded but also allows it to be cleaned up if a thread is not going
to use the library for a very long time.

DS

"David Schwartz" <davids@webmaster.com> wrote in message 
news:1193199710.197935.279740@t8g2000prg.googlegroups.com...
> On Oct 21, 12:43 pm, rani_shar...@hotmail.com wrote:
>
>> I'm quite surprised that POSIX is not allowing safe usage of TSD in
>> shared libraries unless completely missed David Butenhof's point.
>
> Don't blame POSIX. Any program that uses non-system shared libraries
> (that might be unloaded) should provide a sane way to use TSD. If it
> doesn't, it's that program's fault.
[...]

Yup!

http://groups.google.com/group/comp.programming.threads/msg/6b336565ccccbfbc

http://groups.google.com/group/comp.programming.threads/msg/c347fde1b9c919b1 

On Oct 23, 9:21 pm, David Schwartz <dav...@webmaster.com> wrote:
> On Oct 21, 12:43 pm, rani_shar...@hotmail.com wrote:
>
> > I'm quite surprised that POSIX is not allowing safe usage of TSD in
> > shared libraries unless completely missed David Butenhof's point.
>
> Don't blame POSIX. Any program that uses non-system shared libraries
> (that might be unloaded) should provide a sane way to use TSD. If it
> doesn't, it's that program's fault.

When you design facility that generates async notifications, don't you
think about providing reliable shutdown in which no notifications can
arrive after the lifetime of the facility ended (i.e. support
destruction of such facility)?

Otherwise you have to bind the lifetime of such facilities to the
lifetime of the process which is very restrictive (e.g. in shared
libraries). I understand that you can synthesized safe use cases but
that will require you to impose some strict usage restrictions on the
clients of the facilities (e.g. TSD key can be safely destruct if the
client owns all the threads).

The POSIX TSD (pthread_key_t) facility has async notifications
(destructor call on thread exit) but, FWIU, it doesn't provide
reliable shutdown (i.e. pthread_key_delete might not wait for pending
destruct notifications). In case that the owner of the TSD key doesn't
own one of the threads that uses that key then the TSD key owner can't
reliable destruct the key. According to the opengroup.org
specification of pthread_key_delete, proper implementation is still
allowed though supporting delete key within TSD destructor is a tricky
legacy case.

I prefer to avoid using and designing such restricting facilities
altogether in order to simplify the usage (e.g. unfortunately win32
also has some like UnregisterWait).

> One common way is to have a 'thread entry' and a 'thread exit'
> function that is called by every thread that wants to use the library
> or that is done using the library. Every thread that wants to use the
> library must finish using the library before the library is unloaded.

Properly designed facilities don't impose such restrictive usage.

Rani

On Oct 24, 9:01 am, rani_shar...@hotmail.com wrote:

> Properly designed facilities don't impose such restrictive usage.

You are welcome to design facilities with any restrictions or non-
restrictions that you want. POSIX simply provides you the tools to do
it.

POSIX made the choice to provide very low-level tools and to permit
people to create their own high-level tools. I think this decision is
the correct one.

You would only have a valid complaint if you could show that POSIX
didn't provide you sufficient tools to provide properly designed
facilities. POSIX provides plumbing and fixtures, not bathrooms.

DS

"David Schwartz" <davids@webmaster.com> wrote in message 
news:1193378197.753198.41580@y27g2000pre.googlegroups.com...
> On Oct 24, 9:01 am, rani_shar...@hotmail.com wrote:
>
>> Properly designed facilities don't impose such restrictive usage.
>
> You are welcome to design facilities with any restrictions or non-
> restrictions that you want. POSIX simply provides you the tools to do
> it.
>
> POSIX made the choice to provide very low-level tools and to permit
> people to create their own high-level tools. I think this decision is
> the correct one.
>
> You would only have a valid complaint if you could show that POSIX
> didn't provide you sufficient tools to provide properly designed
> facilities. POSIX provides plumbing and fixtures, not bathrooms.

Right:

http://groups.google.com/group/comp.programming.threads/msg/e4f4e37224cf0f3c

http://groups.google.com/group/comp.programming.threads/msg/6b336565ccccbfbc

On Oct 25, 10:56 pm, David Schwartz <dav...@webmaster.com> wrote:
> On Oct 24, 9:01 am, rani_shar...@hotmail.com wrote:
>
> > Properly designed facilities don't impose such restrictive usage.
>
> You are welcome to design facilities with any restrictions or non-
> restrictions that you want. POSIX simply provides you the tools to do
> it.
>
> POSIX made the choice to provide very low-level tools and to permit
> people to create their own high-level tools. I think this decision is
> the correct one.

Even low level building blocks should provide essential guarantees in
order to be useable.

I searched this group for discussions about pthread_key_delete and
found out that it's known to be problematic.
David Butenhof wrote:
> The basic truth here is that pthread_key_delete() was added late, and
> wasn't thoroughly integrated into the standard. The function was well
> intended, but poorly conceived and incompletely defined.

See "TSD key reusing issue Options" http://tinyurl.com/2m8wfq

> You would only have a valid complaint if you could show that POSIX
> didn't provide you sufficient tools to provide properly designed
> facilities. POSIX provides plumbing and fixtures, not bathrooms.

I see that you have no fear in loosing your credibility by writing
such manipulative nonsense.

Rani

<rani_sharoni@hotmail.com> wrote in message 
news:1192995810.144750.60480@i13g2000prf.googlegroups.com...
> Hi,
>
> I'm sending this post following the "The Hoard Scalable Memory
> Allocator Options" discussion in which David Butenhof said:
>> (And, no, POSIX TSD doesn't -- and couldn't -- work that way in any case
>> because there's no association to a shared library. Each thread
>> associates a thread-specific value with a shared key. Even deleting the
>> key doesn't asynchronously remove values from existing threads; nor is
>> operation of the per-thread destructor in any dependent on the location
>> or existence of the pthread_key_t value.)
>
> According to the above it seems that POSIX TSD can't be used safely in
> shared libraries since there might be a race condition between thread
> exit notification and pthread_key_delete called from library unload.
>
> I'm trying to figure out a safe usage of TSD in shared libraries that
> might be dynamically loaded/unloaded (e.g. COM DLL) and therefore
> might have TSD in threads that outlives the shared library.
>
> The basic usage that I thought about is:
> 1) Library load - create TSD key
> 2) Threads calling the library - set TSD on demand
> 3) Thread exit while library loaded - remove TSD
> 4) Library unload - delete the key and clear all TSD's assoc data
[...]


For vzoom per-thread heaps, it creates a single TlsKey in libvzoom.dll which 
all loaded libraries link to. So, for instance in window, you can have:

DllA: linked to libvzoom.dll
DllB: linked to libvzoom.dll


Process1: linked to libvzoom.dll
  libvzoom.dll = instances=1/TlsKey=123;

  LoadLibrary(DllA)
    libvzoom.dll = instances=2/TlsKey=123;

  LoadLibrary(DllB)
    libvzoom.dll = instances=3/TlsKey=123;

  void* block = DllA::function_foo() {
    return libvzoom::vz_malloc(16) {
      heap = TlsGetValue(TlsKey);
      if (! heap) {
        heap = [...];
        TlsSetValue(TlsKey, heap);
      }
    }
  }

  UnlockLibrary(DllA)
    libvzoom.dll = instances=2/TlsKey=123;

  UnlockLibrary(DllB)
    libvzoom.dll = instances=1/TlsKey=123;

  vz_free(block) {
    libvzoom::vz_free(block) {
      if (block) {
        heap = vz_sys_get_heap(block);
        thisheap = TlsGetValue(TlsKey);
        if (heap == thisheap) {
          vz_sys_heap_push_local(_hisheap, block);
        } else {
          vz_sys_heap_push_shared(heap, block);
        }
      }
    }
  }

ExitProcess
  libvzoom.dll = instances=0/TlsKey=freed;



All per-thread heaps are creates are created in libvzoom.dll and use a 
single TlsKey (e.g., 123 in the example). Windows says that there will not 
be two seperate instances of libvzoom.dll. In other words, if you call 
LoadLibrary on DllA, it will notice that libvzoom.dll is already loaded and 
just increment its instance counter. So, there is no tls key per-user dll in 
vzoom. Instead, libvzoom.dll maintins a single TLS key per-process.


Does this make sense to you know?

;^) 

[...]

> DllA: linked to libvzoom.dll
> DllB: linked to libvzoom.dll
> 
> 
> Process1: linked to libvzoom.dll
>  libvzoom.dll = instances=1/TlsKey=123;
> 
>  LoadLibrary(DllA)
>    libvzoom.dll = instances=2/TlsKey=123;
> 
>  LoadLibrary(DllB)
>    libvzoom.dll = instances=3/TlsKey=123;
> 
>  void* block = DllA::function_foo() {
>    return libvzoom::vz_malloc(16) {
>      heap = TlsGetValue(TlsKey);
>      if (! heap) {
>        heap = [...];
>        TlsSetValue(TlsKey, heap);
>      }
>    }
>  }
> 
>  UnlockLibrary(DllA)
>    libvzoom.dll = instances=2/TlsKey=123;
> 
>  UnlockLibrary(DllB)
>    libvzoom.dll = instances=1/TlsKey=123;
^^^^^^^^^^^^^^^^^^^^^^^^^^

  FreeLibrary(DllA)
    libvzoom.dll = instances=2/TlsKey=123;
 
  FreeLibrary(DllB)
    libvzoom.dll = instances=1/TlsKey=123;


>  vz_free(block) {
>    libvzoom::vz_free(block) {
>      if (block) {
>        heap = vz_sys_get_heap(block);
>        thisheap = TlsGetValue(TlsKey);
>        if (heap == thisheap) {
>          vz_sys_heap_push_local(_hisheap, block);
>        } else {
>          vz_sys_heap_push_shared(heap, block);
>        }
>      }
>    }
>  }
> 
> ExitProcess
>  libvzoom.dll = instances=0/TlsKey=freed;
[...]

On Oct 28, 5:22 pm, "Chris Thomasson" <cris...@comcast.net> wrote:
> <rani_shar...@hotmail.com> wrote in message
[...]
> > The basic usage that I thought about is:
> > 1) Library load - create TSD key
> > 2) Threads calling the library - set TSD on demand
> > 3) Thread exit while library loaded - remove TSD
> > 4) Library unload - delete the key and clear all TSD's assoc data
>
> For vzoom per-thread heaps, it creates a single TlsKey in libvzoom.dll which
> all loaded libraries link to. So, for instance in window, you can have:
>
> DllA: linked to libvzoom.dll
> DllB: linked to libvzoom.dll
>
> Process1: linked to libvzoom.dll
[...]
> All per-thread heaps are creates are created in libvzoom.dll and use a
> single TlsKey (e.g., 123 in the example). Windows says that there will not
> be two seperate instances of libvzoom.dll. In other words, if you call
> LoadLibrary on DllA, it will notice that libvzoom.dll is already loaded and
> just increment its instance counter. So, there is no tls key per-user dll in
> vzoom. Instead, libvzoom.dll maintins a single TLS key per-process.
>
> Does this make sense to you know?

Do you require your vzoom to be linked to the process exe and
therefore to have the lifetime of the process?

Do you support static linkage to your memory manager (i.e. no vzoom
DLL)?

When do you destruct your TLS key (and in general the memory manager)?

Do you support the following?
1) Exe starts with vzoom DLL
2) Exe loads DLL linked with vzoom DLL
3) Exe uses the DLL (e.g. just from main thread).
4) Exe unloads the DLL
5) vzoom DLL is unloaded

The above is common scenario for COM DLLs (e.g. loaded by
dllhost.exe).
You can self "leak" your vzoom DLL (via self LoadLibrary) to avoid
dynamic unloading but that's not recommended practice.

FWIW, in windows you can reliably perform proper cleanup in order to
unload your DLL (e.g. on DLL_PROCESS_DETACH, delete the TLS key and
destruct the memory manager with all the per-thread heaps; on
DLL_THREAD_DETACH, reclaim the per-thread heap). Unfortunately you
can't reliably do the same using POSIX TSD (i.e. pthread_key_delete is
not reliable for such usage).

Rani

<rani_sharoni@hotmail.com> wrote in message 
news:1193625923.570434.233120@q3g2000prf.googlegroups.com...
> On Oct 28, 5:22 pm, "Chris Thomasson" <cris...@comcast.net> wrote:
>> <rani_shar...@hotmail.com> wrote in message
> [...]
>> > The basic usage that I thought about is:
>> > 1) Library load - create TSD key
>> > 2) Threads calling the library - set TSD on demand
>> > 3) Thread exit while library loaded - remove TSD
>> > 4) Library unload - delete the key and clear all TSD's assoc data
>>
>> For vzoom per-thread heaps, it creates a single TlsKey in libvzoom.dll 
>> which
>> all loaded libraries link to. So, for instance in window, you can have:
>>
>> DllA: linked to libvzoom.dll
>> DllB: linked to libvzoom.dll
>>
>> Process1: linked to libvzoom.dll
> [...]
>> All per-thread heaps are creates are created in libvzoom.dll and use a
>> single TlsKey (e.g., 123 in the example). Windows says that there will 
>> not
>> be two seperate instances of libvzoom.dll. In other words, if you call
>> LoadLibrary on DllA, it will notice that libvzoom.dll is already loaded 
>> and
>> just increment its instance counter. So, there is no tls key per-user dll 
>> in
>> vzoom. Instead, libvzoom.dll maintins a single TLS key per-process.
>>
>> Does this make sense to you know?
>
> Do you require your vzoom to be linked to the process exe and
> therefore to have the lifetime of the process?

I require at least an explicit LoadLibrary, or implicit linking of 
libvzoom.dll before you call functions.


>
> Do you support static linkage to your memory manager (i.e. no vzoom
> DLL)?

I provide no static library; libvzoom.dll only...

;^(

Is this a deal breaker in your eyes?




> When do you destruct your TLS key (and in general the memory manager)?

I destruct TLS key when the last module reference count of an instance of 
libvzoom.dll within a process is removed.




> Do you support the following?
> 1) Exe starts with vzoom DLL
> 2) Exe loads DLL linked with vzoom DLL
> 3) Exe uses the DLL (e.g. just from main thread).
> 4) Exe unloads the DLL
> 5) vzoom DLL is unloaded

Yes.



> The above is common scenario for COM DLLs (e.g. loaded by
> dllhost.exe).
> You can self "leak" your vzoom DLL (via self LoadLibrary) to avoid
> dynamic unloading but that's not recommended practice.

I have been doing a lot of playing around with explicit dynamic dll loading. 
libvzoom.dll works, it doesn't crash like current version of libhoard.dll...



> FWIW, in windows you can reliably perform proper cleanup in order to
> unload your DLL (e.g. on DLL_PROCESS_DETACH, delete the TLS key and
> destruct the memory manager with all the per-thread heaps; on
> DLL_THREAD_DETACH, reclaim the per-thread heap). Unfortunately you
> can't reliably do the same using POSIX TSD (i.e. pthread_key_delete is
> not reliable for such usage).

On UNIX-like systems, libvzoom.so uses pthread_key_delete with explicit 
dynamic loading well. It does not crash or leak.


I have a binary; include file; and strict license that allows you to 
download bin and benchmark... I am going to post this in about an hour or 
two... You will be able to test for yourself! If you want to use 
commercially and see source, you need another license. I urge you to test to 
your heats content and post all of your results here. Thanks!


:^) 

rani_sharoni@hotmail.com wrote:
> Hi,
> 
> I'm sending this post following the "The Hoard Scalable Memory
> Allocator Options" discussion in which David Butenhof said:
>> (And, no, POSIX TSD doesn't -- and couldn't -- work that way in any case
>> because there's no association to a shared library. Each thread
>> associates a thread-specific value with a shared key. Even deleting the
>> key doesn't asynchronously remove values from existing threads; nor is
>> operation of the per-thread destructor in any dependent on the location
>> or existence of the pthread_key_t value.)
> 
> According to the above it seems that POSIX TSD can't be used safely in
> shared libraries since there might be a race condition between thread
> exit notification and pthread_key_delete called from library unload.
> 
> I'm trying to figure out a safe usage of TSD in shared libraries that
> might be dynamically loaded/unloaded (e.g. COM DLL) and therefore
> might have TSD in threads that outlives the shared library.
> 
> The basic usage that I thought about is:
> 1) Library load - create TSD key
> 2) Threads calling the library - set TSD on demand
> 3) Thread exit while library loaded - remove TSD
> 4) Library unload - delete the key and clear all TSD's assoc data
> 
> Step 4 requires that no thread exit notification will arrive *after*
> TSD delete returns so it's possible to safely clear all the data
> associated with outstanding threads. Obviously threads should not try
> to access such TSD after unload started and that's the responsibility
> of the caller that is not suppose to call the library once it started
> unloading (i.e. not referencing dead object).
> 
> It seems that pthread_key_delete doesn't provide this guarantee and
> therefore the destructor function can be called *after* the library
> was unloaded causing a crash (i.e. destructor function unmapped).
> 
> In windows it seems possible to have safe usage of TSD since the
> thread exist notifications and library unload are serialized via the
> loader's lock (i.e. DllMain notifications). Notice that win32's
> TlsAlloc (create TSD key), like pthread_key_create, guarantees that
> TSDs are nullified which makes on demand usage easily safe.
> 
> I'm quite surprised that POSIX is not allowing safe usage of TSD in
> shared libraries unless completely missed David Butenhof's point.

I've been "not getting around to" replying to this for a while. I guess 
I ought to make some time so I can mark the post "read" and be done with it!

First, in your final statement you talk about safe usage "in shared 
libraries", and that's a misleading statement. Any potential problems 
due to the unspecified interactions of concern here are due to DYNAMIC 
LOADING (and more specifically UNLOADING) of shared libraries in a 
running process. If you don't unload libraries, there's no issue. If the 
implementation doesn't have dlclose() or doesn't actually unmap address 
space on dlclose() (and there's no POSIX requirement that an 
implementation actually do so), there's no problem.

For applications that use dlclose() on an implementation where dlclose() 
unmaps memory, you've got a problem IF that implementation doesn't also 
invalidate destructors on unmap AND if you haven't called 
pthread_key_delete. Although note that if you have deleted the key, that 
doesn't remove TSD values set by existing threads. Since you've removed 
the destructor along with the key, those threads won't be able to "clean 
up after themselves" when they terminate.

Simple synchronization between delete and thread termination is a minor 
issue. It's true the standard doesn't specifically point this out, and 
some naive implementors might miss the implicit requirement. That could 
be improved; but it's also pretty obviously an implementation bug since 
applications can't provide their own synchronization for access to the 
TSD key values & destructor data during thread rundown. It has to be 
solved in the implementation, and any implementor who does any real 
analysis or testing simply won't have that problem.

Perhaps a bigger issue here is that POSIX has no standard for 
init/destroy handlers on dynamic shared library load and unload. Tru64's 
loader supported __init_XXX/__fini_XXX symbols to allow the shared 
library to get control at that point, as well as explicit ld options to 
specify init/fini handlers. Other systems often have similar mechanisms, 
but there's no formal requirement or standard mechanism. There probably 
should be. Anyway, that means that strictly conforming/portable code has 
no mechanism for a dynamic shared library to delete its own TSD keys on 
unload. Since the standard also doesn't require (or even suggest) that 
the implementation invalidate TSD (or atexit/pthread_atfork/etc) 
handlers that point into a library being unmapped, there's definitely a 
hole in the standard. That ought to be addressed, but someone interested 
in researching "current practice" sufficiently to completely describe 
the problem and propose an acceptable solution will need to devote the time.

POSIX does say that a new TSD key shall have the value NULL in all 
threads. Technically that means that IF the implementation reuses the 
same key index for the new key, it needs to asynchronously clear (but 
not DESTRUCT) the value in existing keys. More likely, most 
implementations just minimize the chances that will happen. (On Tru64 I 
re-used a destroyed key value only if there were no more that had never 
been assigned.)

On Oct 29, 4:52 am, Dave Butenhof <david.buten...@hp.com> wrote:
> rani_shar...@hotmail.com wrote:
> > Hi,
>
> > I'm sending this post following the "The Hoard Scalable Memory
> > Allocator Options" discussion in which David Butenhof said:
> >> (And, no, POSIX TSD doesn't -- and couldn't -- work that way in any case
> >> because there's no association to a shared library. Each thread
> >> associates a thread-specific value with a shared key. Even deleting the
> >> key doesn't asynchronously remove values from existing threads; nor is
> >> operation of the per-thread destructor in any dependent on the location
> >> or existence of the pthread_key_t value.)
>
> > According to the above it seems that POSIX TSD can't be used safely in
> > shared libraries since there might be a race condition between thread
> > exit notification and pthread_key_delete called from library unload.
>
> > I'm trying to figure out a safe usage of TSD in shared libraries that
> > might be dynamically loaded/unloaded (e.g. COM DLL) and therefore
> > might have TSD in threads that outlives the shared library.
>
> > The basic usage that I thought about is:
> > 1) Library load - create TSD key
> > 2) Threads calling the library - set TSD on demand
> > 3) Thread exit while library loaded - remove TSD
> > 4) Library unload - delete the key and clear all TSD's assoc data
>
> > Step 4 requires that no thread exit notification will arrive *after*
> > TSD delete returns so it's possible to safely clear all the data
> > associated with outstanding threads. Obviously threads should not try
> > to access such TSD after unload started and that's the responsibility
> > of the caller that is not suppose to call the library once it started
> > unloading (i.e. not referencing dead object).
>
> > It seems that pthread_key_delete doesn't provide this guarantee and
> > therefore the destructor function can be called *after* the library
> > was unloaded causing a crash (i.e. destructor function unmapped).
>
> > In windows it seems possible to have safe usage of TSD since the
> > thread exist notifications and library unload are serialized via the
> > loader's lock (i.e. DllMain notifications). Notice that win32's
> > TlsAlloc (create TSD key), like pthread_key_create, guarantees that
> > TSDs are nullified which makes on demand usage easily safe.
>
> > I'm quite surprised that POSIX is not allowing safe usage of TSD in
> > shared libraries unless completely missed David Butenhof's point.
>
> I've been "not getting around to" replying to this for a while. I guess
> I ought to make some time so I can mark the post "read" and be done with it!

Thanks for taking the time to reply.

> First, in your final statement you talk about safe usage "in shared
> libraries", and that's a misleading statement. Any potential problems
> due to the unspecified interactions of concern here are due to DYNAMIC
> LOADING (and more specifically UNLOADING) of shared libraries in a
> running process. If you don't unload libraries, there's no issue. If the
> implementation doesn't have dlclose() or doesn't actually unmap address
> space on dlclose() (and there's no POSIX requirement that an
> implementation actually do so), there's no problem.

I agree but in most cases it's better to assume that the shared
library can be dynamically loaded/unloaded and shared libraries that
don't support that are typically too restricting. Same is true for any
object - it should be destructible.

> For applications that use dlclose() on an implementation where dlclose()
> unmaps memory, you've got a problem IF that implementation doesn't also
> invalidate destructors on unmap AND if you haven't called
> pthread_key_delete. Although note that if you have deleted the key, that
> doesn't remove TSD values set by existing threads. Since you've removed
> the destructor along with the key, those threads won't be able to "clean
> up after themselves" when they terminate.

Calling pthread_key_delete is essential for proper destruction of its
containing facility (e.g. memory manager) since it stops the async
notifications. IMHO, proper usage is not restricted just for unmapped
libraries but for most use cases of such async facilities. For
example, in non proper usage the TSD destructor functions can
eventually call virtual function of already destructed object.

> Simple synchronization between delete and thread termination is a minor
> issue. It's true the standard doesn't specifically point this out, and
> some naive implementors might miss the implicit requirement. That could
> be improved; but it's also pretty obviously an implementation bug since
> applications can't provide their own synchronization for access to the
> TSD key values & destructor data during thread rundown. It has to be
> solved in the implementation, and any implementor who does any real
> analysis or testing simply won't have that problem.

It's encouraging to see that you think that guaranteeing no TSD
destruction after delete is actually mandatory. I thought that
providing this guarantee and supporting the delete from TSD destructor
might be tricky for most implementation. Avoiding TSD destruction
under the lock (to allow other TSD operations) might also to be non
trivial (the infamous recursive lock might be useful here). The delete/
destructor race seems subtle and I'll be happy to see an the sources
of implementation that handles it correctly.

> Perhaps a bigger issue here is that POSIX has no standard for
> init/destroy handlers on dynamic shared library load and unload. Tru64's
> loader supported __init_XXX/__fini_XXX symbols to allow the shared
> library to get control at that point, as well as explicit ld options to
> specify init/fini handlers. Other systems often have similar mechanisms,
> but there's no formal requirement or standard mechanism. There probably
> should be. Anyway, that means that strictly conforming/portable code has
> no mechanism for a dynamic shared library to delete its own TSD keys on
> unload. Since the standard also doesn't require (or even suggest) that
> the implementation invalidate TSD (or atexit/pthread_atfork/etc)
> handlers that point into a library being unmapped, there's definitely a
> hole in the standard. That ought to be addressed, but someone interested
> in researching "current practice" sufficiently to completely describe
> the problem and propose an acceptable solution will need to devote the time.

It's a good practice for the shared library to provide custom
initialization/uninitialization functions that must be called after-
load/ before-unload. Even in windows that has standard (DllMain)
function that is called upon load/unload there are severe usage
restrictions (e.g. waiting for another thread is big no-no) so it's
better to also provide custom functions.

BTW - I wonder how systems without standard load/unload notifications
supports C++ globals construction/destruction (or atexit callbacks)

> POSIX does say that a new TSD key shall have the value NULL in all
> threads. Technically that means that IF the implementation reuses the
> same key index for the new key, it needs to asynchronously clear (but
> not DESTRUCT) the value in existing keys. More likely, most
> implementations just minimize the chances that will happen. (On Tru64 I
> re-used a destroyed key value only if there were no more that had never
> been assigned.)- Hide quoted text -

I thought that the TSD facility is simply using a link list where
every node is a thread TSDs array so create key should lock the list
and nullify all the TSDs related to the new key. New threads should
create new TSDs arrays (e.g. when their stack is created) and nullify
it before inserting it to the list.

Rani

rani_sharoni@hotmail.com wrote:

> BTW - I wonder how systems without standard load/unload notifications
> supports C++ globals construction/destruction (or atexit callbacks)

I said there's no STANDARD, not that such mechanisms don't exist. I'm 
not aware of any system with dynamic shared library support that doesn't 
have the mechanism in some form. There's just no way for fully portable 
code to use it. Some are based on special symbol names of various 
patterns (like Tru64's __init_/__fini_ prefix), some on special linker 
options, etc.

The fact that the mechanism is non-standard isn't an issue for the 
system itself or its components; but can be an inconvenience for 
applications that want to be cleanly portable.

>> POSIX does say that a new TSD key shall have the value NULL in all
>> threads. Technically that means that IF the implementation reuses the
>> same key index for the new key, it needs to asynchronously clear (but
>> not DESTRUCT) the value in existing keys. More likely, most
>> implementations just minimize the chances that will happen. (On Tru64 I
>> re-used a destroyed key value only if there were no more that had never
>> been assigned.)- Hide quoted text -
> 
> I thought that the TSD facility is simply using a link list where
> every node is a thread TSDs array so create key should lock the list
> and nullify all the TSDs related to the new key. New threads should
> create new TSDs arrays (e.g. when their stack is created) and nullify
> it before inserting it to the list.

There are lots of possible implementations. Many of them will at least 
vaguely resemble your sketch, so it's not unreasonable.

rani_sharoni@hotmail.com wrote:
> On Oct 23, 9:21 pm, David Schwartz <dav...@webmaster.com> wrote:
>> On Oct 21, 12:43 pm, rani_shar...@hotmail.com wrote:
>>
> The POSIX TSD (pthread_key_t) facility has async notifications
> (destructor call on thread exit)

TSD destructors operate SYNCHRONOUSLY with and in the context of the 
thread that's exiting. That is, in fact, specifically why 
pthread_key_delete() doesn't call destructors for values held by running 
threads -- THAT would make the destructors asynchronous.

(Tru64 actually supported not only asynchronous destructors but also 
asynchronous constructors for TSD, as part of an extension to support 
dynamic shared library TLS; but POSIX supports no way to affect the TSD 
values of another thread.)

> but, FWIU, it doesn't provide
> reliable shutdown (i.e. pthread_key_delete might not wait for pending
> destruct notifications).

There are no "pending destructs". Threads exit, and during the process 
they'll atomically observe a TSD key destructor and the thread's current 
TSD value and, if both are non-NULL, call the destructor. An external 
call to pthread_key_delete() shall occur as if either atomically before 
or after that process. So either the thread will see the destructor and 
call it, and the deleter will proceed when its done; OR the deleter will 
clear the destructor and the subsequently exiting thread will see no 
destructor value and throw out the abandoned TSD value.

The process is synchronized and safe, but deterministic (and memory 
conserving) only if the application code is careful to ensure that the 
lifetime of a TSD key matches or exceeds that of all threads carrying 
values for that key. Since only the code owning the key can assign a 
value to the key, and it can get control via a destructor on thread 
termination, it's easy for code to track the lifetime.

> In case that the owner of the TSD key doesn't
> own one of the threads that uses that key then the TSD key owner can't
> reliable destruct the key. According to the opengroup.org
> specification of pthread_key_delete, proper implementation is still
> allowed though supporting delete key within TSD destructor is a tricky
> legacy case.

But the owner CAN know whether all assigned values have been destroyed. 
IF they haven't been, then an attempt to asynchronously unload the 
owning library is arguably erroneous. The result may be failure to 
unload (e.g., dlclose() succeeds but the data and text remain mapped 
into memory) or a memory leak (the abandoned TSD values can't be 
destroyed). (It's even possible that the next thread exit will SIGSEGV 
because the TSD destructor points to unmapped memory, if the key wasn't 
deleted.)

> I prefer to avoid using and designing such restricting facilities
> altogether in order to simplify the usage (e.g. unfortunately win32
> also has some like UnregisterWait).
> 
>> One common way is to have a 'thread entry' and a 'thread exit'
>> function that is called by every thread that wants to use the library
>> or that is done using the library. Every thread that wants to use the
>> library must finish using the library before the library is unloaded.
> 
> Properly designed facilities don't impose such restrictive usage.

You're talking about a property we might label "asynchronous unload 
safety". While I won't deny that it can sometimes be useful, there are 
many "properly designed" facilities that have no interest in this 
property and never will.

If the property is of critical importance to you, fine; you're simply 
not going to be writing standard POSIX code anytime soon. Since it 
doesn't sound like you're writing POSIX code anyway, I guess this isn't 
a big deal.