Hi,

I have two threads, each tries to create a directory of the same name
at the same location using mkdir(). The threads need to be oblivious
of each others' existence. Do I have to worry about concurrency when I
use mkdir() to create this directory? Or will mkdir() always return
success in one thread and EEXIST in the other?

Thanks.

mkaushik <mayank.utexas@gmail.com> writes:
> I have two threads, each tries to create a directory of the same name
> at the same location using mkdir(). The threads need to be oblivious
> of each others' existence. Do I have to worry about concurrency when I
> use mkdir() to create this directory? Or will mkdir() always return
> success in one thread and EEXIST in the other?

AFAIK it's supposed to be atomic (i.e. at most one can succeed), but if
anyone has written this down in a formal way, I haven't found yet it.

Network filesystems are the most likely place to find it -not- being
atomic.

-- 
http://www.greenend.org.uk/rjk/

Richard Kettlewell <rjk@greenend.org.uk> writes:
> mkaushik <mayank.utexas@gmail.com> writes:
>> I have two threads, each tries to create a directory of the same name
>> at the same location using mkdir(). The threads need to be oblivious
>> of each others' existence. Do I have to worry about concurrency when I
>> use mkdir() to create this directory? Or will mkdir() always return
>> success in one thread and EEXIST in the other?
>
> AFAIK it's supposed to be atomic (i.e. at most one can succeed), but if
> anyone has written this down in a formal way, I haven't found yet it.
>
> Network filesystems are the most likely place to find it -not- being
> atomic.

Not really. Ultimatively, the 'network file system' ends up making
system calls on some other computer and if mkdir is atomic on the host
operating system of the network file system (IOW, if that isn't a
really ancient UNIX(*) where directories are created by invoking
mknod, followed by two link calls to create the . and .. entries), the
network file system will have no chance but to expose this property.

On Mar 25, 5:56=A0pm, mkaushik <mayank.ute...@gmail.com> wrote:

> I have two threads, each tries to create a directory of the same name
> at the same location using mkdir(). The threads need to be oblivious
> of each others' existence. Do I have to worry about concurrency when I
> use mkdir() to create this directory? Or will mkdir() always return
> success in one thread and EEXIST in the other?

This will almost certainly happen on every system I can imagine.
However, it is not guaranteed. So long as the directory didn't exist
before you called 'mkdir' and did exist during the call, the standards
permit 'mkdir' to return success.

This could happen in the following case:

1) The implementation confirms the directory does not yet exist.
2) The implementation sends a command to the underlying filesystem to
make the directory.
3) The underlying filesystem fails or connectivity to it is lost
before the reply to the command is read.
4) The implementation determines that the directory does now exist. It
does not know if it was created as a result of its operation in step 2
or not.
5) The implementation returns success as that's preferable in this
case. (And will be correct most of the time.)

This could happen on a network filesystem that uses a stateless
protocol over an unreliable medium.

DS

David Schwartz <davids@webmaster.com> writes:
> On Mar 25, 5:56�pm, mkaushik <mayank.ute...@gmail.com> wrote:
>
>> I have two threads, each tries to create a directory of the same name
>> at the same location using mkdir(). The threads need to be oblivious
>> of each others' existence. Do I have to worry about concurrency when I
>> use mkdir() to create this directory? Or will mkdir() always return
>> success in one thread and EEXIST in the other?
>
> This will almost certainly happen on every system I can imagine.
> However, it is not guaranteed. So long as the directory didn't exist
> before you called 'mkdir' and did exist during the call, the standards
> permit 'mkdir' to return success.
>
> This could happen in the following case:
>
> 1) The implementation confirms the directory does not yet exist.
> 2) The implementation sends a command to the underlying filesystem to
> make the directory.
> 3) The underlying filesystem fails or connectivity to it is lost
> before the reply to the command is read.
> 4) The implementation determines that the directory does now exist. It
> does not know if it was created as a result of its operation in step 2
> or not.
> 5) The implementation returns success as that's preferable in this
> case. (And will be correct most of the time.)
>
> This could happen on a network filesystem that uses a stateless
> protocol over an unreliable medium.

... provided implementation is completely broken because the
responsible people didn't understand what a TOCTOU race actually is:
The check in step 1 is completely useless because it communicates
nothing about the state of the filesystem at the time when 'the
command' will be executed. Step 4 is outright broken: The fact that a
directory came into existence by some means after step 1 communicates
nothing about the result of the command sent in step 2. And it is
certainly not 'preferable' to fabricate 'assumptions' in code based on
the pretext that they are likely going to be correct. Insofar this is
concerned, the implementation could as well just always return success
since this will be 'correct most of the time'.

Also, the UNIX(*) standard demands that mkdir is supposed to fail with
EEXIST when the named file already existed. This is not possible when
mkdir is not an atomic operation.

On Mar 28, 5:59=A0am, Rainer Weikusat <rweiku...@mssgmbh.com> wrote:
> David Schwartz <dav...@webmaster.com> writes:
> > On Mar 25, 5:56 pm, mkaushik <mayank.ute...@gmail.com> wrote:
>
> >> I have two threads, each tries to create a directory of the same name
> >> at the same location using mkdir(). The threads need to be oblivious
> >> of each others' existence. Do I have to worry about concurrency when I
> >> use mkdir() to create this directory? Or will mkdir() always return
> >> success in one thread and EEXIST in the other?
>
> > This will almost certainly happen on every system I can imagine.
> > However, it is not guaranteed. So long as the directory didn't exist
> > before you called 'mkdir' and did exist during the call, the standards
> > permit 'mkdir' to return success.
>
> > This could happen in the following case:
>
> > 1) The implementation confirms the directory does not yet exist.
> > 2) The implementation sends a command to the underlying filesystem to
> > make the directory.
> > 3) The underlying filesystem fails or connectivity to it is lost
> > before the reply to the command is read.
> > 4) The implementation determines that the directory does now exist. It
> > does not know if it was created as a result of its operation in step 2
> > or not.
> > 5) The implementation returns success as that's preferable in this
> > case. (And will be correct most of the time.)
>
> > This could happen on a network filesystem that uses a stateless
> > protocol over an unreliable medium.
>
> .. provided implementation is completely broken because the
> responsible people didn't understand what a TOCTOU race actually is:
> The check in step 1 is completely useless because it communicates
> nothing about the state of the filesystem at the time when 'the
> command' will be executed. Step 4 is outright broken: The fact that a
> directory came into existence by some means after step 1 communicates
> nothing about the result of the command sent in step 2. And it is
> certainly not 'preferable' to fabricate 'assumptions' in code based on
> the pretext that they are likely going to be correct. Insofar this is
> concerned, the implementation could as well just always return success
> since this will be 'correct most of the time'.
>
> Also, the UNIX(*) standard demands that mkdir is supposed to fail with
> EEXIST when the named file already existed. This is not possible when
> mkdir is not an atomic operation.

Thanks for the answers everyone!

mkaushik <mayank.utexas@gmail.com> writes:
> On Mar 28, 5:59�am, Rainer Weikusat <rweiku...@mssgmbh.com> wrote:
>> David Schwartz <dav...@webmaster.com> writes:
>> > On Mar 25, 5:56 pm, mkaushik <mayank.ute...@gmail.com> wrote:
>> >> I have two threads, each tries to create a directory of the same name
>> >> at the same location using mkdir(). The threads need to be oblivious
>> >> of each others' existence. Do I have to worry about concurrency when I
>> >> use mkdir() to create this directory? Or will mkdir() always return
>> >> success in one thread and EEXIST in the other?

[...]

> Thanks for the answers everyone!

Something which hasn't been mentioned explicitly so far: It is
possible that both calls fail with EEXIST despite one created the
directory: Provided that some kind of network filesystem with a
stateless server is being used (IOW, NFS). It is possible that the
server crashes after it created the directory but before it could send
a reply to the client which requested the action. Once the server is
again available, the retransmitted 'create this directory' requests
from the client should lead to EEXIST.

On Mar 28, 5:59=A0am, Rainer Weikusat <rweiku...@mssgmbh.com> wrote:

> Also, the UNIX(*) standard demands that mkdir is supposed to fail with
> EEXIST when the named file already existed. This is not possible when
> mkdir is not an atomic operation.

Yes, it is. The algorithm I specified will always fail if the named
file already existed before the operation started. There is no
requirement for atomicity. You can't just make up requirements because
they seem reasonable to you.

DS

David Schwartz <davids@webmaster.com> writes:
> Rainer Weikusat <rweiku...@mssgmbh.com> wrote:

>> Also, the UNIX(*) standard demands that mkdir is supposed to fail
>> with EEXIST when the named file already existed. This is not possible
>> when mkdir is not an atomic operation.
>
> Yes, it is. The algorithm I specified will always fail if the named
> file already existed before the operation started. There is no
> requirement for atomicity. You can't just make up requirements because
> they seem reasonable to you.

I'd go further than that: even when there is a standardized requirement
for atomicity (or any other property), you may still need to check what
the implementation does.  NFS's failure to properly support O_EXCL is
notorious.

-- 
http://www.greenend.org.uk/rjk/

David Schwartz <davids@webmaster.com> writes:
> On Mar 28, 5:59�am, Rainer Weikusat <rweiku...@mssgmbh.com> wrote:
>> Also, the UNIX(*) standard demands that mkdir is supposed to fail with
>> EEXIST when the named file already existed. This is not possible when
>> mkdir is not an atomic operation.
>
> Yes, it is. The algorithm I specified will always fail if the named
> file already existed before the operation started.

The operation starts when the 'backend' (whatever that happens to be)
starts  to process the mkdir request, not at some arbitrarily distant
random time in the past. It is not possible to fail with EEXIST when a
file already existed before a mkdir request was processed unless no
such file can be created between the time the test for existence is
made and the directory creation request is actually processed. Your
'algorithm' is in no way different from one where the OS creates a
database of all files during bootup and uses that to determine if some
file exists until the next boot in this respect: Some check is
made. Then, and unspecified amount of time passses. Then, an operation
is carried out, based on the presumption that the result of this check
is still valid. Put into pseudocode in a slightly different context:

	rc = stat("/tmp/toastbrot", &st);
        if (rc == -1 && errno == ENOENT) {
        	rc = open("/tmp/toastbrot", O_CREAT | O_RDWR, 0666);
		if (rc != -1) {
                	write(rc, "will we kill /etc/shadow", sizeof("will we kill /etc/shadow"));
        		close(rc);
		}
	}                
        	
and the answer to this question is: The guy who wrote this cannot
imagine that it would be possible. But it is.

> There is no requirement for atomicity. You can't just make up requirements because
> they seem reasonable to you.

And there is also no requirement that the implementation doesn't use
the 'file database created during booting' in the way I outlined
above EXCEPT that this would also be a broken implementation because
of the TOCTOU-race. Coming to think of it, it should also be okay to
let mkdir fail randomly with EEXIST since the result is again
identical: mkdir will or won't fail for some reason which has no
specific relation to the state of the filesystem at the time the
request was executed whatsoever.

Richard Kettlewell <rjk@greenend.org.uk> writes:
> David Schwartz <davids@webmaster.com> writes:
>> Rainer Weikusat <rweiku...@mssgmbh.com> wrote:
>>> Also, the UNIX(*) standard demands that mkdir is supposed to fail
>>> with EEXIST when the named file already existed. This is not possible
>>> when mkdir is not an atomic operation.
>>
>> Yes, it is. The algorithm I specified will always fail if the named
>> file already existed before the operation started. There is no
>> requirement for atomicity. You can't just make up requirements because
>> they seem reasonable to you.
>
> I'd go further than that: even when there is a standardized requirement
> for atomicity (or any other property), you may still need to check what
> the implementation does.  NFS's failure to properly support O_EXCL is
> notorious.

You actually need to do so because Kennedy was shot, clearly proving
that all kinds of things nobody expects CAN and DO happen ...

On Thu, 31 Mar 2011 11:14:25 +0100, Rainer Weikusat
<rweikusat@mssgmbh.com> wrote:

>David Schwartz <davids@webmaster.com> writes:
>> On Mar 28, 5:59�am, Rainer Weikusat <rweiku...@mssgmbh.com> wrote:
>>> Also, the UNIX(*) standard demands that mkdir is supposed to fail with
>>> EEXIST when the named file already existed. This is not possible when
>>> mkdir is not an atomic operation.
>>
>> Yes, it is. The algorithm I specified will always fail if the named
>> file already existed before the operation started.
>
>The operation starts when the 'backend' (whatever that happens to be)
>starts  to process the mkdir request, not at some arbitrarily distant
>random time in the past. It is not possible to fail with EEXIST when a
>file already existed before a mkdir request was processed unless no
>such file can be created between the time the test for existence is
>made and the directory creation request is actually processed. Your
>'algorithm' is in no way different from one where the OS creates a
>database of all files during bootup and uses that to determine if some
>file exists until the next boot in this respect: Some check is
>made. Then, and unspecified amount of time passses. Then, an operation
>is carried out, based on the presumption that the result of this check
>is still valid. Put into pseudocode in a slightly different context:
>
>	rc = stat("/tmp/toastbrot", &st);
>        if (rc == -1 && errno == ENOENT) {
>        	rc = open("/tmp/toastbrot", O_CREAT | O_RDWR, 0666);
>		if (rc != -1) {
>                	write(rc, "will we kill /etc/shadow", sizeof("will we kill /etc/shadow"));
>        		close(rc);
>		}
>	}                
>        	
>and the answer to this question is: The guy who wrote this cannot
>imagine that it would be possible. But it is.
>
>> There is no requirement for atomicity. You can't just make up requirements because
>> they seem reasonable to you.
>
>And there is also no requirement that the implementation doesn't use
>the 'file database created during booting' in the way I outlined
>above EXCEPT that this would also be a broken implementation because
>of the TOCTOU-race. Coming to think of it, it should also be okay to
>let mkdir fail randomly with EEXIST since the result is again
>identical: mkdir will or won't fail for some reason which has no
>specific relation to the state of the filesystem at the time the
>request was executed whatsoever.

Local filesystems (typically) are designed to provide a serial
consistent view ... but distributed filesystems make other choices.

NFS was designed to have serial (transactional) consistency similar to
a local filesystem.   But there are other distributed filesystems,
e.g., AFS (CMU's Andrew file system), that provide much weaker
consistency.   AFS is now the preferred remote file system at many
universities because it scales so much better than NFS.
[If you've wondered why so many CS students on Usenet seem to have so
much trouble getting simple file sharing programs to work properly
across multiple hosts, the widespread use of AFS is (a small) part of
the answer.]

Linux can run the OpenAFS client/server and has had an (incomplete,
maybe abandoned?) native module AFS implementation since 2.6.10.


Good developers have to be defensive.  Regardless of which operating
system you are using, an API can be trusted only over the domain for
which it was defined.  The Unix filesystem API, which Linux borrowed,
was defined for single host local storage - period.  It can be (and
has been) stretched to cache coherent clusters sharing storage, but
once distributed storage enters the mix, any guarantees made by the
Unix API no longer apply.  Excepting where the distributed file system
makes the same guarantees you cannot rely on anything in the local
API.

YMMV.
George

George Neuner <gneuner2@comcast.net> writes:
> On Thu, 31 Mar 2011 11:14:25 +0100, Rainer Weikusat

[...]

>>> There is no requirement for atomicity. You can't just make up requirements because
>>> they seem reasonable to you.
>>
>>And there is also no requirement that the implementation doesn't use
>>the 'file database created during booting' in the way I outlined
>>above EXCEPT that this would also be a broken implementation because
>>of the TOCTOU-race. Coming to think of it, it should also be okay to
>>let mkdir fail randomly with EEXIST since the result is again
>>identical: mkdir will or won't fail for some reason which has no
>>specific relation to the state of the filesystem at the time the
>>request was executed whatsoever.
>
> Local filesystems (typically) are designed to provide a serial
> consistent view ... but distributed filesystems make other choices.
>
> NFS was designed to have serial (transactional) consistency similar to
> a local filesystem.   But there are other distributed filesystems,
> e.g., AFS (CMU's Andrew file system), that provide much weaker
> consistency.

In the context of what is or isn't required for a standard-conforming
mkdir implementation this isn't really relevant. Also, this is another
'all kinds of weird stuff are known to exist' ("Since Kennedy was
shot. ...") statement. One could go so far as to call this
'FUD'. Either you have definite information that mkdir behaves in
such-and-such a (non-standard conforming) way on implementation X of
filesystem Z then please say so. Or you don't. Then, you are adding
nothing of any use to this particular thread.

[...]

> Good developers have to be defensive.

[...]

> you cannot rely on anything in the local API.

Taking this as it was stated, computers cannot be programmed at
all. While you are technically correct (eg, a power outage at the
wrong time or bits flipping themselves in a DRAM could produce almost
arbitrarily weird effects) this is not a practical standpoint insofar
'getting something done' is the objective: This implies that the
assumption that things will generally behave as they are supposed to
while deviations will be rare, identifiable and 'work-aroundable' has
to be made.

On Mar 31, 3:14=A0am, Rainer Weikusat <rweiku...@mssgmbh.com> wrote:

> David Schwartz <dav...@webmaster.com> writes:

> > Yes, it is. The algorithm I specified will always fail if the named
> > file already existed before the operation started.

> The operation starts when the 'backend' (whatever that happens to be)
> starts =A0to process the mkdir request, not at some arbitrarily distant
> random time in the past.

The operation starts when the application calls 'mkdir'. The standard
only requires the operation to fail with EEXIST if the directory
exists before the operation starts. I agree, the operation cannot be
said to have started before the call to 'mkdir'.

> It is not possible to fail with EEXIST when a
> file already existed before a mkdir request was processed unless no
> such file can be created between the time the test for existence is
> made and the directory creation request is actually processed.

That is not true. The implementation is showed does exactly that.

> Your
> 'algorithm' is in no way different from one where the OS creates a
> database of all files during bootup and uses that to determine if some
> file exists until the next boot in this respect: Some check is
> made. Then, and unspecified amount of time passses. Then, an operation
> is carried out, based on the presumption that the result of this check
> is still valid. Put into pseudocode in a slightly different context:
>
> =A0 =A0 =A0 =A0 rc =3D stat("/tmp/toastbrot", &st);
> =A0 =A0 =A0 =A0 if (rc =3D=3D -1 && errno =3D=3D ENOENT) {
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 rc =3D open("/tmp/toastbrot", O_CREAT | O=
_RDWR, 0666);
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 if (rc !=3D -1) {
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 write(rc, "will we kill /=
etc/shadow", sizeof("will we kill /etc/shadow"));
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 close(rc);
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 }
> =A0 =A0 =A0 =A0 } =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0
>
> and the answer to this question is: The guy who wrote this cannot
> imagine that it would be possible. But it is.

The specification only requires that EEXIST be returned if the
directory already exists before the application calls 'mkdir'. It is
not required to return EEXIST if the directory is created by something
else after the call to 'mkdir' begins.

> > There is no requirement for atomicity. You can't just make up requireme=
nts because
> > they seem reasonable to you.

> And there is also no requirement that the implementation doesn't use
> the 'file database created during booting' in the way I outlined
> above EXCEPT that this would also be a broken implementation because
> of the TOCTOU-race.

No, there is such a requirement. The specification says that the
implementation must return EEXIST if the directory exists before the
call to 'mkdir'.

> Coming to think of it, it should also be okay to
> let mkdir fail randomly with EEXIST since the result is again
> identical: mkdir will or won't fail for some reason which has no
> specific relation to the state of the filesystem at the time the
> request was executed whatsoever.

You are speaking nonsense because you are using non-concepts that
assume what you want to prove. By saying things like "At the time the
request was executed" you are implicitly assuming an atomic operation.
But that is the very thing you are trying to prove. There is no such
thing as "at the time the request was executed". There is "before the
request" and "after the request", sure. And the standards refer to
these times.

DS

David Schwartz <davids@webmaster.com> writes:
> On Mar 31, 3:14�am, Rainer Weikusat <rweiku...@mssgmbh.com> wrote:
>> David Schwartz <dav...@webmaster.com> writes:
>
>> > Yes, it is. The algorithm I specified will always fail if the named
>> > file already existed before the operation started.
>
>> The operation starts when the 'backend' (whatever that happens to be)
>> starts �to process the mkdir request, not at some arbitrarily distant
>> random time in the past.
>
> The operation starts when the application calls 'mkdir'. The standard
> only requires the operation to fail with EEXIST if the directory
> exists before the operation starts.

.... and just because the text doesn't explicitly state that 'the
implementation is supposed to work correctly' doesn't mean that it was
meant to allow incorrect implementations. Your non-atomic multi-step
algorithm is such an incorrect implementation: Because of the TOCTOU
race in it, its behaviour is actually undefined and cannot be defined.

David Schwartz <davids@webmaster.com> writes:
> The operation starts when the application calls 'mkdir'. The standard
> only requires the operation to fail with EEXIST if the directory
> exists before the operation starts.

An additional error in here I'd like to point out: Code which runs
after mkdir was called cannot check if such a file existed before
mkdir was called. It would need to be able to travel back in time for
this.

On Apr 1, 1:03=A0pm, Rainer Weikusat <rweiku...@mssgmbh.com> wrote:

> > The operation starts when the application calls 'mkdir'. The standard
> > only requires the operation to fail with EEXIST if the directory
> > exists before the operation starts.

> ... and just because the text doesn't explicitly state that 'the
> implementation is supposed to work correctly' doesn't mean that it was
> meant to allow incorrect implementations. Your non-atomic multi-step
> algorithm is such an incorrect implementation: Because of the TOCTOU
> race in it, its behaviour is actually undefined and cannot be defined.

Again, you are assuming what you want to prove. You state that my
implementation operates "incorrectly", but it is only incorrect if the
requirement you are claiming is in fact a requirement. My position is
that there is no such requirement.

I think we can certainly agree there is no race or undefined behavior
if two applications call 'mkdir' at the same time and it is arbitrary
which receives a 0 and which receives EEXIST. There will always be a
race between unsynchronized tasks that perform conflicting operations.
The only issue is whether returning 0 to both tasks is also a valid
outcome. There is no dispute that there is more than one valid
outcome.

There is only a race if there's a requirement. Otherwise, there are
just many valid outcomes, which there are in any case. So the question
of whether there's a requirement is logically prior to whether there's
a race. I say there is no requirement. You can't counter by saying
there's a race.

DS

On Apr 1, 1:23=A0pm, Rainer Weikusat <rweiku...@mssgmbh.com> wrote:
> David Schwartz <dav...@webmaster.com> writes:
> > The operation starts when the application calls 'mkdir'. The standard
> > only requires the operation to fail with EEXIST if the directory
> > exists before the operation starts.

> An additional error in here I'd like to point out: Code which runs
> after mkdir was called cannot check if such a file existed before
> mkdir was called. It would need to be able to travel back in time for
> this.

I agree. That's the correct reading of the EEXIST requirement. It
demands that if code can prove that the directory must have existed
prior to its invocation of 'mkdir', it must get an 'EEXIST' error
return.

DS

On Fri, 01 Apr 2011 16:52:50 +0100, Rainer Weikusat
<rweikusat@mssgmbh.com> wrote:

>George Neuner <gneuner2@comcast.net> writes:
>>
>> Local filesystems (typically) are designed to provide a serial
>> consistent view ... but distributed filesystems make other choices.
>>
>> NFS was designed to have serial (transactional) consistency similar to
>> a local filesystem.   But there are other distributed filesystems,
>> e.g., AFS (CMU's Andrew file system), that provide much weaker
>> consistency.
>
>In the context of what is or isn't required for a standard-conforming
>mkdir implementation this isn't really relevant. Also, this is another
>'all kinds of weird stuff are known to exist' ("Since Kennedy was
>shot. ...") statement. One could go so far as to call this
>'FUD'. Either you have definite information that mkdir behaves in
>such-and-such a (non-standard conforming) way on implementation X of
>filesystem Z then please say so. Or you don't. Then, you are adding
>nothing of any use to this particular thread.

FUD is a strong word to apply to sound advice.  Linux now supports
more than forty different filesystems and believing that you can
successfully work with all of them based only on understanding the
filesystem API is wrong.


>> you cannot rely on anything in the local API.
>
>Taking this as it was stated, computers cannot be programmed at
>all. While you are technically correct (eg, a power outage at the
>wrong time or bits flipping themselves in a DRAM could produce almost
>arbitrarily weird effects) this is not a practical standpoint insofar
>'getting something done' is the objective: This implies that the
>assumption that things will generally behave as they are supposed to
>while deviations will be rare, identifiable and 'work-aroundable' has
>to be made.

Replying nonsense to statements taken out of context is not helpful.
My full statement, which I stand by, was: "Excepting where the
distributed file system makes the same guarantees you cannot rely on
anything in the local API."

The fact is that the Unix/Linux filesystem API was designed for local
devices.  NFS deliberately was designed to be compatible with local
filesystem semantics.  Nowhere is there a guarantee that the API will
be meaningfully for filesystems.

George

George Neuner <gneuner2@comcast.net> writes:
> On Fri, 01 Apr 2011 16:52:50 +0100, Rainer Weikusat
>>George Neuner <gneuner2@comcast.net> writes:
>>> Local filesystems (typically) are designed to provide a serial
>>> consistent view ... but distributed filesystems make other choices.
>>>
>>> NFS was designed to have serial (transactional) consistency similar to
>>> a local filesystem.   But there are other distributed filesystems,
>>> e.g., AFS (CMU's Andrew file system), that provide much weaker
>>> consistency.
>>
>>In the context of what is or isn't required for a standard-conforming
>>mkdir implementation this isn't really relevant. Also, this is another
>>'all kinds of weird stuff are known to exist' ("Since Kennedy was
>>shot. ...") statement. One could go so far as to call this
>>'FUD'. Either you have definite information that mkdir behaves in
>>such-and-such a (non-standard conforming) way on implementation X of
>>filesystem Z then please say so. Or you don't. Then, you are adding
>>nothing of any use to this particular thread.
>
> FUD is a strong word to apply to sound advice.  Linux now supports
> more than forty different filesystems and believing that you can
> successfully work with all of them based only on understanding the
> filesystem API is wrong.

In the context of what is or isn't required for a standard-conforming
mkdir implementation, this is still irrelevant. Also, this is still a
general statement lacking any real information and consequently, it is
basically just fearmongering: You claim to know something about some
of these 'forty different filesystems' which will cause unnamed
problems when not specifically mentioned operations are carried out in
an unknown environment, the short version of that is 'shit can and
does happen' and in absence of more specific information about this,
the only sensible reaction is 'ignore the possibility until shit does
happen and then, deal with the specific problem as required'.

It is neither possible to write code which can deal with unknown
problems nor to write code which can deal with all conceivable
problems and if it was, it wouldn't be desirable to do so because this
would mean adding a lot of code most of which won't be of any use
except obfuscating the source (a reader who is unfamiliar with the
contents of some text has to start with the assumption that everything
is relevant until the opposite has been proven).

>>> you cannot rely on anything in the local API.
>>
>>Taking this as it was stated, computers cannot be programmed at
>>all. While you are technically correct (eg, a power outage at the
>>wrong time or bits flipping themselves in a DRAM could produce almost
>>arbitrarily weird effects) this is not a practical standpoint insofar
>>'getting something done' is the objective: This implies that the
>>assumption that things will generally behave as they are supposed to
>>while deviations will be rare, identifiable and 'work-aroundable' has
>>to be made.
>
> Replying nonsense to statements taken out of context is not helpful.
> My full statement, which I stand by, was: "Excepting where the
> distributed file system makes the same guarantees you cannot rely on
> anything in the local API."

"It is only insofar possible to program computers as filesystem
operations can be avoided completely" isn't significantly less general
than 'it is impossible to program computers'.