My app processes files as they get dropped into a directory. There's a loop that keeps checking if a file is available, and if it is, then the file gets read.The trouble is that if the app detects a new file before it has been fully copied from another location, and it starts processing, then I get errors. Somehow, I need to be able to detect when the copy process has completed.I tried to test for an exclusive file lock using NIO, but it did not work. In order to lock a file, you have to open it first, and I get a FileNotFoundException when I create a FileInputStream on the file.java.io.FileNotFoundException: C:\mydir\product.xml (The process cannot access the file because it is being used by another process)	at java.io.FileInputStream.open(Native Method)	at java.io.FileInputStream.<init>(FileInputStream.java:106)How can I know when the copy process is complete. (Also, this has to work on both Windows and FreeBSD).

Possible solutions:  A) make a second directory for links to files  D) create a sentinel file for every file copied in; that is say filegetting copied in is named "lots_o_data.text" have the processperforming copy put in another file named something like"lots_o_data.text.finished".opalpaopalpa@gmail.comhttp://opalpa.info/

opalpa opalpa@gmail.com http://opalpa.info wrote On 03/05/07 16:13,:
> Possible solutions:
>   A) make a second directory for links to files
>   D) create a sentinel file for every file copied in; that is say file
> getting copied in is named "lots_o_data.text" have the process
> performing copy put in another file named something like
> "lots_o_data.text.finished".

    G) Copy the file using the name "lots_o_data.temporary"
and after copying rename it to "lots_o_data.text".  Teach
your program to ignore files named "....temporary".

-- 
Eric.Sosman@sun.com

Chris wrote:> My app processes files as they get dropped into a directory. There's a > loop that keeps checking if a file is available, and if it is, then the > file gets read.> > The trouble is that if the app detects a new file before it has been > fully copied from another location, and it starts processing, then I get > errors. Somehow, I need to be able to detect when the copy process has > completed.> > I tried to test for an exclusive file lock using NIO, but it did not > work. In order to lock a file, you have to open it first, and I get a > FileNotFoundException when I create a FileInputStream on the file.> > java.io.FileNotFoundException: C:\mydir\product.xml (The process cannot > access the file because it is being used by another process)>     at java.io.FileInputStream.open(Native Method)>     at java.io.FileInputStream.<init>(FileInputStream.java:106)> > How can I know when the copy process is complete. (Also, this has to > work on both Windows and FreeBSD).It stops throwing the exception.  What's the problem here?  If it throws this exception just wait for a few seconds and try again.-- Knute Johnsonemail s/nospam/knute/

Knute Johnson wrote:> Chris wrote:>> My app processes files as they get dropped into a directory. There's a >> loop that keeps checking if a file is available, and if it is, then >> the file gets read.>>>> The trouble is that if the app detects a new file before it has been >> fully copied from another location, and it starts processing, then I >> get errors. Somehow, I need to be able to detect when the copy process >> has completed.>>>> I tried to test for an exclusive file lock using NIO, but it did not >> work. In order to lock a file, you have to open it first, and I get a >> FileNotFoundException when I create a FileInputStream on the file.>>>> java.io.FileNotFoundException: C:\mydir\product.xml (The process >> cannot access the file because it is being used by another process)>>     at java.io.FileInputStream.open(Native Method)>>     at java.io.FileInputStream.<init>(FileInputStream.java:106)>>>> How can I know when the copy process is complete. (Also, this has to >> work on both Windows and FreeBSD).> > It stops throwing the exception.  What's the problem here?  If it throws > this exception just wait for a few seconds and try again.> Because it's almost always a bad idea to have exceptions thrown in the normal course of execution. Exceptions are just that -- unusual cases. Besides, there's no guarantee that this method will work in a cross-platform way. Maybe a particular operation system doesn't throw the exception. Then my app would start processing a partially-copied file, leading to all kinds of problems.

I believe BSD, being a unix, would not throw the exception.As for throwing exceptions, I've personally changed my mind onthrowing exceptions in course of execution.  I used to treat exceptionas a synonym for error.  Currently I occasionally treat exceptions asa control flow mechanism.opalpaopalpa@gmail.comhttp://opalpa.info/

Chris wrote:
> Knute Johnson wrote:
>> Chris wrote:
>>> My app processes files as they get dropped into a directory. There's 
>>> a loop that keeps checking if a file is available, and if it is, then 
>>> the file gets read.
>>>
>>> The trouble is that if the app detects a new file before it has been 
>>> fully copied from another location, and it starts processing, then I 
>>> get errors. Somehow, I need to be able to detect when the copy 
>>> process has completed.
>>>
>>> I tried to test for an exclusive file lock using NIO, but it did not 
>>> work. In order to lock a file, you have to open it first, and I get a 
>>> FileNotFoundException when I create a FileInputStream on the file.
>>>
>>> java.io.FileNotFoundException: C:\mydir\product.xml (The process 
>>> cannot access the file because it is being used by another process)
>>>     at java.io.FileInputStream.open(Native Method)
>>>     at java.io.FileInputStream.<init>(FileInputStream.java:106)
>>>
>>> How can I know when the copy process is complete. (Also, this has to 
>>> work on both Windows and FreeBSD).
>>
>> It stops throwing the exception.  What's the problem here?  If it 
>> throws this exception just wait for a few seconds and try again.
>>
> 
> Because it's almost always a bad idea to have exceptions thrown in the 
> normal course of execution. Exceptions are just that -- unusual cases. 

I think an incomplete file would be an unusual situation, it's certainly 
not what you want to find.

> Besides, there's no guarantee that this method will work in a 
> cross-platform way. Maybe a particular operation system doesn't throw 
> the exception. Then my app would start processing a partially-copied 
> file, leading to all kinds of problems.

I think you're toast.  The only other idea I have is to check 
periodically to see if the file size stops changing.  But I'm not sure 
that is good either because I have seen file systems that set the length 
of the file before completing the write.  You might try reading it again 
and again until it's checksum becomes stable.

You might just try using both the exception and the lock.  See if that 
works on most OS.

-- 

Knute Johnson
email s/nospam/knute/

> I believe BSD, being a unix, would not throw the exception.The possible absence of this exception is another reason to keep theoperation of coping the files and the operation of processing thefiles seperate in sequence.  If the format of the file permits asubsequence to be a valid file then reading a file too early couldchop of information.---I'm suprised to learn FileNotFoundException covers inaccseibility tofile.  I would have guessed that IOException gets thrown.opalpaopalpa@gmail.comhttp://opalpa.info/

Eric Sosman wrote:
> opalpa opalpa@gmail.com http://opalpa.info wrote On 03/05/07 16:13,:
> > Possible solutions:
> >   A) make a second directory for links to files
> >   D) create a sentinel file for every file copied in; that is say file
> > getting copied in is named "lots_o_data.text" have the process
> > performing copy put in another file named something like
> > "lots_o_data.text.finished".
>
>     G) Copy the file using the name "lots_o_data.temporary"
> and after copying rename it to "lots_o_data.text".  Teach
> your program to ignore files named "....temporary".

J) Use a self-delimiting format for the file contents (checksum, end-of-file
marker, anything which comes "for free" with the internal file format (such as
well-formdedness), MIME-style encapsulation of the "real" contents...)

    -- chris