Hi there,

I am wondering if serious problem could happen if I copy a file that
is actively being written to.
I guess some file corruption might happen, right?

More specifically, I am dealing with the snapshot of  a qcow2 file.
# qemu-img snapshot -c snap base.qcow2
# cp base.qcow2 snap.qcow2
# qemu-img snapshot -a snap snap.qcow2
Do you think I will get an image snap.qcow2 that is completely
consistent with the base.qcow2 image at the time when the snapshot is
taken?

For my case, it is OK if new information written to the disk after
snapshot taken is lost or corrupted since all I want to copy is the
backed up content. But would this copy during write process actually
mess up the whole file and make it even un-readable?

Thanks a lot.
Shi

jinzishuai@gmail.com wrote:

> Hi there,
> 
> I am wondering if serious problem could happen if I copy a file that
> is actively being written to.
> I guess some file corruption might happen, right?
> 
> More specifically, I am dealing with the snapshot of  a qcow2 file.
> # qemu-img snapshot -c snap base.qcow2
> # cp base.qcow2 snap.qcow2
> # qemu-img snapshot -a snap snap.qcow2
> Do you think I will get an image snap.qcow2 that is completely
> consistent with the base.qcow2 image at the time when the snapshot is
> taken?
> 
> For my case, it is OK if new information written to the disk after
> snapshot taken is lost or corrupted since all I want to copy is the
> backed up content. But would this copy during write process actually
> mess up the whole file and make it even un-readable?
> 
> Thanks a lot.
> Shi

Depending on how you copy the file, you can test and be warned if the
file changed while it was being copied.  You should use file locking
(exclusive and shared, etc.) for opening and writing to, and reading
files that could change (or copying files).  I've never used the tool
you've mentioned though, so I don't know what checking, if any, that it
might do.
-- 
Not really a wanna-be, but I don't know everything.

On 2010-02-01, jinzishuai@gmail.com <jinzishuai@gmail.com> wrote:
> Hi there,
>
> I am wondering if serious problem could happen if I copy a file that
> is actively being written to.
> I guess some file corruption might happen, right?

The copied file will not have everything in it.  
>
> More specifically, I am dealing with the snapshot of  a qcow2 file.
> # qemu-img snapshot -c snap base.qcow2
> # cp base.qcow2 snap.qcow2
> # qemu-img snapshot -a snap snap.qcow2
> Do you think I will get an image snap.qcow2 that is completely
> consistent with the base.qcow2 image at the time when the snapshot is
> taken?
Of course not. If the file does not contain all the data, you cannot
copy all the data. Otherwise you could copy it yesterday, and not even
have to take the photo, and the copy will have the photo in it. 


>
> For my case, it is OK if new information written to the disk after
> snapshot taken is lost or corrupted since all I want to copy is the
> backed up content. But would this copy during write process actually
> mess up the whole file and make it even un-readable?

No. It would be readable-- maybe not consistant with the format that the
picture file expects, but the bits would be there. 


>
> Thanks a lot.
> Shi

On Mon, 01 Feb 2010 15:08:04 -0800, jinzishuai@gmail.com wrote:

> Hi there,
> 
> I am wondering if serious problem could happen if I copy a file that is
> actively being written to.
> I guess some file corruption might happen, right?
>
My _guess_ is that any "corruption" would show up as a failed backup 
attempt. The original file is hopefully locked by its owner for exclusive 
write access; thus, any external read attempt will not interfere with it. 
However, the backup attempt will be "blurry" at best.
> 
> More specifically, I am dealing with the snapshot of  a qcow2 file. #
> qemu-img snapshot -c snap base.qcow2 # cp base.qcow2 snap.qcow2
> # qemu-img snapshot -a snap snap.qcow2 Do you think I will get an image
> snap.qcow2 that is completely consistent with the base.qcow2 image at
> the time when the snapshot is taken?
>
My _guess_ is that it is not going to be consistent. Additional measures 
to freeze the processes would be necessary to obtain a consistent, 
instantaneous snapshot. In GNU/Linux, instantaneous moving snapshots are 
handled via the device mapper facility. The object you are attempting to 
backup is probably already be being managed as device mapper object. It 
may be possible to make a snapshot of a snapshot, but I think some manual 
tweaking of the underlying objects would be required to do it.

One other caveat: AFAIK, device mapper suffers a substantial performance 
hit whenever snapshots are active. YMMV. In other words, test that your 
virtual machines' performance remains acceptable when the file system is 
modified in this way.

> 
> For my case, it is OK if new information written to the disk after
> snapshot taken is lost or corrupted since all I want to copy is the
> backed up content. But would this copy during write process actually
> mess up the whole file and make it even un-readable?
> 
> Thanks a lot.
> Shi
>
Note: comments inline.

IMO, you're going to have to run some tests to find out. Please, report 
back with any information that you find along the way.

BTW, AFAIK, the OpenSolaris + zfs appears to offer the best possibilities 
for creating advanced, automatic snapshots. However, its future looks a 
bit muddy with the Oracle takeover of Sun. GNU/Linux is preparing a 
competing filesystem, btrfs, which will offer some of the same features. 
But AFAIK, it is not quite for production systems yet.

-- 
Douglas Mayne

On 2010-02-02, Douglas Mayne <invalid@invalid.com> wrote:
> On Mon, 01 Feb 2010 15:08:04 -0800, jinzishuai@gmail.com wrote:
>
>> Hi there,
>> 
>> I am wondering if serious problem could happen if I copy a file that is
>> actively being written to.
>> I guess some file corruption might happen, right?
>>
> My _guess_ is that any "corruption" would show up as a failed backup 
> attempt. The original file is hopefully locked by its owner for exclusive 
> write access; thus, any external read attempt will not interfere with it. 
> However, the backup attempt will be "blurry" at best.

File locking is something that depends totally on the program, and is a
voluntary thing-- ie the reading program must check to see if the file
is locked. ARAIK, there is no mechanism in Linux to lock a file so that
nothing else can read it. 


>> 
>> More specifically, I am dealing with the snapshot of  a qcow2 file. #
>> qemu-img snapshot -c snap base.qcow2 # cp base.qcow2 snap.qcow2
>> # qemu-img snapshot -a snap snap.qcow2 Do you think I will get an image
>> snap.qcow2 that is completely consistent with the base.qcow2 image at
>> the time when the snapshot is taken?
>>
> My _guess_ is that it is not going to be consistent. Additional measures 
> to freeze the processes would be necessary to obtain a consistent, 
> instantaneous snapshot. In GNU/Linux, instantaneous moving snapshots are 
> handled via the device mapper facility. The object you are attempting to 
> backup is probably already be being managed as device mapper object. It 
> may be possible to make a snapshot of a snapshot, but I think some manual 
> tweaking of the underlying objects would be required to do it.
>
> One other caveat: AFAIK, device mapper suffers a substantial performance 
> hit whenever snapshots are active. YMMV. In other words, test that your 
> virtual machines' performance remains acceptable when the file system is 
> modified in this way.
>

As far as I know, any read file is a consistant file. Ie, it has bytes
from the first location to the last location in the current file, and
thus the file, as a file, is not inconsistant. If the contents of the
file have some format, they it may well be that that the contents of the
file are inconsistant with that format. For example, a pgm file in its
header says that it has m rows and n columns and thus (for a mono 8 bit
image) it has mxn bytes counting from after the end of the header. The
file that copy may not have that many bytes in it. It will be
inconsistant with the format of a pgm file. If you have a database or a
compressed file, or... it also may be inconsistant with the format that
the program which reads that file expects. However, it will still be a
valid Linux file. 


>> 
>> For my case, it is OK if new information written to the disk after
>> snapshot taken is lost or corrupted since all I want to copy is the
>> backed up content. But would this copy during write process actually
>> mess up the whole file and make it even un-readable?

No. It would be readable, just incomplete. 

If your program expects some format for the contents, it might claim the
file to be unuseable.
>

unruh <unruh@wormhole.physics.ubc.ca> wrote:
> On 2010-02-02, Douglas Mayne <invalid@invalid.com> wrote:
>> On Mon, 01 Feb 2010 15:08:04 -0800, jinzishuai@gmail.com wrote:
>>
>>> Hi there,
>>> 
>>> I am wondering if serious problem could happen if I copy a file that is
>>> actively being written to.
>>> I guess some file corruption might happen, right?
>>>
>> My _guess_ is that any "corruption" would show up as a failed backup 
>> attempt. The original file is hopefully locked by its owner for exclusive 
>> write access; thus, any external read attempt will not interfere with it. 
>> However, the backup attempt will be "blurry" at best.
> 
> File locking is something that depends totally on the program, and is a
> voluntary thing-- ie the reading program must check to see if the file
> is locked. ARAIK, there is no mechanism in Linux to lock a file so that
> nothing else can read it. 

Actually there is, see man fcntl:

  Mandatory locks are enforced for all processes.  If a process tries  to
  perform  an  incompatible  access (e.g., read(2) or write(2)) on a file
  region that has an incompatible mandatory lock, then the result depends
  upon  whether the O_NONBLOCK flag is enabled for its open file descrip-
  tion.  If the O_NONBLOCK flag is  not  enabled,  then  system  call  is
  blocked  until  the lock is removed or converted to a mode that is com-
  patible with the access.  If the O_NONBLOCK flag is enabled,  then  the
  system call fails with the error EAGAIN or EWOULDBLOCK.

  To  make use of mandatory locks, mandatory locking must be enabled both
  on the file system that contains the file to be locked, and on the file
  itself.   Mandatory  locking  is enabled on a file system using the "-o
  mand" option to mount(8), or the MS_MANDLOCK flag for mount(2).  Manda-
  tory locking is enabled on a file by disabling group execute permission
  on the file and enabling the set-group-ID permission bit (see  chmod(1)
  and chmod(2)).

  The  Linux implementation of mandatory locking is unreliable.  See BUGS
  below.

Of course this still requires that the writing program use fcntl lock.


	Jerry

On Tue, 02 Feb 2010 22:30:29 +0000, unruh wrote:

> On 2010-02-02, Douglas Mayne <invalid@invalid.com> wrote:
>> On Mon, 01 Feb 2010 15:08:04 -0800, jinzishuai@gmail.com wrote:
>>
>>> Hi there,
>>> 
>>> I am wondering if serious problem could happen if I copy a file that is
>>> actively being written to.
>>> I guess some file corruption might happen, right?
>>>
>> My _guess_ is that any "corruption" would show up as a failed backup 
>> attempt. The original file is hopefully locked by its owner for exclusive 
>> write access; thus, any external read attempt will not interfere with it. 
>> However, the backup attempt will be "blurry" at best.
> 
> File locking is something that depends totally on the program, and is a
> voluntary thing-- ie the reading program must check to see if the file
> is locked. ARAIK, there is no mechanism in Linux to lock a file so that
> nothing else can read it. 
> 
Thanks for the correction. I see now that the *nix permission model does
not include flags for locks. _Services_ which run on top of *nix which
want file locking (such as Samba), are what give the impression that locks
are implemented natively, when they are not.

-- 
Douglas Mayne

Douglas Mayne <doug@localhost.localnet> wrote:
> Thanks for the correction. I see now that the *nix permission model does
> not include flags for locks.

You're looking in the wrong place. *nix-based systems /do/ provide
locking, but it's rarely anything to do with permissions. See flock(1),
flock(2), and lockf(3) for the relevant documentation.

Chris

On 2010-02-05, Chris Davies <chris-usenet@roaima.co.uk> wrote:
> Douglas Mayne <doug@localhost.localnet> wrote:
>> Thanks for the correction. I see now that the *nix permission model does
>> not include flags for locks.
>
> You're looking in the wrong place. *nix-based systems /do/ provide
> locking, but it's rarely anything to do with permissions. See flock(1),
> flock(2), and lockf(3) for the relevant documentation.
>
> Chris

man 2 flock
flock - apply or remove an advisory lock on an open file
....
flock()  places  advisory  locks  only;  given suitable permissions on a file, a
       process is free to ignore the use of flock() and perform I/O on the file.

"jinzishuai@gmail.com" <jinzishuai@gmail.com> writes:

> Hi there,
>
> I am wondering if serious problem could happen if I copy a file that
> is actively being written to.
> I guess some file corruption might happen, right?

Wrong.

Most files are written without locking.
If you are familiar with using the command:

tail -f

you'd know that reading a file as it's being written
causes no problems.

> More specifically, I am dealing with the snapshot of  a qcow2 file.
> # qemu-img snapshot -c snap base.qcow2
> # cp base.qcow2 snap.qcow2
> # qemu-img snapshot -a snap snap.qcow2
> Do you think I will get an image snap.qcow2 that is completely
> consistent with the base.qcow2 image at the time when the snapshot is
> taken?

Some buffering will take place.

> For my case, it is OK if new information written to the disk after
> snapshot taken is lost or corrupted since all I want to copy is the
> backed up content. But would this copy during write process actually
> mess up the whole file and make it even un-readable?

Sounds like you might want to be using the syslog facilities?

On 2010-02-05, Chris Davies <chris-usenet@roaima.co.uk> wrote:
> You're looking in the wrong place. *nix-based systems /do/ provide
> locking, but it's rarely anything to do with permissions. See flock(1),
> flock(2), and lockf(3) for the relevant documentation.


unruh <unruh@wormhole.physics.ubc.ca> wrote:
> man 2 flock
> flock - apply or remove an advisory lock on an open file

Yes, agreed. But also from lockf(3):

    There are also locks.txt and mandatory-locking.txt in the kernel
    source directory Documentation/filesystems

If you follow that through, and google a (very)
short while, you'll come across pages such as this,
http://www.hackinglinuxexposed.com/articles/20030623.html, "Mandatory
Locking can enforce file locks at the kernel level". (The application
must still explicitly apply the lock itself, but it becomes mandatory.)

Chris

On 2010-02-06, Chris Davies <chris-usenet@roaima.co.uk> wrote:
> On 2010-02-05, Chris Davies <chris-usenet@roaima.co.uk> wrote:
>> You're looking in the wrong place. *nix-based systems /do/ provide
>> locking, but it's rarely anything to do with permissions. See flock(1),
>> flock(2), and lockf(3) for the relevant documentation.
>
>
> unruh <unruh@wormhole.physics.ubc.ca> wrote:
>> man 2 flock
>> flock - apply or remove an advisory lock on an open file
>
> Yes, agreed. But also from lockf(3):
>
>     There are also locks.txt and mandatory-locking.txt in the kernel
>     source directory Documentation/filesystems
>
> If you follow that through, and google a (very)
> short while, you'll come across pages such as this,
> http://www.hackinglinuxexposed.com/articles/20030623.html, "Mandatory
> Locking can enforce file locks at the kernel level". (The application
> must still explicitly apply the lock itself, but it becomes mandatory.)

OK, it is possible, but not advised, for all the reasons listed in that
article. 
>
> Chris

unruh <unruh@wormhole.physics.ubc.ca> wrote:
> OK, it is possible, but not advised, for all the reasons listed in that
> article. 

Agreed.

Chris