Our environment is Solaris 2.8 with C++

Inorder to implement proper rollback processing, it is mandatory that the
process writing data to a file be aware of how much data has been written
and flushed to disk.

We are considering implementing the following call.  The process writing the
file will continue to write data in small "chunks", but will invoke a File
Descriptor Flush once the amount of data approaches 9000 bytes.

Am I correct in my understanding that as long as I do not exceed the 9000
byte buffer, only an explicit "flush" or "close" will flush the data to
disk?

    setvbuf(realFD, NULL, _IOFBF, 9000);


Thanks

On Mon, 14 Jun 2004, David Frager wrote:

> Our environment is Solaris 2.8 with C++

That's Solaris 8...

> Inorder to implement proper rollback processing, it is mandatory that the
> process writing data to a file be aware of how much data has been written
> and flushed to disk.
>
> We are considering implementing the following call.  The process writing the
> file will continue to write data in small "chunks", but will invoke a File
> Descriptor Flush once the amount of data approaches 9000 bytes.
>
> Am I correct in my understanding that as long as I do not exceed the 9000
> byte buffer, only an explicit "flush" or "close" will flush the data to
> disk?
>
>     setvbuf(realFD, NULL, _IOFBF, 9000);

setvbuf only affects the buffer used by the standard I/O library
(fread, fwrite, and co), not read() and write().  Notice that its
first argument is a FILE *, not an int.

Apart from that, you're not quite right.  The buffers get marked
to be flushed, but the actually flushing could happen some time
later.

I think I'd use synchronous I/O (unless the performance impact is
too great) using read() and write().  Mixing standard I/O library
functions with read and write on the same file descriptor is NOT
a good idea.

HTH,

-- 
Rich Teer, SCNA, SCSA

President,
Rite Online Inc.

Voice: +1 (250) 979-1638
URL: http://www.rite-online.net

File access is done via fopen, fwrite, and fflush, fclose.  At specific
intervals an fsync is performed.



Rich Teer wrote:

> On Mon, 14 Jun 2004, David Frager wrote:
> 
>> Our environment is Solaris 2.8 with C++
> 
> That's Solaris 8...
> 
>> Inorder to implement proper rollback processing, it is mandatory that the
>> process writing data to a file be aware of how much data has been written
>> and flushed to disk.
>>
>> We are considering implementing the following call.  The process writing
>> the file will continue to write data in small "chunks", but will invoke a
>> File Descriptor Flush once the amount of data approaches 9000 bytes.
>>
>> Am I correct in my understanding that as long as I do not exceed the 9000
>> byte buffer, only an explicit "flush" or "close" will flush the data to
>> disk?
>>
>>     setvbuf(realFD, NULL, _IOFBF, 9000);
> 
> setvbuf only affects the buffer used by the standard I/O library
> (fread, fwrite, and co), not read() and write().  Notice that its
> first argument is a FILE *, not an int.
> 
> Apart from that, you're not quite right.  The buffers get marked
> to be flushed, but the actually flushing could happen some time
> later.
> 
> I think I'd use synchronous I/O (unless the performance impact is
> too great) using read() and write().  Mixing standard I/O library
> functions with read and write on the same file descriptor is NOT
> a good idea.
> 
> HTH,
> 

-- 
David Frager
ESN: 445-2383

David Frager <frager@nortelnetworks.com> writes:

>> On Mon, 14 Jun 2004, David Frager wrote:
>> 
>>> Our environment is Solaris 2.8 with C++
>> 
>> That's Solaris 8...
>> 
>>> Inorder to implement proper rollback processing, it is mandatory that the
>>> process writing data to a file be aware of how much data has been written
>>> and flushed to disk.

[snip]

> File access is done via fopen, fwrite, and fflush, fclose.  At specific
> intervals an fsync is performed.
>

David, if you want that kind of control over the data being or not being
written to the disk, you want to use open(2), write(2), close(2) and
sync(2). Using the functions you mention just adds another layer (streams)
which have their own buffering on top what OS gives you. 

Can you change your program to use open(),write() etc system calls?

Bye, Dragan

-- 
Dragan Cvetkovic, 

To be or not to be is true. G. Boole      No it isn't.  L. E. J. Brouwer

!!! Sender/From address is bogus. Use reply-to one !!!

Note that fclose() will also flush the buffer, and FILE * opened on
devices that respond w/ 1 to isatty(fd) will cause line buffering to 
be applied to that FILE *...

Transactional semantics can be more closely approximated w/ 
open/close/read/write as others have noted.  For more accurate
atomicity, write a new version of the file and close it and then
rename it on top of the old one...

- Bart

David Frager <frager@nortelnetworks.com> writes:

>File access is done via fopen, fwrite, and fflush, fclose.  At specific
>intervals an fsync is performed.

Make sure you precede fsync(fileno(fp)) with fflush(fp).

Casper
-- 
Expressed in this posting are my opinions.  They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.

David Frager wrote:
> File access is done via fopen, fwrite, and fflush, fclose.  At specific
> intervals an fsync is performed.
> 

Commenting only on the fsync() in your current architecture - you may benefit
from Solaris 9 and a kernel update that includes the fix for

4336082 fsync() call on files with no/few dirty pages is very slow

(112233-10 and later).  I don't believe this fix has been backported to 8
or is suitable for backport (it's a substantial change).

If the file you're fsync'ing is small then you won't have a problem.
If it's large and you have enough memory that a lot of it's pages get
cached the fsync's get slower with the number of pages cached (because
the old code would run down the entire list of cached pages for this
file deciding which need writing out, and the longer that list the
longer it took).

Gavin