Why is my kernel (Mageia 1's 2.6.38.8-server-10.mga) _NOT_
writing out dirty blocks to disk?

Yesterday, the contents of a number of files were lost when I
crashed the machine in a failed experiment with SCSI rescan to
remove a hot-unplugged ESATA disk.  (Rebuilding the software
RAID10s is another story...)  Today, recompiling WINE left 860MB
of dirty blocks needing to be written to disk, according to
/proc/meminfo.  After the compilation finished, I waited over 90
seconds, with the system essentially completely idle, and the
amount of dirty data did not go down.  Then, when I did a 'sync'
(as a non-root user), immediately everything flushed out to disk
in about 10-15 seconds time and the amount of dirty data went to
zero.  (I had "vmstat 3" and xcpustate running, along with a
periodic manual "grep Dirty /proc/meminfo".)

There is a value of 3000 in /proc/sys/vm/dirty_expire_centisecs,
which accoring to this page (and others I have read)

    http://www.westnet.com/~gsmith/content/linux-pdflush.htm

means that the kernel's helper processes _should_ get busy and
start to write out any dirty blocks at _MOST_ 30 seconds after
said blocks went dirty.  Even approx. 90 seconds after the
compilation job had finished, nothing was getting written.  Based
on yesterday's experience, nothing would have been written even
many minutes later.

What gives?  Why are my kernel and its helper processes being so
lazy?

Thanks.

-- 
Robert Riches
spamtrap42@jacob21819.net
(Yes, that is one of my email addresses.)

Robert Riches wrote:
> Why is my kernel (Mageia 1's 2.6.38.8-server-10.mga) _NOT_
> writing out dirty blocks to disk?
> 
> Yesterday, the contents of a number of files were lost when I
> crashed the machine in a failed experiment with SCSI rescan to
> remove a hot-unplugged ESATA disk.  (Rebuilding the software
> RAID10s is another story...)  Today, recompiling WINE left 860MB
> of dirty blocks needing to be written to disk, according to
> /proc/meminfo.  After the compilation finished, I waited over 90
> seconds, with the system essentially completely idle, and the
> amount of dirty data did not go down.  Then, when I did a 'sync'
> (as a non-root user), immediately everything flushed out to disk
> in about 10-15 seconds time and the amount of dirty data went to
> zero.  (I had "vmstat 3" and xcpustate running, along with a
> periodic manual "grep Dirty /proc/meminfo".)
> 
> There is a value of 3000 in /proc/sys/vm/dirty_expire_centisecs,
> which accoring to this page (and others I have read)
> 
>     http://www.westnet.com/~gsmith/content/linux-pdflush.htm
> 
> means that the kernel's helper processes _should_ get busy and
> start to write out any dirty blocks at _MOST_ 30 seconds after
> said blocks went dirty.  Even approx. 90 seconds after the
> compilation job had finished, nothing was getting written.  Based
> on yesterday's experience, nothing would have been written even
> many minutes later.
> 
> What gives?  Why are my kernel and its helper processes being so
> lazy?
> 
> Thanks.
> 

I'm no expert BUT I thought pages weren't flushed at all until the 
buffers were needed..or at least until the system was pretty much idle.

http://www.makelinux.net/books/lkd2/ch15lev1sec4

seems to have some detail.

On 2012-01-30, Robert Riches <spamtrap42@jacob21819.net> wrote:
> Why is my kernel (Mageia 1's 2.6.38.8-server-10.mga) _NOT_
> writing out dirty blocks to disk?

man sync

-- 
-----------------------------------------------------------------------------
  Roger Blake (Change "invalid" to "com" for email. Google Groups killfiled.)

  "Climate policy has almost nothing to do anymore with environmental
   protection... the next world climate summit in Cancun is actually
   an economy summit during which the distribution of the world's
   resources will be negotiated." -- Ottmar Edenhofer, IPCC
-----------------------------------------------------------------------------

On 2012-01-30, The Natural Philosopher <tnp@invalid.invalid> wrote:
> Robert Riches wrote:
>> Why is my kernel (Mageia 1's 2.6.38.8-server-10.mga) _NOT_
>> writing out dirty blocks to disk?
>> 
>> Yesterday, the contents of a number of files were lost when I
>> crashed the machine in a failed experiment with SCSI rescan to
>> remove a hot-unplugged ESATA disk.  (Rebuilding the software
>> RAID10s is another story...)  Today, recompiling WINE left 860MB
>> of dirty blocks needing to be written to disk, according to
>> /proc/meminfo.  After the compilation finished, I waited over 90
>> seconds, with the system essentially completely idle, and the
>> amount of dirty data did not go down.  Then, when I did a 'sync'
>> (as a non-root user), immediately everything flushed out to disk
>> in about 10-15 seconds time and the amount of dirty data went to
>> zero.  (I had "vmstat 3" and xcpustate running, along with a
>> periodic manual "grep Dirty /proc/meminfo".)
>> 
>> There is a value of 3000 in /proc/sys/vm/dirty_expire_centisecs,
>> which accoring to this page (and others I have read)
>> 
>>     http://www.westnet.com/~gsmith/content/linux-pdflush.htm
>> 
>> means that the kernel's helper processes _should_ get busy and
>> start to write out any dirty blocks at _MOST_ 30 seconds after
>> said blocks went dirty.  Even approx. 90 seconds after the
>> compilation job had finished, nothing was getting written.  Based
>> on yesterday's experience, nothing would have been written even
>> many minutes later.
>> 
>> What gives?  Why are my kernel and its helper processes being so
>> lazy?
>> 
>> Thanks.
>> 
>
> I'm no expert BUT I thought pages weren't flushed at all until the 
> buffers were needed..or at least until the system was pretty much idle.
>
> http://www.makelinux.net/books/lkd2/ch15lev1sec4
>
> seems to have some detail.

Yes, that page has info similar to what I had already studied.
These are what is in /proc/sys/vm/dirty*:

/proc/sys/vm/dirty_background_bytes:0
/proc/sys/vm/dirty_background_ratio:10
/proc/sys/vm/dirty_bytes:0
/proc/sys/vm/dirty_expire_centisecs:3000
/proc/sys/vm/dirty_ratio:20
/proc/sys/vm/dirty_writeback_centisecs:500

According to the descriptions on that page and elsewhere, the
worker threads should wake up ever 5 seconds and flush out
anything older than 30 second old.  The system had been idle
(>90% idle CPU with 12 logical CPUs for the worker threads to
use, disks essentially completely idle) for over 90 seconds, and
_nothing_ was getting flushed to disk.

On other Linux systems, I have seen writeback happen very well
right around 30 seconds after the pages went dirty.  Manually
running sync ever minute or so should _NOT_ be necessary.

-- 
Robert Riches
spamtrap42@jacob21819.net
(Yes, that is one of my email addresses.)

Robert Riches wrote:
> On 2012-01-30, The Natural Philosopher <tnp@invalid.invalid> wrote:
>> Robert Riches wrote:
>>> Why is my kernel (Mageia 1's 2.6.38.8-server-10.mga) _NOT_
>>> writing out dirty blocks to disk?
>>>
>>> Yesterday, the contents of a number of files were lost when I
>>> crashed the machine in a failed experiment with SCSI rescan to
>>> remove a hot-unplugged ESATA disk.  (Rebuilding the software
>>> RAID10s is another story...)  Today, recompiling WINE left 860MB
>>> of dirty blocks needing to be written to disk, according to
>>> /proc/meminfo.  After the compilation finished, I waited over 90
>>> seconds, with the system essentially completely idle, and the
>>> amount of dirty data did not go down.  Then, when I did a 'sync'
>>> (as a non-root user), immediately everything flushed out to disk
>>> in about 10-15 seconds time and the amount of dirty data went to
>>> zero.  (I had "vmstat 3" and xcpustate running, along with a
>>> periodic manual "grep Dirty /proc/meminfo".)
>>>
>>> There is a value of 3000 in /proc/sys/vm/dirty_expire_centisecs,
>>> which accoring to this page (and others I have read)
>>>
>>>     http://www.westnet.com/~gsmith/content/linux-pdflush.htm
>>>
>>> means that the kernel's helper processes _should_ get busy and
>>> start to write out any dirty blocks at _MOST_ 30 seconds after
>>> said blocks went dirty.  Even approx. 90 seconds after the
>>> compilation job had finished, nothing was getting written.  Based
>>> on yesterday's experience, nothing would have been written even
>>> many minutes later.
>>>
>>> What gives?  Why are my kernel and its helper processes being so
>>> lazy?
>>>
>>> Thanks.
>>>
>> I'm no expert BUT I thought pages weren't flushed at all until the 
>> buffers were needed..or at least until the system was pretty much idle.
>>
>> http://www.makelinux.net/books/lkd2/ch15lev1sec4
>>
>> seems to have some detail.
> 
> Yes, that page has info similar to what I had already studied.
> These are what is in /proc/sys/vm/dirty*:
> 
> /proc/sys/vm/dirty_background_bytes:0
> /proc/sys/vm/dirty_background_ratio:10
> /proc/sys/vm/dirty_bytes:0
> /proc/sys/vm/dirty_expire_centisecs:3000
> /proc/sys/vm/dirty_ratio:20
> /proc/sys/vm/dirty_writeback_centisecs:500
> 
> According to the descriptions on that page and elsewhere, the
> worker threads should wake up ever 5 seconds and flush out
> anything older than 30 second old.  The system had been idle
> (>90% idle CPU with 12 logical CPUs for the worker threads to
> use, disks essentially completely idle) for over 90 seconds, and
> _nothing_ was getting flushed to disk.
> 
> On other Linux systems, I have seen writeback happen very well
> right around 30 seconds after the pages went dirty.  Manually
> running sync ever minute or so should _NOT_ be necessary.
> 

Maybe the drivers were buffering everything...not the kernel.
I had that problem with a RAID once. you could even sync, but it was no 
guarantee the data was on the disks. The RAID sub sytem reported 'all 
written' and that was that. Except it wasn't. it was in the RAID system 
buffers.

Robert Riches <spamtrap42@jacob21819.net> writes:
> Why is my kernel (Mageia 1's 2.6.38.8-server-10.mga) _NOT_
> writing out dirty blocks to disk?
>
> Yesterday, the contents of a number of files were lost when I
> crashed the machine in a failed experiment with SCSI rescan to
> remove a hot-unplugged ESATA disk.  (Rebuilding the software
> RAID10s is another story...)  Today, recompiling WINE left 860MB
> of dirty blocks needing to be written to disk, according to
> /proc/meminfo.  After the compilation finished, I waited over 90
> seconds, with the system essentially completely idle, and the
> amount of dirty data did not go down.  Then, when I did a 'sync'
> (as a non-root user), immediately everything flushed out to disk
> in about 10-15 seconds time and the amount of dirty data went to
> zero.  (I had "vmstat 3" and xcpustate running, along with a
> periodic manual "grep Dirty /proc/meminfo".)
>
> There is a value of 3000 in /proc/sys/vm/dirty_expire_centisecs,
> which accoring to this page (and others I have read)
>
>     http://www.westnet.com/~gsmith/content/linux-pdflush.htm

pdflush no longer exists, and dirty_expire_centisecs is no longer used.
Which doesn't answer your question as such but does explain why the old
documentation isn't helping.

As a starting point for the new logic:

http://kernelnewbies.org/Linux_2_6_32#head-72c3f91947738f1ea52f9ed21a89876730418a61

-- 
http://www.greenend.org.uk/rjk/

On 2012-01-31, The Natural Philosopher <tnp@invalid.invalid> wrote:
>...
>
> Maybe the drivers were buffering everything...not the kernel.
> I had that problem with a RAID once. you could even sync, but it was no 
> guarantee the data was on the disks. The RAID sub sytem reported 'all 
> written' and that was that. Except it wasn't. it was in the RAID system 
> buffers.

The 'Dirty' line in /proc/meminfo said the pages were still in
dirty state.  A manual sync did clear them out.

-- 
Robert Riches
spamtrap42@jacob21819.net
(Yes, that is one of my email addresses.)

On 2012-01-31, Richard Kettlewell <rjk@greenend.org.uk> wrote:
> Robert Riches <spamtrap42@jacob21819.net> writes:
>> Why is my kernel (Mageia 1's 2.6.38.8-server-10.mga) _NOT_
>> writing out dirty blocks to disk?
>>
>> Yesterday, the contents of a number of files were lost when I
>> crashed the machine in a failed experiment with SCSI rescan to
>> remove a hot-unplugged ESATA disk.  (Rebuilding the software
>> RAID10s is another story...)  Today, recompiling WINE left 860MB
>> of dirty blocks needing to be written to disk, according to
>> /proc/meminfo.  After the compilation finished, I waited over 90
>> seconds, with the system essentially completely idle, and the
>> amount of dirty data did not go down.  Then, when I did a 'sync'
>> (as a non-root user), immediately everything flushed out to disk
>> in about 10-15 seconds time and the amount of dirty data went to
>> zero.  (I had "vmstat 3" and xcpustate running, along with a
>> periodic manual "grep Dirty /proc/meminfo".)
>>
>> There is a value of 3000 in /proc/sys/vm/dirty_expire_centisecs,
>> which accoring to this page (and others I have read)
>>
>>     http://www.westnet.com/~gsmith/content/linux-pdflush.htm
>
> pdflush no longer exists, and dirty_expire_centisecs is no longer used.
> Which doesn't answer your question as such but does explain why the old
> documentation isn't helping.
>
> As a starting point for the new logic:
>
> http://kernelnewbies.org/Linux_2_6_32#head-72c3f91947738f1ea52f9ed21a89876730418a61

Thanks.  I hadn't heard solid reporting of the demise of pdflush,
but I had noticed some changes in the names of the kernel worker
or helper processes (but wrote it off as a result of going to my
first SMP machine).  I bookmarked that page for study.

-- 
Robert Riches
spamtrap42@jacob21819.net
(Yes, that is one of my email addresses.)