Hi,

we have the following problem. Our oracle-admin changed something in
his oracle-database which resulted in a change of the filesystemsize.
In detail he has created a tablespace and dropped another tablespace on
the same filesystem. Now we have the effect, that with a "df -k" we get
the wrong filesystemsize. With a "du -sk" on the filesystem, we get the
right size.

The system is installed with Solaris 9 SPARC. It is a UFS-filesystem
with logging enabled. The filesystem resides on a LUN of a EMC
Symmetrix DMX 800, shared over two paths with EMC Powerpath.

Any idea?

THX!!!

WolverineJR wrote:
> Hi,
>
> we have the following problem. Our oracle-admin changed something in
> his oracle-database which resulted in a change of the filesystemsize.
> In detail he has created a tablespace and dropped another tablespace on
> the same filesystem. Now we have the effect, that with a "df -k" we get
> the wrong filesystemsize. With a "du -sk" on the filesystem, we get the
> right size.
>
> The system is installed with Solaris 9 SPARC. It is a UFS-filesystem
> with logging enabled. The filesystem resides on a LUN of a EMC
> Symmetrix DMX 800, shared over two paths with EMC Powerpath.
>
> Any idea?
>
> THX!!!

Daniel Rock posted the reason for this under the thread 'Missing disk
space' on september 20.

If I might paraphrase, basically if you delete an open file, df still
reports the file space in use.

Excerpt from his post: "If you have
deleted large logfiles by accident which are still open you can find
them
with searching in /proc:

find /proc -links 0 -type f -size +XXXXc -ls

will print you out any unlinked file with a size larger than XXXX
bytes."

My thought as to why df exhibits this behaviour is that it reports file
handles (pointers), and the last known file size associated with those
handles. If an open file is deleted, the data goes but the handle
locked by the application and will continue to be reported along with
it's last known
size.

Daniel, if this is incorrect please interject.

HTH

Dean

On 27 Sep 2006 06:27:48 -0700
"slackware guy" <dgbrown1@cox.net> wrote:

> My thought as to why df exhibits this behaviour is that it reports
> file handles (pointers), and the last known file size associated with
> those handles. If an open file is deleted, the data goes but the
> handle locked by the application and will continue to be reported
> along with it's last known size.

Actually, the file only disappears when it is closed. This property of
Unix files is used to create "self-destructing" temporary files (simply
unlink the file whilst keeping it open). The reason is that in Unix a
file name is merely a pointer to the file, which explains why one can
have several "hard links", or different file names, for the same file.
The "real" file is defined by the inode, and as long as there is a link
to an inode, or the file is open, it is not deleted from the file
system.

Consider:

$ echo "I am a file..." > thefile
$ ln thefile foo
$ rm thefile
$ cat foo
I am a file...
$

Note that this makes it possible to upgrade a shared object or program
while it is in use. Windows file systems don't have this feature, and
files cannot be deleted or renamed while in use (which is, if I may be
so candid, a bloody nuisance).

Take care,

-- 
Stefaan A Eeckels
-- 
Governments are like babies: digestive tracts with a big appetite at
one end and no sense of responsibility at the other. The better run
ones from time to time get clean diapers...   

Stefaan A Eeckels wrote:
> On 27 Sep 2006 06:27:48 -0700
> "slackware guy" <dgbrown1@cox.net> wrote:
>
> > My thought as to why df exhibits this behaviour is that it reports
> > file handles (pointers), and the last known file size associated with
> > those handles. If an open file is deleted, the data goes but the
> > handle locked by the application and will continue to be reported
> > along with it's last known size.
>
> Actually, the file only disappears when it is closed. This property of
> Unix files is used to create "self-destructing" temporary files (simply
> unlink the file whilst keeping it open). The reason is that in Unix a
> file name is merely a pointer to the file, which explains why one can
> have several "hard links", or different file names, for the same file.
> The "real" file is defined by the inode, and as long as there is a link
> to an inode, or the file is open, it is not deleted from the file
> system.
>
Why would du then report what I would think is the 'correct' result of
the formerly used disk
space now being available?

> Consider:
>
> $ echo "I am a file..." > thefile
> $ ln thefile foo
> $ rm thefile
> $ cat foo
> I am a file...
> $
>
> Note that this makes it possible to upgrade a shared object or program
> while it is in use. Windows file systems don't have this feature, and
> files cannot be deleted or renamed while in use (which is, if I may be
> so candid, a bloody nuisance).
>
> Take care,
>
> --
> Stefaan A Eeckels
> --
> Governments are like babies: digestive tracts with a big appetite at
> one end and no sense of responsibility at the other. The better run
> ones from time to time get clean diapers...

On 27 Sep 2006 07:25:17 -0700
"slackware guy" <dgbrown1@cox.net> wrote:

> Why would du then report what I would think is the 'correct' result of
> the formerly used disk space now being available?

"du" walks the directory tree and counts the blocks that are allocated
to the files.

When the file is unlinked, "du" can no longer find its inode, and
hence cannot include the blocks it uses in its tally.

"df", which uses information in the file system headers, correctly
shows that the blocks are still in use.

Take care,

-- 
Stefaan A Eeckels
-- 
I can understand that some people are cheap. I can't accept that 
they also appear deeply hurt when someone else makes money. 
-- Artur Swietanowski in c.o.l.x

slackware guy wrote:

> Why would du then report what I would think is the 'correct' result of
> the formerly used disk
> space now being available?

Because all du does is walk over the directory structure counting file
sizes.  If a file has no remaining directory entries, it never finds
it.

--tim

slackware guy <dgbrown1@cox.net> wrote:
> My thought as to why df exhibits this behaviour is that it reports file
> handles (pointers), and the last known file size associated with those
> handles. If an open file is deleted, the data goes but the handle
> locked by the application and will continue to be reported along with
> it's last known size.

No, you have to get rid of the simple connection file <-> filename.
A file is identified by an inode. A directory entry is just a pointer to
a file. A file can have 0 to many pointers to itself (called hard links).

A file gets deleted (free data blocks freed, inode released) if the link
count drops to 0 *AND* the file isn't opened by any process.

So df reports the right size: The file is still there (but cannot be
referenced by any file name[1]). Therefor the space of this file cannot be
accounted to the free space. Worse, the file might even be growing,
eventually filling up the filesystem (a common mistake by UNIX newbies
who just delete large log files and wonder a few days later why the
filesystem has filled up).

df: free/allocated space of the entire filesystem
du: allocated space of files in a given directory tree

Since there is no directory entry associated with this file, du cannot
find the file and the output of du will be different than the one from df.



[1] ignoring /proc

Tim Bradshaw wrote:
> slackware guy wrote:
>
> > Why would du then report what I would think is the 'correct' result of
> > the formerly used disk
> > space now being available?
>
> Because all du does is walk over the directory structure counting file
> sizes.  If a file has no remaining directory entries, it never finds
> it.
>
> --tim

Thanks guys! Major thought process shift in progress........

Dean

slackware guy <dgbrown1@cox.net> wrote:
> Why would du then report what I would think is the 'correct' result of
> the formerly used disk space now being available?

The disk space isn't "formerly" used but still in use. So it isn't yet
available (only after the file has been closed).
This is a common conception in UNIX to create temporary files where you
don't have to worry to clean up your files after use. See man tmpfile(3C):

http://www.opengroup.org/onlinepubs/009695399/functions/tmpfile.html

and a possible implementation:

http://cvs.opensolaris.org/source/xref/on/usr/src/lib/libc/port/stdio/tmpfile.c

(function _common()): see the unlink() call directly after the mkstemp()
call.

-- 
Daniel

Hi!

Thank you for your detailed explanations!
So that I understand it correctly, the seizure of the filesystem is
correct. But because Oracle still keeps the files open or has still
some locks on the files/links, "df" show the wrong size. So the next
time, the DB will be restarted, the filesystem-size in "df" is correct?

Hi!

Thank you for your detailed explanations!
For my understanding, the seizure of the filesystem is correct, but
"df" still shows me the wrong size, because oracle still keeps the
files open or has some locks on the files. So next time the db will be
restarted, the size of "df" will be correct?

THX!!

Hi!

Thank you for your detailed explanations!
For my understanding, the seizure of the filesystem is correct, but
"df" still shows me the wrong size, because oracle still keeps the
files open or has some locks on the files. So next time the db will be
restarted, the size of "df" will be correct?

THX!!

On 27 Sep 2006 22:13:22 -0700
"WolverineJR" <jreitwiessner@samson.de> wrote:

> So that I understand it correctly, the seizure of the filesystem is
> correct. But because Oracle still keeps the files open or has still
> some locks on the files/links, "df" show the wrong size. So the next
> time, the DB will be restarted, the filesystem-size in "df" is
> correct?

"df" shows the correct information. It is "du" that cannot show files
that are open, but no longer referenced in any directory. The file
system "size" of course is a relatively stable attribute (you can make
a UFS file system larger through "growfs", but it is unlikely that your
DBA would have been able to change the size of the file system):

$ df -h .
Filesystem             size   used  avail capacity  Mounted on
/dev/dsk/c1t1d0s7       46G    27G    19G    59%    /export/home

du in any case cannot give information on the size of a file system,
only on the space used by the files. 

If you care to give more detailed information on what you observed,
instead of "right size" and "wrong size", it would be possible to tell
you exactly what the reason is for your observations. Please show the
commands you used and the output they produced. 

Take care,

-- 
Stefaan A Eeckels
-- 
Ninety-Ninety Rule of Project Schedules:
        The first ninety percent of the task takes ninety percent of
the time, and the last ten percent takes the other ninety percent.

WolverineJR wrote:
> Hi!
>
> Thank you for your detailed explanations!
> So that I understand it correctly, the seizure of the filesystem is
> correct. But because Oracle still keeps the files open or has still
> some locks on the files/links, "df" show the wrong size. So the next
> time, the DB will be restarted, the filesystem-size in "df" is correct?

If I've finally got right. The file was never deleted, the directory
entry (pointer) was deleted.
The inode exists as long as an application has it open whether it has a
directory entry or not.

df always the correct usage, du gives the file sizes associated with
the directory tree. So you can
have an inode in use with no name (because an rm was executed while it
was open).

The application (I assume in this case oracle) needs to be stopped. the
inode associated will
the deleted filename will be closed and the data removed and then the
results of df and du will
coincide.

Dean

slackware guy <dgbrown1@cox.net> wrote:
> The application (I assume in this case oracle) needs to be stopped. the
> inode associated will
> the deleted filename will be closed and the data removed and then the
> results of df and du will
> coincide.

Normally the database closes its files automatically if you destroy the
tablespace. Maybe the Oracle administrator has deleted the wrong files?
If this were the case Oracle would be very unhappy on shutdown/startup.
You can check which files are open by the database by running pfiles
on the pid of the ora_dbwN_SID process.

The DBA should verify all database files still in use are on the location
they should be:

	select file_name, tablespace_name, status from dba_data_files;

If the wrong files have been deleted, not everything is lost, as long the
database is still running. The DBA should put the tablespace in read only
mode and then copy the datafiles back from /proc (unfortunately linking
back files from /proc (via ln or link) doesn't work). He then should take
the tablespace offline and back online.

-- 
Daniel

On 2006-09-28 15:00:10 +0100, "slackware guy" <dgbrown1@cox.net> said:

> If I've finally got right. The file was never deleted, the directory
> entry (pointer) was deleted.

Precisely.  And in fact the system call that does this has an 
indicative name:  unlink(2): what it's doing is removing a name for a 
file.  There *is* no way to delete a file in Unix really (at the user 
level): you just remove links to its inode until there are none left, 
at which point the filesystem will (or may) reclaim the space, *if* 
there are no other references to the inode, for instance from running 
programs.

--tim