Major File System Problems! Help

  • Follow


I am getting a lot of "file system" full error messages. But the / file 
system is not full from "df -k".

SunOS ff1 5.8 Generic_108528-29 sun4u sparc SUNW,Ultra-1  with Dsksuite 
or Solaris Volume Manager and mirroring.

Could this be a an issue with the number of files I that are on the 
system? What is the limit on the number of files that the system can 
support?

/dev/md/dsk/d10      16634162 13906615 2561206    85%    /
/proc                      0       0       0     0%    /proc
fd                         0       0       0     0%    /dev/fd
mnttab                     0       0       0     0%    /etc/mnttab
swap                  605848      24  605824     1%    /var/run
swap                  605840      16  605824     1%    /tmp


Here is the dmesg log:

Dec 19 03:10:40 ff1 ufs: [ID 845546 kern.notice] NOTICE: alloc: /: file 
system full
Dec 19 03:16:23 ff1 last message repeated 5 times
Dec 19 03:20:00 ff1 ufs: [ID 845546 kern.notice] NOTICE: alloc: /: file 
system full
Dec 19 03:21:23 ff1 last message repeated 1 time
Dec 19 03:23:05 ff1 ufs: [ID 213553 kern.notice] NOTICE: realloccg /: 
file system full
Dec 19 03:23:05 ff1 sendmail[27074]: [ID 801593 mail.crit] i99BA1f00930: 
SYSERR(root): Can't create transcript file ./xfi99BA1f00930: No space 
left on device
Dec 19 03:23:05 ff1 sendmail[27074]: [ID 801593 mail.crit] i99BA1f00930: 
SYSERR(root): Can't create transcript file ./xfiBJ8N5c27074: No space 
left on device
Dec 19 03:23:05 ff1 syslogd: /var/log/syslog: No space left on device
Dec 19 03:23:05 ff1 sendmail[27074]: [ID 801593 mail.alert] 
iBJ8N5c27074: SYSERR(root): queueup: cannot create queue temp file 
../tfiBJ8N5c27074, uid=0: No space left on device
Dec 19 03:26:23 ff1 ufs: [ID 845546 kern.notice] NOTICE: alloc: /: file 
system full
Dec 26 03:10:41 ff1 ufs: [ID 845546 kern.notice] NOTICE: alloc: /: file 
system full
Dec 26 03:15:00 ff1 last message repeated 4 times
Dec 26 03:18:26 ff1 ufs: [ID 845546 kern.notice] NOTICE: alloc: /: file 
system full
Dec 26 03:23:26 ff1 last message repeated 2 times
Dec 26 03:23:51 ff1 ufs: [ID 213553 kern.notice] NOTICE: realloccg /: 
file system full
Dec 26 03:23:51 ff1 sendmail[10751]: [ID 801593 mail.crit] i99BA1f00930: 
SYSERR(root): Can't create transcript file ./xfi99BA1f00930: No space 
left on device
Dec 26 03:23:51 ff1 sendmail[10751]: [ID 801593 mail.crit] i99BA1f00930: 
SYSERR(root): Can't create transcript file ./xfiBQ8Npc10751: No space 
left on device
Dec 26 03:23:51 ff1 syslogd: /var/log/syslog: No space left on device
Dec 26 03:23:51 ff1 sendmail[10751]: [ID 801593 mail.alert] 
iBQ8Npc10751: SYSERR(root): queueup: cannot create queue temp file 
../tfiBQ8Npc10751, uid=0: No space left on device
Dec 26 03:28:26 ff1 ufs: [ID 845546 kern.notice] NOTICE: alloc: /: file 
system full
Jan  2 03:10:40 ff1 ufs: [ID 845546 kern.notice] NOTICE: alloc: /: file 
system full
Jan  2 03:11:32 ff1 last message repeated 5 times
Jan  2 03:12:22 ff1 ufs: [ID 845546 kern.notice] NOTICE: alloc: /: file 
system full
Jan  2 03:17:09 ff1 last message repeated 9 times
Jan  2 03:18:53 ff1 ufs: [ID 845546 kern.notice] NOTICE: alloc: /: file 
system full
Jan  2 03:22:40 ff1 last message repeated 8 times
Jan  2 03:24:36 ff1 ufs: [ID 213553 kern.notice] NOTICE: realloccg /: 
file system full
Jan  2 03:24:36 ff1 sendmail[7707]: [ID 801593 mail.crit] i99BA1f00930: 
SYSERR(root): Can't create transcript file ./xfi99BA1f00930: No space 
left on device
Jan  2 03:24:36 ff1 sendmail[7707]: [ID 801593 mail.crit] i99BA1f00930: 
SYSERR(root): Can't create transcript file ./xfj028Oac07707: No space 
left on device
Jan  2 03:24:36 ff1 syslogd: /var/log/syslog: No space left on device
Jan  9 03:10:41 ff1 ufs: [ID 845546 kern.notice] NOTICE: alloc: /: file 
system full
Jan  9 03:12:04 ff1 last message repeated 2 times
Jan  9 03:13:00 ff1 ufs: [ID 845546 kern.notice] NOTICE: alloc: /: file 
system full
Jan  9 03:18:16 ff1 last message repeated 9 times
Jan  9 03:20:00 ff1 ufs: [ID 845546 kern.notice] NOTICE: alloc: /: file 
system full
Jan  9 03:23:09 ff1 last message repeated 5 times
Jan  9 03:25:19 ff1 ufs: [ID 213553 kern.notice] NOTICE: realloccg /: 
file system full
Jan  9 03:25:19 ff1 sendmail[25106]: [ID 801593 mail.crit] i99BA1f00930: 
SYSERR(root): Can't create transcript file ./xfi99BA1f00930: No space 
left on device
Jan  9 03:25:19 ff1 sendmail[25106]: [ID 801593 mail.crit] i99BA1f00930: 
SYSERR(root): Can't create transcript file ./xfj098PJc25106: No space 
left on device
Jan  9 03:25:19 ff1 syslogd: /var/log/syslog: No space left on device
Jan 16 03:10:41 ff1 ufs: [ID 845546 kern.notice] NOTICE: alloc: /: file 
system full
Jan 16 03:11:02 ff1 ufs: [ID 213553 kern.notice] NOTICE: realloccg /: 
file system full
Jan 16 03:11:02 ff1 sendmail[11118]: [ID 801593 mail.crit] i99BA1f00930: 
SYSERR(root): Can't create transcript file ./xfi99BA1f00930: No space 
left on device
Jan 16 03:11:02 ff1 sendmail[11118]: [ID 801593 mail.crit] i99BA1f00930: 
SYSERR(root): Can't create transcript file ./xfj0G8B2c11118: No space 
left on device
Jan 16 03:11:02 ff1 syslogd: /var/log/syslog: No space left on device
Jan 16 03:11:02 ff1 sendmail[11118]: [ID 801593 mail.alert] 
j0G8B2c11118: SYSERR(root): queueup: cannot create queue temp file 
../tfj0G8B2c11118, uid=0: No space left on device
Jan 16 03:13:00 ff1 ufs: [ID 845546 kern.notice] NOTICE: alloc: /: file 
system full
Jan 16 03:15:00 ff1 last message repeated 2 times
Jan 16 03:20:00 ff1 ufs: [ID 845546 kern.notice] NOTICE: alloc: /: file 
system full
Jan 16 03:20:04 ff1 last message repeated 1 time
0
Reply Deke (41) 1/23/2005 2:42:13 AM

Deke <Deke@nospam.com> writes:
>I am getting a lot of "file system" full error messages. But the / file 
>system is not full from "df -k".
>Could this be a an issue with the number of files I that are on the 
>system? What is the limit on the number of files that the system can 
>support?

df -g /

look for total files and free files.  

Later

Mark Hittinger
bugs@pu.net
0
Reply bugs 1/23/2005 3:08:07 AM


Not sure hot to read this but here it is :

xxff1:23:41:/> df -g /
/                  (/dev/md/dsk/d10   ):         8192 block size 
   1024 frag size
33268324 total blocks    5455092 free blocks  5122410 available 
2061056 total files
   318802 free files     22282250 filesys id
      ufs fstype       0x00000004 flag             255 filename length



Mark Hittinger wrote:
> Deke <Deke@nospam.com> writes:
> 
>>I am getting a lot of "file system" full error messages. But the / file 
>>system is not full from "df -k".
>>Could this be a an issue with the number of files I that are on the 
>>system? What is the limit on the number of files that the system can 
>>support?
> 
> 
> df -g /
> 
> look for total files and free files.  
> 
> Later
> 
> Mark Hittinger
> bugs@pu.net
0
Reply Deke 1/23/2005 4:43:01 AM

Also check for free inodes (df -i) and finally make sure your
filesystem is healthy/fsck'd.

0
Reply Filippos 1/23/2005 5:19:26 AM

On Sat, 22 Jan 2005 21:19:26 -0800, Filippos C wrote:

> Also check for free inodes (df -i) and finally make sure your
> filesystem is healthy/fsck'd.

Before I post a suggested command and arguments I try them on my own
machine.

0
Reply Dave 1/23/2005 5:32:08 AM

If you mean

xxff1:1:32:/> df -e
Filesystem              ifree
/dev/md/dsk/d10        318802  << "/"
/proc                    3852
fd                          0
mnttab                      0
swap                    26114
swap                    26114


Also the filesystem uses logging and is clean.  Thanks
Filippos C wrote:
> Also check for free inodes (df -i) and finally make sure your
> filesystem is healthy/fsck'd.
> 
0
Reply Deke 1/23/2005 6:34:07 AM

Deke wrote:
> If you mean
> 
> xxff1:1:32:/> df -e
> Filesystem              ifree
> /dev/md/dsk/d10        318802  << "/"
> /proc                    3852
> fd                          0
> mnttab                      0
> swap                    26114
> swap                    26114
> 
> 
> Also the filesystem uses logging and is clean.  Thanks
> Filippos C wrote:
> 
>> Also check for free inodes (df -i) and finally make sure your
>> filesystem is healthy/fsck'd.
>>

You may also want to check out SRDB 74799 at 
http://sunsolve.sun.com/search/document.do?assetkey=1-25-74799-1 to see 
whether or not ou are sufferring from a fragmentation problem.

alan (who wrote that doc).
-- 
Alan Hargreaves - http://blogs.sun.com/tpenta
Kernel/VOSJEC/Performance Engineer
Product Technical Support (APAC)
Sun Microsystems
0
Reply Alan 1/24/2005 2:53:52 AM

Deke,

Sometimes this happens if a process has a filehandle open and is
writing to the file and the file is removed by a user. I've seen this
happen with log files and database files. Use "fsck -N" to find
unallocated inodes and lsof to locate the inode in the filesystem. You
can get lsof from sunfreeware.com.

Good Luck,
Vic Engle
vic@summerseas.com



Deke wrote:
> I am getting a lot of "file system" full error messages. But the /
file
> system is not full from "df -k".
>
> SunOS ff1 5.8 Generic_108528-29 sun4u sparc SUNW,Ultra-1  with
Dsksuite
> or Solaris Volume Manager and mirroring.
>
> Could this be a an issue with the number of files I that are on the
> system? What is the limit on the number of files that the system can
> support?
>
> /dev/md/dsk/d10      16634162 13906615 2561206    85%    /
> /proc                      0       0       0     0%    /proc
> fd                         0       0       0     0%    /dev/fd
> mnttab                     0       0       0     0%    /etc/mnttab
> swap                  605848      24  605824     1%    /var/run
> swap                  605840      16  605824     1%    /tmp
>
>
> Here is the dmesg log:
>
> Dec 19 03:10:40 ff1 ufs: [ID 845546 kern.notice] NOTICE: alloc: /:
file
> system full
> Dec 19 03:16:23 ff1 last message repeated 5 times
> Dec 19 03:20:00 ff1 ufs: [ID 845546 kern.notice] NOTICE: alloc: /:
file
> system full
> Dec 19 03:21:23 ff1 last message repeated 1 time
> Dec 19 03:23:05 ff1 ufs: [ID 213553 kern.notice] NOTICE: realloccg /:

> file system full
> Dec 19 03:23:05 ff1 sendmail[27074]: [ID 801593 mail.crit]
i99BA1f00930:
> SYSERR(root): Can't create transcript file ./xfi99BA1f00930: No space

> left on device
> Dec 19 03:23:05 ff1 sendmail[27074]: [ID 801593 mail.crit]
i99BA1f00930:
> SYSERR(root): Can't create transcript file ./xfiBJ8N5c27074: No space

> left on device
> Dec 19 03:23:05 ff1 syslogd: /var/log/syslog: No space left on device
> Dec 19 03:23:05 ff1 sendmail[27074]: [ID 801593 mail.alert]
> iBJ8N5c27074: SYSERR(root): queueup: cannot create queue temp file
> ./tfiBJ8N5c27074, uid=0: No space left on device
> Dec 19 03:26:23 ff1 ufs: [ID 845546 kern.notice] NOTICE: alloc: /:
file
> system full
> Dec 26 03:10:41 ff1 ufs: [ID 845546 kern.notice] NOTICE: alloc: /:
file
> system full
> Dec 26 03:15:00 ff1 last message repeated 4 times
> Dec 26 03:18:26 ff1 ufs: [ID 845546 kern.notice] NOTICE: alloc: /:
file
> system full
> Dec 26 03:23:26 ff1 last message repeated 2 times
> Dec 26 03:23:51 ff1 ufs: [ID 213553 kern.notice] NOTICE: realloccg /:

> file system full
> Dec 26 03:23:51 ff1 sendmail[10751]: [ID 801593 mail.crit]
i99BA1f00930:
> SYSERR(root): Can't create transcript file ./xfi99BA1f00930: No space

> left on device
> Dec 26 03:23:51 ff1 sendmail[10751]: [ID 801593 mail.crit]
i99BA1f00930:
> SYSERR(root): Can't create transcript file ./xfiBQ8Npc10751: No space

> left on device
> Dec 26 03:23:51 ff1 syslogd: /var/log/syslog: No space left on device
> Dec 26 03:23:51 ff1 sendmail[10751]: [ID 801593 mail.alert]
> iBQ8Npc10751: SYSERR(root): queueup: cannot create queue temp file
> ./tfiBQ8Npc10751, uid=0: No space left on device
> Dec 26 03:28:26 ff1 ufs: [ID 845546 kern.notice] NOTICE: alloc: /:
file
> system full
> Jan  2 03:10:40 ff1 ufs: [ID 845546 kern.notice] NOTICE: alloc: /:
file
> system full
> Jan  2 03:11:32 ff1 last message repeated 5 times
> Jan  2 03:12:22 ff1 ufs: [ID 845546 kern.notice] NOTICE: alloc: /:
file
> system full
> Jan  2 03:17:09 ff1 last message repeated 9 times
> Jan  2 03:18:53 ff1 ufs: [ID 845546 kern.notice] NOTICE: alloc: /:
file
> system full
> Jan  2 03:22:40 ff1 last message repeated 8 times
> Jan  2 03:24:36 ff1 ufs: [ID 213553 kern.notice] NOTICE: realloccg /:

> file system full
> Jan  2 03:24:36 ff1 sendmail[7707]: [ID 801593 mail.crit]
i99BA1f00930:
> SYSERR(root): Can't create transcript file ./xfi99BA1f00930: No space

> left on device
> Jan  2 03:24:36 ff1 sendmail[7707]: [ID 801593 mail.crit]
i99BA1f00930:
> SYSERR(root): Can't create transcript file ./xfj028Oac07707: No space

> left on device
> Jan  2 03:24:36 ff1 syslogd: /var/log/syslog: No space left on device
> Jan  9 03:10:41 ff1 ufs: [ID 845546 kern.notice] NOTICE: alloc: /:
file
> system full
> Jan  9 03:12:04 ff1 last message repeated 2 times
> Jan  9 03:13:00 ff1 ufs: [ID 845546 kern.notice] NOTICE: alloc: /:
file
> system full
> Jan  9 03:18:16 ff1 last message repeated 9 times
> Jan  9 03:20:00 ff1 ufs: [ID 845546 kern.notice] NOTICE: alloc: /:
file
> system full
> Jan  9 03:23:09 ff1 last message repeated 5 times
> Jan  9 03:25:19 ff1 ufs: [ID 213553 kern.notice] NOTICE: realloccg /:

> file system full
> Jan  9 03:25:19 ff1 sendmail[25106]: [ID 801593 mail.crit]
i99BA1f00930:
> SYSERR(root): Can't create transcript file ./xfi99BA1f00930: No space

> left on device
> Jan  9 03:25:19 ff1 sendmail[25106]: [ID 801593 mail.crit]
i99BA1f00930:
> SYSERR(root): Can't create transcript file ./xfj098PJc25106: No space

> left on device
> Jan  9 03:25:19 ff1 syslogd: /var/log/syslog: No space left on device
> Jan 16 03:10:41 ff1 ufs: [ID 845546 kern.notice] NOTICE: alloc: /:
file
> system full
> Jan 16 03:11:02 ff1 ufs: [ID 213553 kern.notice] NOTICE: realloccg /:

> file system full
> Jan 16 03:11:02 ff1 sendmail[11118]: [ID 801593 mail.crit]
i99BA1f00930:
> SYSERR(root): Can't create transcript file ./xfi99BA1f00930: No space

> left on device
> Jan 16 03:11:02 ff1 sendmail[11118]: [ID 801593 mail.crit]
i99BA1f00930:
> SYSERR(root): Can't create transcript file ./xfj0G8B2c11118: No space

> left on device
> Jan 16 03:11:02 ff1 syslogd: /var/log/syslog: No space left on device
> Jan 16 03:11:02 ff1 sendmail[11118]: [ID 801593 mail.alert]
> j0G8B2c11118: SYSERR(root): queueup: cannot create queue temp file
> ./tfj0G8B2c11118, uid=0: No space left on device
> Jan 16 03:13:00 ff1 ufs: [ID 845546 kern.notice] NOTICE: alloc: /:
file
> system full
> Jan 16 03:15:00 ff1 last message repeated 2 times
> Jan 16 03:20:00 ff1 ufs: [ID 845546 kern.notice] NOTICE: alloc: /:
file 
> system full
> Jan 16 03:20:04 ff1 last message repeated 1 time

0
Reply vic 1/24/2005 3:04:28 PM

In article <pan.2005.01.23.05.32.08.279832@yahoo.com>,
Dave Uhring  <daveuhring@yahoo.com> wrote:
>On Sat, 22 Jan 2005 21:19:26 -0800, Filippos C wrote:
>
>> Also check for free inodes (df -i) and finally make sure your
>> filesystem is healthy/fsck'd.
>
>Before I post a suggested command and arguments I try them on my own
>machine.

Strange that '-i' is not in the Solaris df.  Try /usr/ucb/df -i
The output is somewhat easier to understand than  df -g.
-- 
Tom Schulz
schulz@adi.com
0
Reply schulz 1/24/2005 5:59:00 PM

On Mon, 24 Jan 2005 17:59:00 +0000, Thomas Schulz wrote:

> In article <pan.2005.01.23.05.32.08.279832@yahoo.com>,
> Dave Uhring  <daveuhring@yahoo.com> wrote:

>>Before I post a suggested command and arguments I try them on my own
>>machine.
> 
> Strange that '-i' is not in the Solaris df.  Try /usr/ucb/df -i
> The output is somewhat easier to understand than  df -g.

df(1M)  s10_b72 and Solaris 9:

     The following example displays inode usage on all  ufs  file
     systems:

     example%/usr/bin/df -F ufs -o i

0
Reply Dave 1/24/2005 6:26:04 PM

"Alan Hargreaves - Product Technical Support (APAC)" wrote:
> 
> Deke wrote:
> > If you mean
> >
> > xxff1:1:32:/> df -e
> > Filesystem              ifree
> > /dev/md/dsk/d10        318802  << "/"
> > /proc                    3852
> > fd                          0
> > mnttab                      0
> > swap                    26114
> > swap                    26114
> >
> >
> > Also the filesystem uses logging and is clean.  Thanks
> > Filippos C wrote:
> >
> >> Also check for free inodes (df -i) and finally make sure your
> >> filesystem is healthy/fsck'd.
> >>
> 
> You may also want to check out SRDB 74799 at
> http://sunsolve.sun.com/search/document.do?assetkey=1-25-74799-1 to see
> whether or not ou are sufferring from a fragmentation problem.
> 
> alan (who wrote that doc).

srdb 74799 is slightly inaccurate these days, we changed the way how the
cylinder group summary information is recorded and updated in s9 thus 
the described view of it doesn no longer apply in any case when logging
is enabled.

As for the original post and how to determine if the filesystem got
fragmented enough to cause those allocation failures, check fstyp(1M) -v
output and look at the following values in the suberblock:

nbfree  0       ndir    15327   nifree  368874  nffree  212694

nbfree = number of full sized blocks free
nffree = number of fragments free

if the number of available full sized blocks is 0, but you still have
fragments (nffree) available, the available space df(1M) does show
is scattered throughout fragments and you've hit the fragmentation problem.

If you don't like the fstyp(1M) routine, you can use fsck(1M) instead.
At the very end, it reports (example):
 
35834 files, 467 used, 8691007 free (119 frags, 1086361 blocks, 0.0%
fragmentation)
 
In order, those are the number of:
- inodes in use
- full-sized blocks in use
- fragments that are free
- fragments that are free but can not be part of a block due to
  fragmentation (aka isolated)
- full-size blocks that are free
- isolated fragments / total number of non-meta-data fragments 
 
Note that fragments free should always equal isolated fragments plus
(fs_frag times full-size free blocks).  fs_frag is the number of fragments 
in a block, you get this again from fstyp(1M) -v and field "frag".

PS:

Alan, if you want to update that document, drop me a note and I'll 
provide you with the details.

---
frankB
0
Reply Frank 2/5/2005 1:10:08 PM

10 Replies
614 Views

(page loaded in 0.196 seconds)

Similiar Articles:













7/24/2012 4:58:55 AM


Reply: