Hello,

What is the maximum number of files allowed in a directory under UFS
in Solaris 10 ?

What is the maximum number of files allowed in a directory under
Veritas File System 5.0 in Solaris 10 ?

I'd like to know where to locate this maximum number under UFS and
Veritas File System 5.0 in Solaris 10.  Is there any way that we could
modify this maximum number ?

Thanks,

Bill

On 05/19/10 10:10 AM, underh20 wrote:
> Hello,
>
> What is the maximum number of files allowed in a directory under UFS
> in Solaris 10 ?

It depends.

> What is the maximum number of files allowed in a directory under
> Veritas File System 5.0 in Solaris 10 ?

Pass.

> I'd like to know where to locate this maximum number under UFS and
> Veritas File System 5.0 in Solaris 10.  Is there any way that we could
> modify this maximum number ?

df -o i reports the inode data for a UFS filesystem.

To increase the number, you have to recreate the filesystem.

See df_ufs(1M) and newfs(1M).

-- 
Ian Collins

On 18-May-2010, underh20 <underh20.scubadiving@gmail.com> wrote:

> What is the maximum number of files allowed in a directory under UFS
> in Solaris 10 ?

It depends on the size of the file system as UFS uses inodes .. the bigger
the file system, the larger the number of inodes available. You can check on
the inode usage by using "df -F ufs -o i"

I dont think ZFS has a limit on the number of files in a directory... the
limit for UFS is *quite* large .. it's in *at least* the 10's of thousands
.... I know Ive had 20k files in a single directory ..

Not sure about thet other's Im afraid... but ZFS is definitely good to go :D

Cya

underh20 wrote:
> Hello,
> 
> What is the maximum number of files allowed in a directory under UFS
> in Solaris 10 ?
> 
> What is the maximum number of files allowed in a directory under
> Veritas File System 5.0 in Solaris 10 ?
> 
> I'd like to know where to locate this maximum number under UFS and
> Veritas File System 5.0 in Solaris 10.  Is there any way that we could
> modify this maximum number ?
> 
> Thanks,
> 
> Bill

If there is a limit, it's huge!  It's far more files than it would be 
reasonable to catalog in a single directory.

What problem are you trying to solve??

In article <_dydncBiFoRGr27WnZ2dnUVZ_h2dnZ2d@giganews.com>,
 "Richard B. Gilbert" <rgilbert88@comcast.net> wrote:

> underh20 wrote:
> > Hello,
> > 
> > What is the maximum number of files allowed in a directory under UFS
> > in Solaris 10 ?
> > 
> > What is the maximum number of files allowed in a directory under
> > Veritas File System 5.0 in Solaris 10 ?
> > 
> > I'd like to know where to locate this maximum number under UFS and
> > Veritas File System 5.0 in Solaris 10.  Is there any way that we could
> > modify this maximum number ?
> > 
> > Thanks,
> > 
> > Bill
> 
> If there is a limit, it's huge!  It's far more files than it would be 
> reasonable to catalog in a single directory.
> 
> What problem are you trying to solve??

No one in this thread has mentioned that it's a singularly Bad Idea(tm) 
to put a large number of files in a single directory.  At some point, 
there is a internal directory cache that gets filled and name lookups 
get progressively longer (I may have the details of this wrong).  I 
don't know if VxFS has this problem since, AFAIK, it doesn't use inodes 
and linearly searched directory files.

As a sysadmin, I had to clean up many lazy developer's "dirty" 
implementations of a project.  These are ones that leave lots of files 
in an application's defined temporary directory or much worse, the 
system's /tmp.  On Solaris, that sucks up swap space and memory for 
systems that stay up long periods of time using swap rather than real 
file system space.

Usually by the time I get involved the application has been running in 
production for a while and all of a sudden it starts slowing down, 
sucking memory up or taking forever to do certain things.  The memory is 
from /tmp filling up.  A simple rm can take for freak'in ever to go 
linearly through the directory to find the file's inode in the directory 
file (uses a linear search).  find works better.  But once the damage is 
cleaned up, a cron can be implemented to clean up after the lazy, evil 
developers.

Rather than put lots of files in a single directory, name them in a 
specific way such that you can put them in at least 2 levels of hashed 
subdirectories.  Netscape used to do this with the local cache.

What's the magic number where things fall to crap?  Dunno.  More than 
10,000 but probably less than 100,000.  You can do performance 
measurements with mail using a /var/mail with 10,000 mboxes and see if 
things get better or worse.  sendmail has to read /var/mail every time 
it delivers mail to a user's inbox.

All this is pretty old based on Solaris 7.  If things have improved in 
Solaris 10, I'm sure someone will jump in here and correct me.

-- 
DeeDee, don't press that button!  DeeDee!  NO!  Dee...
[I filter all Goggle Groups posts, so any reply may be automatically ignored]

On Tuesday 18 May 2010 21:09, Michael Vilain (vilain@NOspamcop.net) opined:

> In article <_dydncBiFoRGr27WnZ2dnUVZ_h2dnZ2d@giganews.com>,
>  "Richard B. Gilbert" <rgilbert88@comcast.net> wrote:
> 
>> underh20 wrote:
>> > Hello,
>> > 
>> > What is the maximum number of files allowed in a directory under UFS
>> > in Solaris 10 ?
>> > 
>> > What is the maximum number of files allowed in a directory under
>> > Veritas File System 5.0 in Solaris 10 ?
>> > 
>> > I'd like to know where to locate this maximum number under UFS and
>> > Veritas File System 5.0 in Solaris 10.  Is there any way that we could
>> > modify this maximum number ?
>> > 
>> > Thanks,
>> > 
>> > Bill
>> 
>> If there is a limit, it's huge!  It's far more files than it would be
>> reasonable to catalog in a single directory.
>> 
>> What problem are you trying to solve??
> 
> No one in this thread has mentioned that it's a singularly Bad Idea(tm)
> to put a large number of files in a single directory.  At some point,
> there is a internal directory cache that gets filled and name lookups
> get progressively longer (I may have the details of this wrong).  I
> don't know if VxFS has this problem since, AFAIK, it doesn't use inodes
> and linearly searched directory files.
> 
> As a sysadmin, I had to clean up many lazy developer's "dirty"
> implementations of a project.  These are ones that leave lots of files
> in an application's defined temporary directory or much worse, the
> system's /tmp.  On Solaris, that sucks up swap space and memory for
> systems that stay up long periods of time using swap rather than real
> file system space.
> 
> Usually by the time I get involved the application has been running in
> production for a while and all of a sudden it starts slowing down,
> sucking memory up or taking forever to do certain things.  The memory is
> from /tmp filling up.  A simple rm can take for freak'in ever to go
> linearly through the directory to find the file's inode in the directory
> file (uses a linear search).  find works better.  But once the damage is
> cleaned up, a cron can be implemented to clean up after the lazy, evil
> developers.
> 
> Rather than put lots of files in a single directory, name them in a
> specific way such that you can put them in at least 2 levels of hashed
> subdirectories.  Netscape used to do this with the local cache.
> 
> What's the magic number where things fall to crap?  Dunno.  More than
> 10,000 but probably less than 100,000.  You can do performance
> measurements with mail using a /var/mail with 10,000 mboxes and see if
> things get better or worse.  sendmail has to read /var/mail every time
> it delivers mail to a user's inbox.
> 
> All this is pretty old based on Solaris 7.  If things have improved in
> Solaris 10, I'm sure someone will jump in here and correct me.
> 
Yeah.  Just because you CAN do something doesn't mean you SHOULD do it.
Granted we have terabyte+ hard drives even on our home boxes these days,
that doesn't mean we necessarily should store giga- or tera-bytes of
whatever in a single directory.  Conventional wisdom is for multiple
directories, each containing a limited number of related files (once upon
a time this was 4096 files max, but today?  Who knows?).  This applies not
just to Solaris but to any reasonably administered system.

Bob Melson

-- 
Robert G. Melson | Rio Grande MicroSolutions | El Paso, Texas
-----
Nothing astonishes men so much as common sense and plain dealing.
Ralph Waldo Emerson

Michael Vilain wrote:
> In article <_dydncBiFoRGr27WnZ2dnUVZ_h2dnZ2d@giganews.com>,
>  "Richard B. Gilbert" <rgilbert88@comcast.net> wrote:
> 
>> underh20 wrote:
>>> Hello,
>>>
>>> What is the maximum number of files allowed in a directory under UFS
>>> in Solaris 10 ?
>>>
>>> What is the maximum number of files allowed in a directory under
>>> Veritas File System 5.0 in Solaris 10 ?
>>>
>>> I'd like to know where to locate this maximum number under UFS and
>>> Veritas File System 5.0 in Solaris 10.  Is there any way that we could
>>> modify this maximum number ?
>>>
>>> Thanks,
>>>
>>> Bill
>> If there is a limit, it's huge!  It's far more files than it would be 
>> reasonable to catalog in a single directory.
>>
>> What problem are you trying to solve??
> 
> No one in this thread has mentioned that it's a singularly Bad Idea(tm) 
> to put a large number of files in a single directory.  At some point, 
> there is a internal directory cache that gets filled and name lookups 
> get progressively longer (I may have the details of this wrong).  I 
> don't know if VxFS has this problem since, AFAIK, it doesn't use inodes 
> and linearly searched directory files.
> 
> As a sysadmin, I had to clean up many lazy developer's "dirty" 
> implementations of a project.  These are ones that leave lots of files 
> in an application's defined temporary directory or much worse, the 
> system's /tmp.  On Solaris, that sucks up swap space and memory for 
> systems that stay up long periods of time using swap rather than real 
> file system space.
> 
> Usually by the time I get involved the application has been running in 
> production for a while and all of a sudden it starts slowing down, 
> sucking memory up or taking forever to do certain things.  The memory is 
> from /tmp filling up.  A simple rm can take for freak'in ever to go 
> linearly through the directory to find the file's inode in the directory 
> file (uses a linear search).  find works better.  But once the damage is 
> cleaned up, a cron can be implemented to clean up after the lazy, evil 
> developers.
> 
> Rather than put lots of files in a single directory, name them in a 
> specific way such that you can put them in at least 2 levels of hashed 
> subdirectories.  Netscape used to do this with the local cache.
> 
> What's the magic number where things fall to crap?  Dunno.  More than 
> 10,000 but probably less than 100,000.  You can do performance 
> measurements with mail using a /var/mail with 10,000 mboxes and see if 
> things get better or worse.  sendmail has to read /var/mail every time 
> it delivers mail to a user's inbox.
> 
> All this is pretty old based on Solaris 7.  If things have improved in 
> Solaris 10, I'm sure someone will jump in here and correct me.
> 

I once had to deal with ~70,000 little files on a disk, all in one 
directory.  Some idiot developer left something running while she went 
on vacation.  It wasn't Solaris but NO O/S that I know of would handle 
the situation very well.

If I had to do it over again, I'd have the developer killed and then 
initialize the disk and restore from backup.  It took about three days 
to delete those files one at a time.

Michael Vilain wrote:
>
> No one in this thread has mentioned that it's a singularly Bad Idea(tm) 
> to put a large number of files in a single directory.  At some point, 
> there is a internal directory cache that gets filled and name lookups 
> get progressively longer (I may have the details of this wrong).  I 
> don't know if VxFS has this problem since, AFAIK, it doesn't use inodes 
> and linearly searched directory files.

Right.  On UFS the directory structure started out as a linked list so
it could grow without limits until it took up all of the inodes on the
device.  I think in some release it was switched to a self balancing
b-tree but I never tracked down for sure when or if.  On VXFS it started
out as a self balancing b-tree.  In either case it's possible to fill
the device's inode table before hitting a limit on files in a directory.
But it remains a bad idea if there's any way out of it.

> As a sysadmin, I had to clean up many lazy developer's "dirty" 
> implementations of a project.  These are ones that leave lots of files 
> in an application's defined temporary directory or much worse, the 
> system's /tmp.  On Solaris, that sucks up swap space and memory for 
> systems that stay up long periods of time using swap rather than real 
> file system space.

I had one case of a mount point with about 100K files in various dirs
and one particular dir with 1.5 million plus files and growing by 1000+
per day by the time I was called in.  I negotiated an aging policy but
the initial "find ... | xargs ..." expression to do the initial clean up
ran for hours.  I set it in cron daily and it ran for 15 minutes.  I
eventually switched it to run weekly.

> Rather than put lots of files in a single directory, name them in a 
> specific way such that you can put them in at least 2 levels of hashed 
> subdirectories.  Netscape used to do this with the local cache.

It's *far* better to talk the developers into hashing into a directory
tree any time there are 1000+ files in any one dir.  There are libraries
available to do that with just a library call and it reduces the
overhead considerably.  Sometimes it's not possible to dictate to the
developers but at very least open up a feature request ticket.

There's another fun aspect of directories in UFS, VXFS and for that
matter HFS and so on - They do not track depth because they are just
implemented in a tree.  Once I got called in because an installation
process had run all night on a developer's system and it was complaining
about a full drive when he got back in the next morning.  It turns out
it was in a loop creating directories then cd-ing into them.  It ran
until the mount point hit 100%.  It was so deep that "rm -rf *" did a
core dump before it just to the bottom. I ended up doing a loop like

while true ; do
cd *
done

and I waited until the shell ran out of stack space and it failed out
of the loop! At that point I moved * to lost+found, returned to the
top, deleted the older of the two trees and ran the loop again.  I put
the nested loop in a acript and it ran for most of the day clearing out
the directory chain.  Even with only one "file" per directory things can
get bad.

On 2010-05-18, underh20 <underh20.scubadiving@gmail.com> wrote:
> Hello,
>
> What is the maximum number of files allowed in a directory under UFS
> in Solaris 10 ?

32767.  See MAXLINK in sys/param.h.

> What is the maximum number of files allowed in a directory under
> Veritas File System 5.0 in Solaris 10 ?

32767.

> I'd like to know where to locate this maximum number under UFS and
> Veritas File System 5.0 in Solaris 10.  Is there any way that we could
> modify this maximum number ?

For vxfs, add "set vxfs:vx_maxlink=65534" to /etc/system and reboot.
For UFS, I don't know that you can.

Ceri
-- 
That must be wonderful!  I don't understand it at all.
                                                  -- Moliere

On 2010-05-19, Ceri Davies <ceri_usenet@submonkey.net> wrote:
> On 2010-05-18, underh20 <underh20.scubadiving@gmail.com> wrote:
>> Hello,
>>
>> What is the maximum number of files allowed in a directory under UFS
>> in Solaris 10 ?
>
> 32767.  See MAXLINK in sys/param.h.

some people here wrote they've seen 70k+ files in single directory...

anyway, MAXLINK seems to be the maximum number of (hard?)links to a
file and also the limit of subdirectories, see Solaris Internals,
Second Edition, Page 740-741 "ic_nlink"

I'm still trying to figure out the max. number of files in a
directory though... maybe someone else can shed some light on this :)

On 2010-05-19, Stefan Krueger <stadtkind2@gmx.de> wrote:
> On 2010-05-19, Ceri Davies <ceri_usenet@submonkey.net> wrote:
>> On 2010-05-18, underh20 <underh20.scubadiving@gmail.com> wrote:
>>> Hello,
>>>
>>> What is the maximum number of files allowed in a directory under UFS
>>> in Solaris 10 ?
>>
>> 32767.  See MAXLINK in sys/param.h.
>
> some people here wrote they've seen 70k+ files in single directory...

Not on UFS.

> anyway, MAXLINK seems to be the maximum number of (hard?)links to a
> file and also the limit of subdirectories, see Solaris Internals,
> Second Edition, Page 740-741 "ic_nlink"
>
> I'm still trying to figure out the max. number of files in a
> directory though... maybe someone else can shed some light on this :)

Or I could, as I'd already looked before replying:

From usr/src/uts/common/fs/ufs/ufs_dir.c:

 804  * Write a new directory entry for DE_LINK, DE_SYMLINK or DE_RENAME operations.
 805  * If tvpp is non-null, return with the pointer to the target vnode.
 806  */
 807 int
 808 ufs_direnter_lr(
....
 877         if (sip->i_nlink == MAXLINK) {
 878                 rw_exit(&sip->i_contents);
 879                 return (EMLINK);
 880         }

MAXLINK is defined (and included from) sys/param.h as:

#define	MAXLINK		32767	/* max links */

Ceri
-- 
That must be wonderful!  I don't understand it at all.
                                                  -- Moliere

On 2010-05-19, Ceri Davies <ceri_usenet@submonkey.net> wrote:
> On 2010-05-19, Stefan Krueger <stadtkind2@gmx.de> wrote:
>> On 2010-05-19, Ceri Davies <ceri_usenet@submonkey.net> wrote:
>>> On 2010-05-18, underh20 <underh20.scubadiving@gmail.com> wrote:
>>>> Hello,
>>>>
>>>> What is the maximum number of files allowed in a directory under UFS
>>>> in Solaris 10 ?
>>>
>>> 32767.  See MAXLINK in sys/param.h.
>>
>> some people here wrote they've seen 70k+ files in single directory...
>
> Not on UFS.
>
>> anyway, MAXLINK seems to be the maximum number of (hard?)links to a
>> file and also the limit of subdirectories, see Solaris Internals,
>> Second Edition, Page 740-741 "ic_nlink"
>>
>> I'm still trying to figure out the max. number of files in a
>> directory though... maybe someone else can shed some light on this :)
>
> Or I could, as I'd already looked before replying:
>
> From usr/src/uts/common/fs/ufs/ufs_dir.c:
>
>  804  * Write a new directory entry for DE_LINK, DE_SYMLINK or
>  DE_RENAME operations.
>  805  * If tvpp is non-null, return with the pointer to the target
>  vnode.
>  806  */
>  807 int
>  808 ufs_direnter_lr(
> ...
>  877         if (sip->i_nlink == MAXLINK) {
>  878                 rw_exit(&sip->i_contents);
>  879                 return (EMLINK);
>  880         }
>
> MAXLINK is defined (and included from) sys/param.h as:
>
> #define	MAXLINK		32767	/* max links */

"Write a new directory entry", directory != file and this basically
proofs what I wrote, so thanks for that :-)

anyway, to stop guessing I wrote a small shell script which just
touch'es files (on Solaris 10, UFS), I made it stop at 50.000, I hope
that's ok

$ ls | wc -l
   50001

So... I think the max. num of files in a directory is only limited by
the number of free inodes

HTH

On 19-May-2010, Stefan Krueger <stadtkind2@gmx.de> wrote:

> $ ls | wc -l
>    50001
>
> So... I think the max. num of files in a directory is only limited by
> the number of free inodes

Agreed... which is inadvertently determined by the size of the file system 
8]

Stefan Krueger wrote:
> On 2010-05-19, Ceri Davies <ceri_usenet@submonkey.net> wrote:
>> On 2010-05-18, underh20 <underh20.scubadiving@gmail.com> wrote:
>>> Hello,
>>>
>>> What is the maximum number of files allowed in a directory under UFS
>>> in Solaris 10 ?
>> 32767.  See MAXLINK in sys/param.h.
> 
> some people here wrote they've seen 70k+ files in single directory...
> 
> anyway, MAXLINK seems to be the maximum number of (hard?)links to a
> file and also the limit of subdirectories, see Solaris Internals,
> Second Edition, Page 740-741 "ic_nlink"
> 
> I'm still trying to figure out the max. number of files in a
> directory though... maybe someone else can shed some light on this :)

I don't know what the maximum number of files that can be cataloged in a 
directory.  I do know that it's a very poor idea to put thousands or 
tens of thousands of files in one directory.  Performance, to put it 
bluntly, will suck!

Ceri Davies <ceri_usenet@submonkey.net> writes:

>On 2010-05-18, underh20 <underh20.scubadiving@gmail.com> wrote:
>> Hello,
>>
>> What is the maximum number of files allowed in a directory under UFS
>> in Solaris 10 ?

>32767.  See MAXLINK in sys/param.h.

Not correct; that is the limit on the number of sub directories inside
a single directory.  You can created millions of files inside a directory
but the performance (esp for UFS) will be poor.

Casper
-- 
Expressed in this posting are my opinions.  They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.

Ceri Davies <ceri_usenet@submonkey.net> writes:

>On 2010-05-19, Stefan Krueger <stadtkind2@gmx.de> wrote:
>> On 2010-05-19, Ceri Davies <ceri_usenet@submonkey.net> wrote:
>>> On 2010-05-18, underh20 <underh20.scubadiving@gmail.com> wrote:
>>>> Hello,
>>>>
>>>> What is the maximum number of files allowed in a directory under UFS
>>>> in Solaris 10 ?
>>>
>>> 32767.  See MAXLINK in sys/param.h.
>>
>> some people here wrote they've seen 70k+ files in single directory...

>Not on UFS.

>> anyway, MAXLINK seems to be the maximum number of (hard?)links to a
>> file and also the limit of subdirectories, see Solaris Internals,
>> Second Edition, Page 740-741 "ic_nlink"
>>
>> I'm still trying to figure out the max. number of files in a
>> directory though... maybe someone else can shed some light on this :)

>Or I could, as I'd already looked before replying:

>From usr/src/uts/common/fs/ufs/ufs_dir.c:

> 804  * Write a new directory entry for DE_LINK, DE_SYMLINK or DE_RENAME operations.
> 805  * If tvpp is non-null, return with the pointer to the target vnode.
> 806  */
> 807 int
> 808 ufs_direnter_lr(
>...
> 877         if (sip->i_nlink == MAXLINK) {
> 878                 rw_exit(&sip->i_contents);
> 879                 return (EMLINK);
> 880         }

>MAXLINK is defined (and included from) sys/param.h as:

>#define	MAXLINK		32767	/* max links */


That's only for directories, not for files.

Casper
-- 
Expressed in this posting are my opinions.  They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.

In article <4bf4f6b0$0$22938$e4fe514c@news.xs4all.nl>,
	Casper H.S. Dik <Casper.Dik@Sun.COM> writes:
> Ceri Davies <ceri_usenet@submonkey.net> writes:
> 
>>On 2010-05-18, underh20 <underh20.scubadiving@gmail.com> wrote:
>>> Hello,
>>>
>>> What is the maximum number of files allowed in a directory under UFS
>>> in Solaris 10 ?
> 
>>32767.  See MAXLINK in sys/param.h.
> 
> Not correct; that is the limit on the number of sub directories inside
> a single directory.  You can created millions of files inside a directory
> but the performance (esp for UFS) will be poor.

Even if the filesystem and application does it efficiently, waiting
for ls(1) to sort a million files into alphabetic order still makes
it an admin's nightmare when they have to dive in to see what's gone
wrong, and things like rm {some shell expression} will blow up with
too many arguments. You just don't want to go there...

-- 
Andrew Gabriel
[email address is not usable -- followup in the newsgroup]