I have a directory which contains 21 thousand files. when I am running
following command I am getting error:

ls -t /path-to-my-dir/*.Z | head -1

Error is:
ksh: /bin/ls: arg list too long

Can anybody help me understanding the issue and the fix for this?

Thanks in advance,
Amit

On 2009-07-06, kamit28 <kamit28@gmail.com> wrote:
> I have a directory which contains 21 thousand files. when I am running
> following command I am getting error:
>
> ls -t /path-to-my-dir/*.Z | head -1
>
> Error is:
> ksh: /bin/ls: arg list too long
>
> Can anybody help me understanding the issue and the fix for this?

There is a limit how long argument lists can become, named {ARG_MAX}
(see "man -s 2 exec"). This limit is usually at 1048320.

In this case I would recommend to use

   ls -t /path-to-my-dir | grep '\.Z$' | head -1

instead to avoid the overlong argument list.

Andreas.

kamit28 <kamit28@gmail.com> writes:
> I have a directory which contains 21 thousand files. when I am running
> following command I am getting error:
> 
> ls -t /path-to-my-dir/*.Z | head -1
> 
> Error is:
> ksh: /bin/ls: arg list too long
> 
> Can anybody help me understanding the issue and the fix for this?

   It's the shell (ksh), not /bin/ls, that expands expressions like "*.Z".
ksh's expansion of "/path-to-my-dir/*.Z" creates more command line
arguments than ksh is willing to pass. 

You could try the same command under /bin/sh, or, as Andreas already posted,

	ls -t /path-to-my-dir | grep '\.Z$' | head -1

HTH,
 -WBE

Winston wrote:

>> ksh: /bin/ls: arg list too long

> [...] You could try the same command under /bin/sh, or, as Andreas
> already posted,

You can read Andreas' response literally:
This limit is not related to shells or tools.
It happens, if - and only if - one of the exec(2) system calls
is called with a "length of arguments" exceeding ARG_MAX (1M or
2M on Solaris for a 32bit or 64bit exec, respectively).

-- 
more background on http://www.in-ulm.de/~mascheck/various/argmax/

I previously posted:
>>> ksh: /bin/ls: arg list too long

>> [...] You could try the same command under /bin/sh, or, as Andreas
>> already posted,

Sven Mascheck <mascheck@email.invalid> replied:
> You can read Andreas' response literally:
> This limit is not related to shells or tools.

We agree the ARG_MAX limit isn't.

> It happens, if - and only if - one of the exec(2) system calls
> is called with a "length of arguments" exceeding ARG_MAX (1M or
> 2M on Solaris for a 32bit or 64bit exec, respectively).

I've encountered old shell programs (other than /bin/sh) that had their own
internal limits on how many arguments they could pass -- limits which had
nothing to do with exec's limits.  One, for example, had a limit of
~10,000(IIRC) arguments (regardless of total length).  I don't know ksh's
internals and didn't want to guess whether the OP's version of ksh might
have such a limit.

In any event, the solution is the same -- pass the directory to ls rather
than have the shell expand the wildcard expression.
 -WBE

Winston wrote:

> I don't know ksh's internals and didn't want to guess whether the OP's
> version of ksh might have such a limit.

Reasonable unix programs must not come with such arbitrary limits.
And no widely known bourne compatible shell nor toolchest does.
You should be able to safely assume that only ARG_MAX is relevant.
(The maximum length of an interactive literal command line is
 usually limited, but that's not related to expansion anymore.)

The only shell I know with such limits is a (clearing my throat)
bourne-incompatible shareware marketed unix shell called 'bsh' from
a DOS programmer.  No, it is not exactly widespread...

(The name "bsh" is not to confuse with the renamed bourne shells on AIX
 and IRIX, where a korn shell became the /bin/sh.)
-- 
http://www.in-ulm.de/~mascheck/various/shells/

Sven Mascheck <mascheck@email.invalid> writes:
>You can read Andreas' response literally:
>This limit is not related to shells or tools.
>It happens, if - and only if - one of the exec(2) system calls
>is called with a "length of arguments" exceeding ARG_MAX (1M or
>2M on Solaris for a 32bit or 64bit exec, respectively).

hubcap% ps
   PID TTY         TIME CMD
 13997 pts/5       0:00 csh
 14028 pts/5       0:00 ps
hubcap% ls *
Arguments too long
hubcap% sh
$ ls *
                                    mdking.access.log
Permission                          mdking.home.tar
Variable                            mdking.mail.tar
,pd                                  mdking.pop.tar
01-README                            mdking.web.log
                         .
                         .
                         .

csh poops out before sh when expanding an asterisk...

-Mike

On Jul 7, 7:42=A0am, Mike Marshall <hub...@clemson.edu> wrote:
> Sven Mascheck <masch...@email.invalid> writes:
> >You can read Andreas' response literally:
> >This limit is not related to shells or tools.
> >It happens, if - and only if - one of the exec(2) system calls
> >is called with a "length of arguments" exceeding ARG_MAX (1M or
> >2M on Solaris for a 32bit or 64bit exec, respectively).
>
> hubcap% ps
> =A0 =A0PID TTY =A0 =A0 =A0 =A0 TIME CMD
> =A013997 pts/5 =A0 =A0 =A0 0:00 csh
> =A014028 pts/5 =A0 =A0 =A0 0:00 ps
> hubcap% ls *
> Arguments too long
> hubcap% sh
> $ ls *
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 m=
dking.access.log
> Permission =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0mdking.home=
..tar
> Variable =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0mdking.ma=
il.tar
> ,pd =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0md=
king.pop.tar
> 01-README =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0mdking.w=
eb.log
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0.
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0.
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0.
>
> csh poops out before sh when expanding an asterisk...
>
> -Mike

Thanks everybody for the comprehensive answer.

Sven, the link you posted was really helpful in understanding the
internals.

wbe@ubeblock.psr.com.invalid (Winston) writes:

>kamit28 <kamit28@gmail.com> writes:
>> I have a directory which contains 21 thousand files. when I am running
>> following command I am getting error:
>> 
>> ls -t /path-to-my-dir/*.Z | head -1
>> 
>> Error is:
>> ksh: /bin/ls: arg list too long
>> 
>> Can anybody help me understanding the issue and the fix for this?

>   It's the shell (ksh), not /bin/ls, that expands expressions like "*.Z".
>ksh's expansion of "/path-to-my-dir/*.Z" creates more command line
>arguments than ksh is willing to pass. 

>You could try the same command under /bin/sh, or, as Andreas already posted,

>	ls -t /path-to-my-dir | grep '\.Z$' | head -1

It's probably not the shell but the kernel.  Since we're talking about
21000 files, you only have around 40 bytes per pathname.
This may actually work:

	(cd /path-to-my-dir; ls -t *.Z | head -1)

Casper
-- 
Expressed in this posting are my opinions.  They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.

Maybe worthwhile to point out the xargs tool, and in
what cases it might be useful.   Someone.  

David