f



Efficient removal of directories

	I have a directory called SData, in my home directory, that I 
want to remove, together with all its contents. In order to that, I do 
'rm -rf SData'. Now the problem is that SData contains a rather 
complicated hierarchy of subdirectories and files, numbering in the tens 
of thousands. Which implies that the command above takes a long time.

	Are there ways to tackle this issue, that are more efficient than 
'rm -rf'? 

0
S
12/1/2016 5:41:22 PM
comp.os.linux.misc 33599 articles. 1 followers. amosa69 (78) is leader. Post Follow

11 Replies
661 Views

Similar Articles

[PageSpeed] 13

S.K.R. de Jong <SKRdJ@nowhere.net> wrote:
>        I have a directory called SData, in my home directory, that I 
> want to remove, together with all its contents. In order to that, I do 
> 'rm -rf SData'. Now the problem is that SData contains a rather 
> complicated hierarchy of subdirectories and files, numbering in the tens 
> of thousands. Which implies that the command above takes a long time.
> 
>        Are there ways to tackle this issue, that are more efficient than 
> 'rm -rf'? 

You may or may not have the "fastrm" [1] command installed.

If you do, it may speed things up slightly.

But in the end, a complicated hierarchy of subdirectories and files
will take some time to remove, esp.  if you have a mechanical hard
disk, because part of the time consumed will be latency for the disk
head to move about to retreive the necessary data to know what to drop
from the filesystem.


[1] https://linux.die.net/man/1/fastrm
0
Rich
12/1/2016 5:52:04 PM
On 01/12/16 17:41, S.K.R. de Jong wrote:
> 	I have a directory called SData, in my home directory, that I
> want to remove, together with all its contents. In order to that, I do
> 'rm -rf SData'. Now the problem is that SData contains a rather
> complicated hierarchy of subdirectories and files, numbering in the tens
> of thousands. Which implies that the command above takes a long time.
>
> 	Are there ways to tackle this issue, that are more efficient than
> 'rm -rf'?
>

copy what you want to retain elsewhere, and re partition/format?


-- 
No Apple devices were knowingly used in the preparation of this post.
0
The
12/1/2016 6:25:21 PM
On 01/12/16 17:52, Rich wrote:
> S.K.R. de Jong <SKRdJ@nowhere.net> wrote:
>>        I have a directory called SData, in my home directory, that I
>> want to remove, together with all its contents. In order to that, I do
>> 'rm -rf SData'. Now the problem is that SData contains a rather
>> complicated hierarchy of subdirectories and files, numbering in the tens
>> of thousands. Which implies that the command above takes a long time.
>>
>>        Are there ways to tackle this issue, that are more efficient than
>> 'rm -rf'?
>
> You may or may not have the "fastrm" [1] command installed.
>
> If you do, it may speed things up slightly.
>
> But in the end, a complicated hierarchy of subdirectories and files
> will take some time to remove, esp.  if you have a mechanical hard
> disk, because part of the time consumed will be latency for the disk
> head to move about to retreive the necessary data to know what to drop
> from the filesystem.
>

Er no. I am fairly sure that nearly all Linux file systems contain 
*directory* entries in very small areas of the disk, and that area is of 
course the most apposite for caching.

IN short you should be able to delete the directory entries working 
entirely in a cached memory copy of the directory and inode structures, 
and simply flush that to disk in a single operation once finished.

Virtually no seek time involved

>
> [1] https://linux.die.net/man/1/fastrm
>


-- 
No Apple devices were knowingly used in the preparation of this post.
0
The
12/1/2016 6:31:30 PM
On 12/01/2016 12:52 PM, Rich wrote:
> S.K.R. de Jong <SKRdJ@nowhere.net> wrote:
>>        I have a directory called SData, in my home directory, that I 
>> want to remove, together with all its contents. In order to that, I do 
>> 'rm -rf SData'. Now the problem is that SData contains a rather 
>> complicated hierarchy of subdirectories and files, numbering in the tens 
>> of thousands. Which implies that the command above takes a long time.
>>
>>        Are there ways to tackle this issue, that are more efficient than 
>> 'rm -rf'? 
> 
If that SData directory is actually in a separate partition, just
reformat the partition.

On my system /home/boinc is a separate partition from all the other
/home/* directories.

I do this so that even if boinc processes go out of control, they cannot
hog the rest of the file system. And boinc user can access very limited
amounts of the rest of the file system.


-- 
  .~.  Jean-David Beyer          Registered Linux User 85642.
  /V\  PGP-Key:166D840A 0C610C8B Registered Machine  1935521.
 /( )\ Shrewsbury, New Jersey    http://linuxcounter.net
 ^^-^^ 13:25:01 up 15 days, 21:35, 2 users, load average: 4.44, 4.42, 4.42
0
Jean
12/1/2016 6:31:48 PM
The Natural Philosopher <tnp@invalid.invalid> wrote:
> On 01/12/16 17:52, Rich wrote:
>> S.K.R. de Jong <SKRdJ@nowhere.net> wrote:
>>>        I have a directory called SData, in my home directory, that I
>>> want to remove, together with all its contents. In order to that, I do
>>> 'rm -rf SData'. Now the problem is that SData contains a rather
>>> complicated hierarchy of subdirectories and files, numbering in the tens
>>> of thousands. Which implies that the command above takes a long time.
>>>
>>>        Are there ways to tackle this issue, that are more efficient than
>>> 'rm -rf'?
>>
>> You may or may not have the "fastrm" [1] command installed.
>>
>> If you do, it may speed things up slightly.
>>
>> But in the end, a complicated hierarchy of subdirectories and files
>> will take some time to remove, esp.  if you have a mechanical hard
>> disk, because part of the time consumed will be latency for the disk
>> head to move about to retreive the necessary data to know what to drop
>> from the filesystem.
>>
> 
> Er no. I am fairly sure that nearly all Linux file systems contain 
> *directory* entries in very small areas of the disk, and that area is of 

You are thinking of the inode tables.  They are in specific constrained
areas of the disks.

The blocks that make up "directories" are just allocated from free
space the same as are the blocks that contain actual file data.  So
they end up "spread around".

And to know which inodes need to be marked as "deleted" (well,
actually, have their reference counter decremented) one has to find,
and read those directory blocks.
0
Rich
12/1/2016 6:36:30 PM
On Thu, 1 Dec 2016 17:41:22 +0000 (UTC), S.K.R. de Jong wrote:
>
> 	I have a directory called SData, in my home directory, that I 
> want to remove, together with all its contents. In order to that, I do 
> 'rm -rf SData'. Now the problem is that SData contains a rather 
> complicated hierarchy of subdirectories and files, numbering in the tens 
> of thousands. Which implies that the command above takes a long time.
>
> 	Are there ways to tackle this issue, that are more efficient than 
> 'rm -rf'? 

It shouldn't take long. I once removed a quite complex 10 GB directory
structure (ext3 here).

Well not sure how long it took. But no matter how long, it isn't that you
stare at the cursor and wait to complete. Just do other things in the
meantime.
-- 
Andreas
You know you are a redneck if
you ever filled your deer tag on the golf course.
0
Andreas
12/1/2016 9:36:26 PM
At Thu, 01 Dec 2016 16:36:26 -0500 Andreas Kohlbach <ank@spamfence.net> wrote:

> 
> On Thu, 1 Dec 2016 17:41:22 +0000 (UTC), S.K.R. de Jong wrote:
> >
> > 	I have a directory called SData, in my home directory, that I 
> > want to remove, together with all its contents. In order to that, I do 
> > 'rm -rf SData'. Now the problem is that SData contains a rather 
> > complicated hierarchy of subdirectories and files, numbering in the tens 
> > of thousands. Which implies that the command above takes a long time.
> >
> > 	Are there ways to tackle this issue, that are more efficient than 
> > 'rm -rf'? 
> 
> It shouldn't take long. I once removed a quite complex 10 GB directory
> structure (ext3 here).
> 
> Well not sure how long it took. But no matter how long, it isn't that you
> stare at the cursor and wait to complete. Just do other things in the
> meantime.

Right.  The "rm -rf" does not really that long, unless the disk in question 
has been freshly mounted.  *But* if you follow up the "rm -rf" with a "sync" 
(or umount), *that* may take some time.  Otherwise the "sync'ing" will happen 
in the background as you continue to do "other things".

-- 
Robert Heller             -- 978-544-6933
Deepwoods Software        -- Custom Software Services
http://www.deepsoft.com/  -- Linux Administration Services
heller@deepsoft.com       -- Webhosting Services
                                                                                                    
0
Robert
12/1/2016 10:21:19 PM
On 2016-12-01, The Natural Philosopher <tnp@invalid.invalid> wrote:
> On 01/12/16 17:52, Rich wrote:
>> S.K.R. de Jong <SKRdJ@nowhere.net> wrote:
>>>        I have a directory called SData, in my home directory, that I
>>> want to remove, together with all its contents. In order to that, I do
>>> 'rm -rf SData'. Now the problem is that SData contains a rather
>>> complicated hierarchy of subdirectories and files, numbering in the tens
>>> of thousands. Which implies that the command above takes a long time.
>>>
>>>        Are there ways to tackle this issue, that are more efficient than
>>> 'rm -rf'?
>>
>> You may or may not have the "fastrm" [1] command installed.
>>
>> If you do, it may speed things up slightly.
>>
>> But in the end, a complicated hierarchy of subdirectories and files
>> will take some time to remove, esp.  if you have a mechanical hard
>> disk, because part of the time consumed will be latency for the disk
>> head to move about to retreive the necessary data to know what to drop
>> from the filesystem.
>>
>
> Er no. I am fairly sure that nearly all Linux file systems contain 
> *directory* entries in very small areas of the disk, and that area is of 
> course the most apposite for caching.

?? A directory is just a file, just like any other file. AFAIK it is not
stored in any particular place. 
Plus, if you have a whole bunch of subdirectories, all of the files in
that subdirectory have to be erased before the subdirectory can be
removed.

>
> IN short you should be able to delete the directory entries working 
> entirely in a cached memory copy of the directory and inode structures, 
> and simply flush that to disk in a single operation once finished.

>
> Virtually no seek time involved
>
>>
>> [1] https://linux.die.net/man/1/fastrm
>>
>
>
0
William
12/2/2016 2:14:26 AM
On 2016-12-01, Robert Heller <heller@deepsoft.com> wrote:
>
> Right.  The "rm -rf" does not really that long, unless the disk in question 
> has been freshly mounted.  *But* if you follow up the "rm -rf" with a "sync" 
> (or umount), *that* may take some time.  Otherwise the "sync'ing" will happen 
> in the background as you continue to do "other things".

It can take a while, if there are thousands or tens of thousands (or
more) inodes that rm has to traverse and unlink (especially if the
directory structure is itself fairly deep).  (Here I count "a while" as
more than 1-2 minutes.)  But you can run it in a screen, or background
it, or put it in another xterm window, and then just ignore it.  It's
probably not worth the effort to optimize rm -rf unless there are on the
order of millions or more inodes that need to be unlinked.  (And if
you're looking to have that many files you should probably come up with
this strategy before you actually have all those files.)

--keith

-- 
kkeller-usenet@wombat.san-francisco.ca.us
(try just my userid to email me)
AOLSFAQ=http://www.therockgarden.ca/aolsfaq.txt

0
Keith
12/2/2016 6:33:33 AM
S.K.R. de Jong wrote:

> I have a directory called SData, in my home directory, that I
> want to remove, together with all its contents. In order to that, I do
> 'rm -rf SData'. Now the problem is that SData contains a rather
> complicated hierarchy of subdirectories and files, numbering in the tens
> of thousands. Which implies that the command above takes a long time.
> 
> Are there ways to tackle this issue, that are more efficient than
> 'rm -rf'?

Interesting that this popped up.

http://www.tecmint.com/empty-delete-file-content-linux/#

It does not state if it works with directories though


-- 
faeychild
Running kde on 3.14.43-desktop-1.mga4 kernel.
Mageia release 4 (Official) for x86_64

0
faeychild
12/3/2016 9:01:53 PM
On Thu, 01 Dec 2016 17:41:22 +0000, S.K.R. de Jong wrote:

> 	Are there ways to tackle this issue, that are more efficient than
> 'rm -rf'?

Of course. The "mkfs" command will be an order of magnitude faster and 
very thorough.



-- 
Mladen Gogala
The Oracle Whisperer
http://mgogala.byethost5.com
0
Mladen
12/3/2016 9:19:59 PM
Reply: