Unix administration - Problems deleting a large directory

Author

Problems deleting a large directory

Carla Tironi Farinati

2006-01-09, 2:56 am

hi group!,

I'm trying to delete a very large directory full of small files (not
more than 20k each). The directory weights around 40GB. I'm running
Centos4.

Now when I try "rm -rf mydir", the system hangs and do nothing.

No change is displayed when I look for the free space.

I also tried to do a "rm *" inside the directory to delete to see if
the system displayed the question "remove the regular file XXX?", but
the console again hangs there as nothing.

Any other deletion of small directories or files works OK in the sys.

Any ideas?.

best,
Carla Tironi Farinati

Alexey G. Khramkov

2006-01-09, 6:02 pm

"Carla Tironi Farinati" <carlatf@gmail.com> writes:

> Now when I try "rm -rf mydir", the system hangs and do nothing.
>
> No change is displayed when I look for the free space.
>
> I also tried to do a "rm *" inside the directory to delete to see if
> the system displayed the question "remove the regular file XXX?", but
> the console again hangs there as nothing.

In both cases those commands generate full listing (the same long time).

>
> Any other deletion of small directories or files works OK in the sys.
>
> Any ideas?.

I don't sure it helps (i have no experience in the same environment).
Try a*, b* instead of *. Increase the statis part (2, 3, n letters)
if it will hang again.

Hint: use separate partition for that stuff:

# umount /monster
# newfs /dev/new_monster
# mount /dev/new_monster /monster

HTH,
--
status = rule("RFC1855"); /* avoid flame wars */

Doug Freyburger

2006-01-09, 6:02 pm

Carla Tironi Farinati wrote:
>
> I'm trying to delete a very large directory full of small files (not
> more than 20k each). The directory weights around 40GB. I'm running
> Centos4.
>
> Now when I try "rm -rf mydir", the system hangs and do nothing.
> No change is displayed when I look for the free space.
>
> I also tried to do a "rm *" inside the directory to delete to see ...

With thousands of files the "rm *" will overflow the shell command
line buffer and fail.

With thousands of files "rm -rf mydir" probably needs to run a lot
longer to work.

Brute force method - This will take much longer but will run well:

find mydir \( \! -type d \) -print | xargs rm -f
find mydir -depth -print | xargs -1 rm -rf
rm -rf mydir

Since the first one will locate all non-directories and delete them
using a legal command line length, it will run for a very long time.
The second will run much faster. The third will run as fast as you
can hit the return.

Here's a bizzare case. I include it just in case the tree is so
deep that find runs out of virtual memory remembering how deep
it is:

Once I had a use try to install Clearcase on a workstation they
had the root password. (This is also a lesson why users should
never get root). They typed wrong and somehow convinced it
to try installing recursively. It made a chain of about 5-6 levels
of empty directory, cd'd to the bottom and then looped to create
the chain all over again. It started running and since the disk
access light was one the users went home and let it run. About
200 MB later the process finally ran out of virtual memory and failed.
By that time it created a single chain of directories so deep no
shell could cd that deep.

Clearly if I wrote a loop of moving the dir one level down up one
to flatten it the script would have run for a week. It had millions of
inodes in the chain .. to . each link. But if I wrote a loop to "cd *"
that would fail as well.

The good part was that -

while true ; do
cd *
done

ran for several minutes then failed. But when i tried to type "pwd"
it took minutes to start printing its name. Great -

cd ..
mv * /mountpt/lost+found
rm -rf /mountpt/lost+found/*
cd /mountput/lost+found
cd *

lather rinse repeat. Each attempt removed as many as their
names would fit in the shell's virtual memory and also moved the
rest up to an easily accessible location. However many dirs
the original chain was deep, that's how many times I had to
run that sequence. Prine pump by mooving to lost+found,
dive as deep as possible, move up one so it fits in VM, move
up to lost+found, attempt deletion, the "short"er chain suceeds.

base60

2006-01-09, 11:01 pm

Carla Tironi Farinati wrote:
> hi group!,
>
> I'm trying to delete a very large directory full of small files (not
> more than 20k each). The directory weights around 40GB. I'm running
> Centos4.
>
> Now when I try "rm -rf mydir", the system hangs and do nothing.
>
> No change is displayed when I look for the free space.
>
> I also tried to do a "rm *" inside the directory to delete to see if
> the system displayed the question "remove the regular file XXX?", but
> the console again hangs there as nothing.
>
> Any other deletion of small directories or files works OK in the sys.
>
> Any ideas?.

If you want to watch them:

cd mydir
find . -type f -ls -exec rm {} \;

>
> best,
> Carla Tironi Farinati
>