|
Home > Archive > Unix administration > April 2005 > multi-second hang in getdents (Linux 2.4, x86, ext3fs)
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
multi-second hang in getdents (Linux 2.4, x86, ext3fs)
|
|
| Jeremiah DeWitt Weiner 2005-04-26, 5:59 pm |
| I'm seeing a strange performance problem on one of our systems.
When running a find on one filesystem, there are long hangs (up to 30
seconds plus) during the find. Strace reveals that some of the getdents
calls are taking that much time!
From 'strace -T':
getdents64(0x4, 0x805ed28, 0x1000, 0x805ed28) = 48 <35.798045>
Oddly enough, not all of them are taking so long:
getdents64(0x4, 0x805ed28, 0x1000, 0x805ed28) = 48 <0.000016>
getdents64(0x4, 0x8059cd0, 0x1000, 0x8059cd0) = 240 <0.000017>
The system is otherwise almost completely idle during the find.
Nothing else is putting much of a demand on the filesystem. As you can
see, the time the getdents takes does not seem to be a function of how
much it returns. None of the directories the find traverses have very
many entries - none have more than 100, even. During the hang, vmstat
shows a blocked process, but no time spent in iowait. Any attempt to
access the same filesystem during a hang also hangs, so it's not just
the one process. The hangs are not repeatable at the same directory -
if I try just running a find at the point where a find hung before, it's
more likely than not to proceed just fine.
The filesystem is ext3, on a mirrored pair of Maxtor 180 GB SATA
drives attached to a 3Ware 8500-4. The 3Ware software does not show any
problems with the mirror. These are the only drives in the system, so I
can't directly compare them against anything else. Nothing out of the
ordinary appears in dmesg or /var/log/messages. I'm pretty out of
ideas...the length of time feels like an ATA timeout to me, but I don't
have any real evidence for that. Any suggestions would be much
appreciated.
JDW
| |
| John Reiser 2005-04-26, 5:59 pm |
| > I'm seeing a strange performance problem on one of our systems.
> When running a find on one filesystem, there are long hangs (up to 30
> seconds plus) during the find. Strace reveals that some of the getdents
> calls are taking that much time!
It is exceedingly probable that the delay is caused by writeback
to the disk of the updated access time for all of the directories
that have been visited by the 'find'. This saturates the disk
queues for a while.
The workaround is easy: add "nodiratime" to the mount options in
/etc/fstab. For example:
/dev/hda3 / ext3 defaults,nodiratime 1 1
Of course, this means that the recorded time of last access on
directories won't be "correct", but almost nobody uses it anyway,
because it is _not_ updated just because a file within it gets
changed. Thus access time on a directory cannot be used as a hint
of whether a backup program needs to descend into the directory
in order to see all files that were modified after a given date.
--
| |
| Jeremiah DeWitt Weiner 2005-04-27, 5:58 pm |
| John Reiser <jreiser@bitwagon.com> wrote:
> The workaround is easy: add "nodiratime" to the mount options in
> /etc/fstab.
This looks like it does the trick. Thanks very much!
JDW
|
|
|
|
|