|
Home > Archive > Unix administration > June 2006 > /var out of disk space, but not really
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
/var out of disk space, but not really
|
|
| ckbcouture@gmail.com 2006-05-31, 1:18 pm |
| HI.
So I have a problem with my /var partition. I am unable to write to it
and get "/var: write failed, file system is full" and "close failed:
[Errno 28] No space left on device" when I do.
If I look at disk usage using df, I get:
Filesystem 1M-blocks Used Avail Capacity Mounted on
/dev/ld0a 4585 2784 1571 63% /
/dev/ld0f 4851 4617 -8 100% /var
/dev/ld0e 9695 6129 3081 66% /usr
/dev/ld0g 9695 2872 6337 31% /home
/dev/ld0h 38788 9461 27386 25% /www
kernfs 0 0 0 100% /kern
procfs 0 0 0 100%
/usr/pkg/emul/linux/proc
elk:/www/staging 38788 10463 26384 28%
/www/staging
rhino:/www/production 268340 12743 241966 5%
/www/production
If I look at /var using du, I get:
srvrs01# du -sPm /var
1125 /var
Clearly there is a discrepancy here and I can't find anything to delete
to free up the disk space. Any ideas on what is wrong and what I can
do to fix it?
Thanks,
Cameron
| |
| Adam Price 2006-05-31, 1:18 pm |
| On 31 May 2006 08:18:14 -0700, ckbcouture@gmail.com wrote:
> HI.
>
> So I have a problem with my /var partition. I am unable to write to it
> and get "/var: write failed, file system is full" and "close failed:
> [Errno 28] No space left on device" when I do.
>
> If I look at disk usage using df, I get:
>
> Filesystem 1M-blocks Used Avail Capacity Mounted on
> /dev/ld0a 4585 2784 1571 63% /
> /dev/ld0f 4851 4617 -8 100% /var
> /dev/ld0e 9695 6129 3081 66% /usr
> /dev/ld0g 9695 2872 6337 31% /home
> /dev/ld0h 38788 9461 27386 25% /www
> kernfs 0 0 0 100% /kern
> procfs 0 0 0 100%
> /usr/pkg/emul/linux/proc
> elk:/www/staging 38788 10463 26384 28%
> /www/staging
> rhino:/www/production 268340 12743 241966 5%
> /www/production
>
> If I look at /var using du, I get:
>
> srvrs01# du -sPm /var
> 1125 /var
>
> Clearly there is a discrepancy here and I can't find anything to delete
> to free up the disk space. Any ideas on what is wrong and what I can
> do to fix it?
>
> Thanks,
> Cameron
Perhaps an open file has been deleted. If downtime is easy to arrange then
try a reboot, this will restart whatever is holding open the file and have
the added bonus that /var can be checked with fsck on reboot.
If you can't get downtime on the machine then I suggest that you look for
open files using fuser or lsof and see what process can be restarted.
If you don't find any open processes then you are probably stuck with
downtime and fsck anyway.
Adam
| |
| Lew Pitcher 2006-05-31, 1:18 pm |
| ckbcouture@gmail.com wrote:
> HI.
>
> So I have a problem with my /var partition. I am unable to write to it
> and get "/var: write failed, file system is full" and "close failed:
> [Errno 28] No space left on device" when I do.
>
> If I look at disk usage using df, I get:
>
> Filesystem 1M-blocks Used Avail Capacity Mounted on
> /dev/ld0a 4585 2784 1571 63% /
> /dev/ld0f 4851 4617 -8 100% /var
> /dev/ld0e 9695 6129 3081 66% /usr
> /dev/ld0g 9695 2872 6337 31% /home
> /dev/ld0h 38788 9461 27386 25% /www
> kernfs 0 0 0 100% /kern
> procfs 0 0 0 100%
> /usr/pkg/emul/linux/proc
> elk:/www/staging 38788 10463 26384 28%
> /www/staging
> rhino:/www/production 268340 12743 241966 5%
> /www/production
>
> If I look at /var using du, I get:
>
> srvrs01# du -sPm /var
> 1125 /var
>
> Clearly there is a discrepancy here and I can't find anything to delete
> to free up the disk space. Any ideas on what is wrong and what I can
> do to fix it?
It could be that you have deleted the directory entry for an open file.
In such a case, the space taken up by the file won't be returned to the
filesystem until the file is closed (and thus, df(1) would show the
space as allocated), but the file (lacking a directory entry) won't
appear in a directory listing (and thus, du(1) won't show the space as
allocated).
Have you deleted any files (like those allocated by syslog) recently?
Have you recycled the processes that keep those files open?
--
Lew Pitcher, IT Specialist, Corporate Technology Solutions,
Enterprise Technology Solutions, TD Bank Financial Group
(Opinions expressed here are my own, not my employer's)
| |
| ckbcouture@gmail.com 2006-05-31, 7:16 pm |
| Indeed this was the case, I had deleted named.run without restarting
named.
Thanks!
Cameron
Lew Pitcher wrote:
> ckbcouture@gmail.com wrote:
>
> It could be that you have deleted the directory entry for an open file.
> In such a case, the space taken up by the file won't be returned to the
> filesystem until the file is closed (and thus, df(1) would show the
> space as allocated), but the file (lacking a directory entry) won't
> appear in a directory listing (and thus, du(1) won't show the space as
> allocated).
>
> Have you deleted any files (like those allocated by syslog) recently?
> Have you recycled the processes that keep those files open?
>
>
> --
>
> Lew Pitcher, IT Specialist, Corporate Technology Solutions,
> Enterprise Technology Solutions, TD Bank Financial Group
>
> (Opinions expressed here are my own, not my employer's)
>
>
> --------------enig5D08B35B5D979B26B4AE44A2
> Content-Type: application/pgp-signature
> Content-Transfer-Encoding: base64
> Content-Disposition: inline;
> filename="signature.asc"
> Content-Description: OpenPGP digital signature
> X-Google-AttachSize: 251
| |
| Doug Freyburger 2006-05-31, 7:16 pm |
| Lew Pitcher wrote:
>
> It could be that you have deleted the directory entry for an open file.
> In such a case, the space taken up by the file won't be returned to the
> filesystem until the file is closed (and thus, df(1) would show the
> space as allocated), but the file (lacking a directory entry) won't
> appear in a directory listing (and thus, du(1) won't show the space as
> allocated).
>
> Have you deleted any files (like those allocated by syslog) recently?
> Have you recycled the processes that keep those files open?
BTW, the professional society for sysadmins is Usenix/SAGE
and they maintain an acheivement scale that goes Novice,
Junior, Intermediate/Advanced, Senior. Each step is deliberately
more difficult than the previous.
The reasons that "df" and "du" output can differ are classic
qualifying questions to reach the Intermediate/Advanced level
of acheivement. It's understanding how inodes and directories
work. In this case see similar to /usr/include/sys/{v,i}node.h
for how an on-line inode gets mapped to memory when open
and the structures added to it. The one in question is the
"reference count". Both the "link count" and the "reference
count" need to go to zero before the backgroud process of
adding the blocks to the free chain starts.
Extra points for coming up with assorted other reasons that
"df" and "du" output differ. ;^)
| |
|
|
| Doug Freyburger 2006-06-01, 7:16 pm |
| Tim Skirvin wrote:
> Doug Freyburger writes:
>
>
> Well, one of them is anyway. Is anybody here besides me in LOPSA?
First I've heard of it but they seem a good idea. Still taking charter
memberships? I read through the list of founding members - I've
worked with Theo van Dinter and Benjy Feen directly and have dealt
with a fair number of the other folks indirectly over the years. I
wish I had a quarter of Theo's talent, sigh.
| |
| Doug Freyburger 2006-06-02, 1:23 am |
| Tim Skirvin wrote:
>
> Is anybody here besides me in LOPSA?
I went ahead and signed up.
| |
| Tim Skirvin 2006-06-02, 1:23 am |
| "Doug Freyburger" <dfreybur@yahoo.com> writes:
[vbcol=seagreen]
> I went ahead and signed up.
Did I get the super-ultra-nifty referral bonus?
- Tim Skirvin (tskirvin@killfile.org)
--
http://www.killfile.org/~tskirvin/ Skirv's Homepage <FISH>< <*>
http://tskirvin.livejournal.com/ Skirv's LiveJournal
| |
| Doug Freyburger 2006-06-02, 1:15 pm |
| Tim Skirvin wrote:
> Doug Freyburger writes:
>
>
>
> Did I get the super-ultra-nifty referral bonus?
Depends on if I guessed your username right. ;^)
| |
| Michael Paoli 2006-06-12, 1:30 am |
| ckbcouture@gmail.com wrote:
> Indeed this was the case, I had deleted named.run without restarting
> named.
And with the proc filesystem and/or lsof(8), one can often isolate the
unlinked open file and the PID that has it open. For some OS flavors,
it's even possible to relink the file, if that's desired.
And as mentioned in other post(s), that's not the only way to end up
with a df vs. du discrepancy.[vbcol=seagreen]
|
|
|
|
|