HI.

So I have a problem with my /var partition.  I am unable to write to it
and get "/var: write failed, file system is full" and "close failed:
[Errno 28] No space left on device" when I do.

If I look at disk usage using df, I get:

Filesystem            1M-blocks     Used     Avail Capacity  Mounted on
/dev/ld0a                  4585     2784      1571    63%    /
/dev/ld0f                  4851     4617        -8   100%    /var
/dev/ld0e                  9695     6129      3081    66%    /usr
/dev/ld0g                  9695     2872      6337    31%    /home
/dev/ld0h                 38788     9461     27386    25%    /www
kernfs                        0        0         0   100%    /kern
procfs                        0        0         0   100%
/usr/pkg/emul/linux/proc
elk:/www/staging          38788    10463     26384    28%
/www/staging
rhino:/www/production    268340    12743    241966     5%
/www/production

If I look at /var using du, I get:

srvrs01# du -sPm /var
1125    /var

Clearly there is a discrepancy here and I can't find anything to delete
to free up the disk space.  Any ideas on what is wrong and what I can
do to fix it?

Thanks,
Cameron

[ Post a follow-up to this message ]

On 31 May 2006 08:18:14 -0700, ckbcouture@gmail.com wrote:

> HI.
>
> So I have a problem with my /var partition.  I am unable to write to it
> and get "/var: write failed, file system is full" and "close failed:
> [Errno 28] No space left on device" when I do.
>
> If I look at disk usage using df, I get:
>
> Filesystem            1M-blocks     Used     Avail Capacity  Mounted on
> /dev/ld0a                  4585     2784      1571    63%    /
> /dev/ld0f                  4851     4617        -8   100%    /var
> /dev/ld0e                  9695     6129      3081    66%    /usr
> /dev/ld0g                  9695     2872      6337    31%    /home
> /dev/ld0h                 38788     9461     27386    25%    /www
> kernfs                        0        0         0   100%    /kern
> procfs                        0        0         0   100%
> /usr/pkg/emul/linux/proc
> elk:/www/staging          38788    10463     26384    28%
> /www/staging
> rhino:/www/production    268340    12743    241966     5%
> /www/production
>
> If I look at /var using du, I get:
>
> srvrs01# du -sPm /var
> 1125    /var
>
> Clearly there is a discrepancy here and I can't find anything to delete
> to free up the disk space.  Any ideas on what is wrong and what I can
> do to fix it?
>
> Thanks,
> Cameron

Perhaps an open file has been deleted. If downtime is easy to arrange then
try a reboot, this will restart whatever is holding open the file and have
the added bonus that /var can be checked with fsck on reboot.
If you can't get downtime on the machine then I suggest that you look for
open files using fuser or lsof and see what process can be restarted.
If you don't find any open processes then you are probably stuck with
downtime and fsck anyway.
Adam

[ Post a follow-up to this message ]

ckbcouture@gmail.com wrote:
> HI.
> 
> So I have a problem with my /var partition.  I am unable to write to it
> and get "/var: write failed, file system is full" and "close failed:
> [Errno 28] No space left on device" when I do.
> 
> If I look at disk usage using df, I get:
> 
> Filesystem            1M-blocks     Used     Avail Capacity  Mounted on
> /dev/ld0a                  4585     2784      1571    63%    /
> /dev/ld0f                  4851     4617        -8   100%    /var
> /dev/ld0e                  9695     6129      3081    66%    /usr
> /dev/ld0g                  9695     2872      6337    31%    /home
> /dev/ld0h                 38788     9461     27386    25%    /www
> kernfs                        0        0         0   100%    /kern
> procfs                        0        0         0   100%
> /usr/pkg/emul/linux/proc
> elk:/www/staging          38788    10463     26384    28%
> /www/staging
> rhino:/www/production    268340    12743    241966     5%
> /www/production
> 
> If I look at /var using du, I get:
> 
> srvrs01# du -sPm /var
> 1125    /var
> 
> Clearly there is a discrepancy here and I can't find anything to delete
> to free up the disk space.  Any ideas on what is wrong and what I can
> do to fix it?

It could be that you have deleted the directory entry for an open file.
In such a case, the space taken up by the file won't be returned to the
filesystem until the file is closed (and thus, df(1) would show the
space as allocated), but the file (lacking a directory entry) won't
appear in a directory listing (and thus, du(1) won't show the space as
allocated).

Have you deleted any files (like those allocated by syslog) recently?
Have you recycled the processes that keep those files open?


-- 

Lew Pitcher, IT Specialist, Corporate Technology Solutions,
Enterprise Technology Solutions, TD Bank Financial Group

(Opinions expressed here are my own, not my employer's)

Attachment: signature.asc
This has been downloaded 0 time(s).

[ Post a follow-up to this message ]

Indeed this was the case, I had deleted named.run without restarting
named.

Thanks!
Cameron



Lew Pitcher wrote:
> ckbcouture@gmail.com wrote: 
>
> It could be that you have deleted the directory entry for an open file.
> In such a case, the space taken up by the file won't be returned to the
> filesystem until the file is closed (and thus, df(1) would show the
> space as allocated), but the file (lacking a directory entry) won't
> appear in a directory listing (and thus, du(1) won't show the space as
> allocated).
>
> Have you deleted any files (like those allocated by syslog) recently?
> Have you recycled the processes that keep those files open?
>
>
> --
>
> Lew Pitcher, IT Specialist, Corporate Technology Solutions,
> Enterprise Technology Solutions, TD Bank Financial Group
>
> (Opinions expressed here are my own, not my employer's)
>
>
> --------------enig5D08B35B5D979B26B4AE44A2
> Content-Type: application/pgp-signature
> Content-Transfer-Encoding: base64
> Content-Disposition: inline;
> 	filename="signature.asc"
> Content-Description: OpenPGP digital signature
> X-Google-AttachSize: 251

[ Post a follow-up to this message ]

Lew Pitcher wrote:
>
> It could be that you have deleted the directory entry for an open file.
> In such a case, the space taken up by the file won't be returned to the
> filesystem until the file is closed (and thus, df(1) would show the
> space as allocated), but the file (lacking a directory entry) won't
> appear in a directory listing (and thus, du(1) won't show the space as
> allocated).
>
> Have you deleted any files (like those allocated by syslog) recently?
> Have you recycled the processes that keep those files open?

BTW, the professional society for sysadmins is Usenix/SAGE
and they maintain an acheivement scale that goes Novice,
Junior, Intermediate/Advanced, Senior.  Each step is deliberately
more difficult than the previous.

The reasons that "df" and "du" output can differ are classic
qualifying questions to reach the Intermediate/Advanced level
of acheivement.  It's understanding how inodes and directories
work.  In this case see similar to /usr/include/sys/{v,i}node.h
for how an on-line inode gets mapped to memory when open
and the structures added to it.  The one in question is the
"reference count".  Both the "link count" and the "reference
count" need to go to zero before the backgroud process of
adding the blocks to the free chain starts.

Extra points for coming up with assorted other reasons that
"df" and "du" output differ.  ;^)

[ Post a follow-up to this message ]

"Doug Freyburger" <dfreybur@yahoo.com> writes:

> BTW, the professional society for sysadmins is Usenix/SAGE

Well, one of them is anyway.  Is anybody here besides me in LOPSA?

- Tim Skirvin (tskirvin@killfile.org)
--
http://www.killfile.org/~tskirvin/  Skirv's Homepage <FISH>< <*>
http://www.killfile.org/~tskirvin/software/     Skirv's Software

[ Post a follow-up to this message ]

Tim Skirvin wrote:
> Doug Freyburger writes:
> 
>
> Well, one of them is anyway.  Is anybody here besides me in LOPSA?

First I've heard of it but they seem a good idea.  Still taking charter
memberships?  I read through the list of founding members - I've
worked with Theo van Dinter and Benjy Feen directly and have dealt
with a fair number of the other folks indirectly over the years.  I
wish I had a quarter of Theo's talent, sigh.

[ Post a follow-up to this message ]

Tim Skirvin wrote:
>
>  Is anybody here besides me in LOPSA?

I went ahead and signed up.

[ Post a follow-up to this message ]

"Doug Freyburger" <dfreybur@yahoo.com> writes:
 
[vbcol=seagreen]
> I went ahead and signed up.

Did I get the super-ultra-nifty referral bonus?

- Tim Skirvin (tskirvin@killfile.org)
--
http://www.killfile.org/~tskirvin/  Skirv's Homepage <FISH>< <*>
http://tskirvin.livejournal.com/             Skirv's LiveJournal

[ Post a follow-up to this message ]

Tim Skirvin wrote:
> Doug Freyburger writes:
> 
> 
>
> Did I get the super-ultra-nifty referral bonus?

Depends on if I guessed your username right.  ;^)

[ Post a follow-up to this message ]