|
Home > Archive > Debian Developers > February 2004 > Where should rrd files generated by a program be stored ?
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
Where should rrd files generated by a program be stored ?
|
|
| Jean-Michel Kelbert 2004-02-15, 10:33 pm |
| Hi,
I have a discussion with my usptream (amavis-stats) where rrd files
should be stored. (This mail is an adaptation of a mail send by the
upstram.)
In the FHS, there is :
/var/cache is intended for cached data from applications. Such data is
locally generated as a result of time-consuming I/O or calculation. The
application must be able to regenerate or restore the data. Unlike
/var/spool, the cached files can be deleted without data loss. The data
must remain valid between invocations of the application and rebooting
the system.
Files located under /var/cache may be expired in an application specific
manner, by the system administrator, or both. The application must always
be able to recover from manual deletion of these files (generally because
of a disk space shortage). No other requirements are made on the data
format of the cache directories.
The rrd files are not stored on disk to save time or calculations. They
are the output of the program. There is no other reason for someone to
install amavis-stats other than to view the graphs generated from the
rrds.
Since the amavis log files are rotated, and eventually removed there is no
way to re-create the data.
/var/lib however has the following purpose:
This hierarchy holds state information pertaining to an application or
the system. State information is data that programs modify while they run,
and that pertains to one specific host. Users must never need to modify
files in /var/lib to configure a package's operation.
State information is generally used to preserve the condition of an
application (or a group of inter-related applications) between
invocations and between different instances of the same application.
State information should generally remain valid after a reboot, should
not be logging output, and should not be spooled data.
Therefore rrd files should be stored in /var/lib, isn't it ?
However when looking to cacti, or mailgraph, they store rrd files in
/var/lib.
Any suggestion ?
Thanks.
--
Jean-Michel Kelbert
--
To UNSUBSCRIBE, email to debian-devel-request@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
| |
| Oliver Kurth 2004-02-15, 10:33 pm |
| On Mon, 2004-02-16 at 12:02, Jean-Michel Kelbert wrote:
> Therefore rrd files should be stored in /var/lib, isn't it ?
> However when looking to cacti, or mailgraph, they store rrd files in
> /var/lib.
ntop stores its rrd files in /var/lib/. I also think this is the correct
location.
Greetings,
Oliver
| |
| Mark Lawrence 2004-02-15, 11:33 pm |
| On Mon, 16 Feb 2004, Jean-Michel Kelbert wrote:
> Therefore rrd files should be stored in /var/lib, isn't it ?
> However when looking to cacti, or mailgraph, they store rrd files in
> /var/lib.
Actually you meant to write, that cacti or mailgraph packages store the
rrds in /var/*cache*.
Cheers, Mark.
--
Mark Lawrence (nomad@null.net)
--
To UNSUBSCRIBE, email to debian-devel-request@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
| |
| Jean-Michel Kelbert 2004-02-15, 11:33 pm |
| Le 16/02/04 à 12:48 Oliver Kurth (okurth@gmx.net) écrivait :
> On Mon, 2004-02-16 at 12:02, Jean-Michel Kelbert wrote:
>
>
> ntop stores its rrd files in /var/lib/. I also think this is the correct
> location.
Ok !
Therefore a bug report should be openned on packages storing rrd files
in /var/cache, isn't it ?
--
Jean-Michel Kelbert
| |
| Oliver Kurth 2004-02-16, 12:33 am |
| On Mon, 2004-02-16 at 13:36, Jean-Michel Kelbert wrote:
> Le 16/02/04 à 12:48 Oliver Kurth (okurth@gmx.net) écrivait :
>
> Ok !
>
> Therefore a bug report should be openned on packages storing rrd files
> in /var/cache, isn't it ?
Although I think this is dependending on the exact purpose of the rrd
files, I would say, yes. I would like to mount /var/cache to a ram disk,
but this is not possible, because debconf stores its data there - there
is/was a bug filed about that in debconf, but I cannot find that
anymore.
Greetings,
Oliver
| |
| sean finney 2004-02-16, 1:33 am |
| On Mon, Feb 16, 2004 at 01:36:33PM +0100, Jean-Michel Kelbert wrote:
i agree that /var/lib would be the more appropriate of the two places.
/var/cache is supposedly for stuff that can be deleted without
data-loss, which isn't the case for rrd files (unless they can be
completely generated from another data source).
[color=blue]
> Therefore a bug report should be openned on packages storing rrd files
> in /var/cache, isn't it ?
definitely.
sean
| |
| Goswin von Brederlow 2004-02-16, 2:38 am |
| Oliver Kurth <okurth@gmx.net> writes:
> On Mon, 2004-02-16 at 13:36, Jean-Michel Kelbert wrote:
>
> Although I think this is dependending on the exact purpose of the rrd
> files, I would say, yes. I would like to mount /var/cache to a ram disk,
> but this is not possible, because debconf stores its data there - there
> is/was a bug filed about that in debconf, but I cannot find that
> anymore.
If you keep the directory structure that should work. And regarding
debconf, it would just ask the questions again on the next update. Its
operation is not compromised by loosing the DB, just its comfort. Or
not?
MfG
Goswin
--
To UNSUBSCRIBE, email to debian-devel-request@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
| |
| Oliver Kurth 2004-02-16, 2:38 am |
| On Mon, 2004-02-16 at 16:12, Goswin von Brederlow wrote:
> Oliver Kurth <okurth@gmx.net> writes:
>
> If you keep the directory structure that should work. And regarding
> debconf, it would just ask the questions again on the next update. Its
> operation is not compromised by loosing the DB, just its comfort. Or
> not?
True. But I do not want to loose the comfort ;-), ie. I do not want to
answer all these questions again and again on each upgrade.
Greetings,
Oliver
| |
| Goswin von Brederlow 2004-02-16, 2:38 am |
| Oliver Kurth <okurth@gmx.net> writes:
> On Mon, 2004-02-16 at 16:12, Goswin von Brederlow wrote:
>
>
> True. But I do not want to loose the comfort ;-), ie. I do not want to
> answer all these questions again and again on each upgrade.
>
> Greetings,
> Oliver
Then don't delete the cache. 
I suggest using tmpfs instead of a ramdisk too.
MfG
Goswin
--
To UNSUBSCRIBE, email to debian-devel-request@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
| |
| Oliver Kurth 2004-02-16, 2:38 am |
| On Mon, 2004-02-16 at 16:38, Goswin von Brederlow wrote:
> Oliver Kurth <okurth@gmx.net> writes:
>
[color=blue]
> Then don't delete the cache.
Sigh. I want to mount /var/cache to a ram disk. Then /var/cache/debconf
is also on ram disk. So it's gone when I reboot. To save it, I would
have to tweak it, eg. with bind mounting, so that /var/cache/debconf is
on disk again. This trouble would not be necessary if the debconf DB was
in /var/lib in the first place.
> I suggest using tmpfs instead of a ramdisk too.
Agreed. It is not relevant for the problem though.
Greetings,
Oliver
| |
| Goswin von Brederlow 2004-02-16, 3:33 am |
| Oliver Kurth <okurth@gmx.net> writes:
> On Mon, 2004-02-16 at 16:38, Goswin von Brederlow wrote:
>
>
> Sigh. I want to mount /var/cache to a ram disk. Then /var/cache/debconf
> is also on ram disk. So it's gone when I reboot. To save it, I would
> have to tweak it, eg. with bind mounting, so that /var/cache/debconf is
> on disk again. This trouble would not be necessary if the debconf DB was
> in /var/lib in the first place.
You have to store and restore the directory structure and permissions
on every reboot too. Its not ment to be lost completly on a regular
basis.
>
> Agreed. It is not relevant for the problem though.
>
> Greetings,
> Oliver
MfG
Goswin
--
To UNSUBSCRIBE, email to debian-devel-request@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
| |
| Tim Dijkstra 2004-02-16, 3:33 am |
| On Mon, 16 Feb 2004 16:46:47 +0100
Oliver Kurth <okurth@gmx.net> wrote:
> On Mon, 2004-02-16 at 16:38, Goswin von Brederlow wrote:
>
>
> Sigh. I want to mount /var/cache to a ram disk. Then
> /var/cache/debconf is also on ram disk. So it's gone when I reboot. To
> save it, I would have to tweak it, eg. with bind mounting, so that
> /var/cache/debconf is on disk again. This trouble would not be
> necessary if the debconf DB was in /var/lib in the first place.
I think the debconf creator wants to emphasise the fact that debconf
database is not a registry. So I don't think you can convince him to put
it in /var/lib.
grts Tim
--
To UNSUBSCRIBE, email to debian-devel-request@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
| |
| Oliver Kurth 2004-02-16, 3:33 am |
| On Mon, 2004-02-16 at 17:34, Tim Dijkstra wrote:
> On Mon, 16 Feb 2004 16:46:47 +0100
> Oliver Kurth <okurth@gmx.net> wrote:
>
>
> I think the debconf creator wants to emphasise the fact that debconf
> database is not a registry. So I don't think you can convince him to put
> it in /var/lib.
Well, it seems that debconf can be configured to put the files somewhere
else by editing /etc/debconf.conf.
I should have looked there before I started this sub-thread. Sorry.
Greetings,
Oliver
| |
| Marco d'Itri 2004-02-16, 5:34 am |
| On Feb 16, Jean-Michel Kelbert <kelbert@debian.org> wrote:
>Therefore a bug report should be openned on packages storing rrd files
>in /var/cache, isn't it ?
Yes.
--
ciao, |
Marco | [4621 sba92rWU47.Ro]
| |
| Norbert Tretkowski 2004-02-16, 7:34 am |
| * Jean-Michel Kelbert wrote:
> Therefore a bug report should be openned on packages storing rrd files
> in /var/cache, isn't it ?
I don't agree.
,----[ FHS 5.2.1 ]---
| /var/cache is intended for cached data from applications. Such data is
| locally generated as a result of time-consuming I/O or calculation. The
| application must be able to regenerate or restore the data. Unlike
| /var/spool, the cached files can be deleted without data loss. The data
| should remain valid between invocations of the application and
| rebooting the system.
`----
That's why I moved mailgraph's rrd files from /var/lib to /var/cache.
Norbert
--
To UNSUBSCRIBE, email to debian-devel-request@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
| |
| Jean-Michel Kelbert 2004-02-16, 9:34 am |
| Le 16/02/04 à 22:07 Norbert Tretkowski (tretkowski@inittab.de) écrivait:
> | /var/spool, the cached files can be deleted without data loss. The data
> | should remain valid between invocations of the application and
> | rebooting the system.
> `----
>
> That's why I moved mailgraph's rrd files from /var/lib to /var/cache.
It's impossible to rebuild without data loss, since there is log
rotation.
--
Jean-Michel Kelbert
| |
| Norbert Tretkowski 2004-02-16, 11:33 am |
| * Jean-Michel Kelbert wrote:
> Le 16/02/04 à 22:07 Norbert Tretkowski (tretkowski@inittab.de) écrivait :
>
> It's impossible to rebuild without data loss, since there is log
> rotation.
OK, misunderstanding. I jumbled something here.
Currently, mailgraph stores it's graphs and his rrd files under
/var/cache/mailgraph, but after this thread and a discussion with
Alexander Reelsen on IRC I decided to move the rrd files to
/var/lib/mailgraph. The graphs will stay in /var/cache/mailgraph.
Norbert
--
To UNSUBSCRIBE, email to debian-devel-request@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
| |
| sean finney 2004-02-16, 11:33 am |
| hi norbert,
On Tue, Feb 17, 2004 at 01:07:59AM +0100, Norbert Tretkowski wrote:
> Currently, mailgraph stores it's graphs and his rrd files under
> /var/cache/mailgraph, but after this thread and a discussion with
> Alexander Reelsen on IRC I decided to move the rrd files to
> /var/lib/mailgraph. The graphs will stay in /var/cache/mailgraph.
which i'd agree is the right thinking. the rrd's can't be reliably
regenerated thanks to logrotation (your logs will only go so far back and
even then it's a pita to get all of that loaded back into the rrd, i had
to do just that a couple weeks ago) the graphs (png images), however,
can be quickly regenerated from the rrd file, so it makes sense to put
them in /var/cache.
sean
| |
| Matt Zimmerman 2004-02-16, 11:34 am |
| On Mon, Feb 16, 2004 at 01:36:33PM +0100, Jean-Michel Kelbert wrote:
> Le 16/02/04 ? 12:48 Oliver Kurth (okurth@gmx.net) ?crivait :
>
> Ok !
>
> Therefore a bug report should be openned on packages storing rrd files
> in /var/cache, isn't it ?
Not necessarily; the correct location depends on how the program uses the
files, not their file format. Some RRDs would correctly belong in
/var/cache.
--
- mdz
--
To UNSUBSCRIBE, email to debian-devel-request@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
| |
| Julian Mehnle 2004-02-24, 6:34 am |
| sean finney <seanius@seanius.net> wrote:
> On Tue, Feb 17, 2004 at 01:07:59AM +0100, Norbert Tretkowski wrote:
>
> which i'd agree is the right thinking. the rrd's can't be reliably
> regenerated thanks to logrotation (your logs will only go so far back
> and even then it's a pita to get all of that loaded back into the rrd, i
> had to do just that a couple weeks ago) the graphs (png images),
> however, can be quickly regenerated from the rrd file, so it makes sense
> to put them in /var/cache.
Regarding storing the graphs in /var/cache, I disagree. For files in
/var/cache, the FHS mandates:
| The application must always be able to recover from manual deletion of
| these files (generally because of a disk space shortage).
The problem is that the graphs probably are *not* automatically
re-generated if they are missing when a user requests them via the web
server. So, the graphs would probably best stored in /var/lib, too.
--
To UNSUBSCRIBE, email to debian-devel-request@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
| |
| sean finney 2004-02-24, 8:34 am |
| hi julian,
On Tue, Feb 24, 2004 at 08:50:36PM +0100, Julian Mehnle wrote:
> Regarding storing the graphs in /var/cache, I disagree. For files in
> /var/cache, the FHS mandates:
>
> | The application must always be able to recover from manual deletion of
> | these files (generally because of a disk space shortage).
>
> The problem is that the graphs probably are *not* automatically
> re-generated if they are missing when a user requests them via the web
> server. So, the graphs would probably best stored in /var/lib, too.
with the example of mailgraph, you *can* safely delete the png images.
if you look at the mail graph statistics page it generates, the images
are output from a cgi script, which re-creates the images on the fly
if they're missing or need to be updated.
if you want to link to the static images on the web server, i think
what you'd really want to do is rsync them out of there and into your
webserver's document root. you could even make sure the images were
present and up to date by running wget on the cgi script before copying
them.
i've actually sent a patch to the upstream of mailgraph around a month
ago that gives the ability to generate the from the cmdline/cron and
place them somewhere else, which i think is what you'd really want in
a case where you linked to the static images on the webserver.
so anyway, i think there probably are cases where you are right that
the images shouldn't go in /var/cache, and that this does need to
be a question answered on a package by package basis, but i can
certainly also think of situations where that's exactly where you want
them to be.
sean
|
|
|
|
|