|
Home > Archive > Unix administration > December 2007 > In need of archiving software
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
In need of archiving software
|
|
| Faravid 2007-12-19, 7:32 am |
| Hello
Can anyone suggest me a good application for automated archiving of
data?
System it will be for: Debian based samba server.
Base: Console based, server/client based.
What it needs to be able to do:
- Possibly client/server system, server runs in the linux box and
client is in another computer prompting for the archiving and burning
the contents, with functions mentioned below (database for
recommending disk to be fetched).
- Prompt when a mounted drive is getting full and suggest operations.
- Burn old files to CD or DVD in the same format as in the drive (with
directory tree intact).
- Recommend what disk (CD/DVD) to be fetched when searching for
particular files or folders and create a printable report of the files
and folders in the disk.
Thanks in advance,
Janne
| |
| Russell Wood 2007-12-19, 7:32 am |
| man tar
On 2007-12-19, Faravid <faravid@gmail.com> wrote:
> Hello
>
> Can anyone suggest me a good application for automated archiving of
> data?
>
> System it will be for: Debian based samba server.
> Base: Console based, server/client based.
> What it needs to be able to do:
> - Possibly client/server system, server runs in the linux box and
> client is in another computer prompting for the archiving and burning
> the contents, with functions mentioned below (database for
> recommending disk to be fetched).
> - Prompt when a mounted drive is getting full and suggest operations.
> - Burn old files to CD or DVD in the same format as in the drive (with
> directory tree intact).
> - Recommend what disk (CD/DVD) to be fetched when searching for
> particular files or folders and create a printable report of the files
> and folders in the disk.
>
> Thanks in advance,
> Janne
--
Russell Wood
<http://www.dynode.net/~rjw/>
| |
| Faravid 2007-12-19, 7:32 am |
| How in the earth are you supposed to create a system I asked about
with just tar? Except coding an application to do the job, which I am
not going to do.
On 19 joulu, 11:48, Russell Wood <bitbuc...@dynode.net> wrote:
> man tar
>
> On 2007-12-19, Faravid <fara...@gmail.com> wrote:
>
>
>
>
>
>
>
> --
> Russell Wood
> <http://www.dynode.net/~rjw/>
| |
| Russell Wood 2007-12-19, 7:32 am |
| Bit of shell scripting + tar sounds about right to me. Maybe I don't
know what I'm talking about. Maybe you're ignorant and don't know what
your asking.
On 2007-12-19, Faravid <faravid@gmail.com> wrote:
> How in the earth are you supposed to create a system I asked about
> with just tar? Except coding an application to do the job, which I am
> not going to do.
>
> On 19 joulu, 11:48, Russell Wood <bitbuc...@dynode.net> wrote:
>
--
Russell Wood
<http://www.dynode.net/~rjw/>
| |
| Faravid 2007-12-19, 7:32 am |
| Sorry, thought you were fooling with me.
I thought there might have been a program to do this.
I guess I'll have to learn shell scripting then, is it really powerul
enough for my use? (moving only needed amount of old files with
directory tree intact and without taking the whole folder when newer
files exist and only when disk space is running out etc...)
I need truly automated system, I'm not able to monitor it continuously
and no one else in the firm got enough skills to do anything manually
in linux or advanced stuff in windows environment.
Got any directions to good documentation/tutorials to scripting?
On 19 joulu, 13:19, Russell Wood <bitbuc...@dynode.net> wrote:
> Bit of shell scripting + tar sounds about right to me. Maybe I don't
> know what I'm talking about. Maybe you're ignorant and don't know what
> your asking.
>
> On 2007-12-19, Faravid <fara...@gmail.com> wrote:
>
>
>
>
>
>
>
>
>
>
>
> --
> Russell Wood
> <http://www.dynode.net/~rjw/>
| |
| Doug Freyburger 2007-12-19, 1:23 pm |
| Faravid <fara...@gmail.com> wrote:
>
> Can anyone suggest me a good application for automated archiving of
> data?
What you ask for is called HSM - Hierarchical Storage Methodology.
The problem is HSM when done right requires rewriting the filesystem
layer to support an online/nearline/offline flag in the inode in
addition
to a database running in daemon mode to handle the media substitions.
Translation - You want a product that is quite expensive if you want
it
properly implemented.
> System it will be for: Debian based samba server.
> Base: Console based, server/client based.
> What it needs to be able to do:
> - Possibly client/server system, server runs in the linux box and
> client is in another computer prompting for the archiving and burning
> the contents, with functions mentioned below (database for
> recommending disk to be fetched).
> - Prompt when a mounted drive is getting full and suggest operations.
> - Burn old files to CD or DVD in the same format as in the drive (with
> directory tree intact).
> - Recommend what disk (CD/DVD) to be fetched when searching for
> particular files or folders and create a printable report of the files
> and folders in the disk.
Russell Wood suggested using tar. Some sort of script that runs
out of cron that does as you suggest and tar's the files in question
onto the DVDs. When it comes down to it this is not a bad
suggestion. It will cost you a lot less than a real HSM based
fileserver with a jukebox to automatically mount DVDs as needed.
I suggest a similar system that requires retrieving the data by hand.
Experiment with DVD burner software until you learn how to use the
command line to generate a data DVD that has a filesystem format
you know how to mount. ISO, reiserfs, ext2fs, whatever. Build a
script that uses find to identify candidates. Move them to a staging
directory. Replace them with a symbolic link that looks something
like "/offline.dvd/$SERIALOFDVD/$FILENAME". Then umount and
eject the previous DVD, burn the new one, mount it as
/offline.dvd/$SERIALOFDVD. You'll need a simple counting scheme
for the serial numbers, a jukebox that can handle ejecting a DVD
and so on. Plus something to generate a physical label when it
finally comes out of the physical box. One advantage to such a
scheme is you end up with the most recent migrations having
symbolic links that actually point to the currently mounted DVD
and the old ones have their location in the text of the symbolic link.
I've written similar scripts in the past "http://relocation.wopr.net/"
and I can tell you it's a large undertaking to pull off. If you only
handle "find -type f" files you make it much easier. It's a rabid
bear hungry fresh out of hiberation to get the symbolic links right if
you also try to handle symbolic links. It's a regular old bear in the
wild to handle directories if you want to figure out how to wrap up
directory trees. Stuff like pipes and random devices end up
almost as easy as regular files.
This method only works if you have some sort of aging active.
If you want to handle files recently modified you need to add a
layer of doing CRC checks or similar. And it fails utterly to handle
database files. In your stated use those aren't going to be a problem
but it remains a large project.
| |
| Thorbjoern Ravn Andersen 2007-12-19, 1:23 pm |
| Faravid <faravid@gmail.com> writes:
> Hello
>
> Can anyone suggest me a good application for automated archiving of
> data?
Look at amanda. Last time I looked at this it could do all this with
tapes. Perhaps DVD-burning has been added by now.
--
Thorbjørn Ravn Andersen
| |
| Pat Welch 2007-12-19, 1:23 pm |
| Faravid wrote:
> Hello
>
> Can anyone suggest me a good application for automated archiving of
> data?
>
> System it will be for: Debian based samba server.
> Base: Console based, server/client based.
> What it needs to be able to do:
> - Possibly client/server system, server runs in the linux box and
> client is in another computer prompting for the archiving and burning
> the contents, with functions mentioned below (database for
> recommending disk to be fetched).
> - Prompt when a mounted drive is getting full and suggest operations.
> - Burn old files to CD or DVD in the same format as in the drive (with
> directory tree intact).
> - Recommend what disk (CD/DVD) to be fetched when searching for
> particular files or folders and create a printable report of the files
> and folders in the disk.
>
> Thanks in advance,
> Janne
If you want a mature, proven commercial product, look at BackupEdge from
www.microlite.com.
I've used it on Unix systems for 20 years, and they have been making a
Linux Backup/emergency boot/bare metal restore version for the past 5 years.
It will backup over the network to any media supported by your Linux,
including NAS & iSCSI boxes, and can use SFTP to backup to any system
running any OS that has an FTP/SFTP server running.
You can get a 60 day, full featured demo at their web site to kick the
tires and make sure it does what you want.
Disclaimer: UBB is an authorized Edge reseller.
--
----------------------------------------------------
Pat Welch, UBB Computer Services, a WCS Affiliate
SCO Authorized Partner
Unix/Linux/Windows/Hardware Sales/Support
Temporary number! (209) 251-7201
(209) 745-1401 Cell: (209) 251-9120
E-mail: patubb@inreach.com
----------------------------------------------------
| |
| Moe Trin 2007-12-19, 7:27 pm |
| On Wed, 19 Dec 2007, in the Usenet newsgroup comp.unix.admin, in article
<760e38a5-2053-41d3-a24c-58dfa9a8d8f7@l1g2000hsa.googlegroups.com>, Faravid
wrote:
NOTE: Posting from groups.google.com (or some web-forums) dramatically
reduces the chance of your post being seen. Find a real news server.
[top posting fixed]
There are many Debian based distributions, but most come with a recent
copy of the HOWTOs and the 'Advanced Bash Scripting' guide from the
LDP. If you can find a real news server (thankfully, google doesn't
carry the group), have a look at the (very active) newsgroup
alt.os.linux.ubuntu, where there have been a number of recent threads
about system backup tools.
[vbcol=seagreen]
>I guess I'll have to learn shell scripting then, is it really powerul
>enough for my use? (moving only needed amount of old files with
>directory tree intact and without taking the whole folder when newer
>files exist and only when disk space is running out etc...)
"more powerful than you can possibly believe." quoting Obi-Wan out of
context - yes it can do all that you need. It's basically automating
what you can do from the command line. You are looking at 'find' and
'tar' by in large. Disk-space? Try 'df' as a test to see if you need
to do the find and tar routine.
>I need truly automated system, I'm not able to monitor it continuously
>and no one else in the firm got enough skills to do anything manually
>in linux or advanced stuff in windows environment.
Once it's debugged, you can make it a cron job, and virtually forget
about it.
>Got any directions to good documentation/tutorials to scripting?
-rw-rw-r-- 1 gferg ldp 31540 Jul 27 2000 Bash-Prog-Intro-HOWTO
which should be in /usr/share/HOWTO/ (or you can find it using any
search engine). Then grab a copy of
* Advanced Bash-Scripting Guide
version: 5.1
author: Mendel Cooper, <thegrendel(at)theriver.com>
last update: Nov 2007
available formats:
1. HTML (read online) 2. HTML (read online, single file, 1.9M)
3. HTML (tarred and gzipped package, 560K) 4. PDF (2.2M)
5. PostScript (1.2M) 6. text (513k)
7. PluckerDB (582k)
This document is both a tutorial and a reference on shell scripting
with Bash. It assumes no previous knowledge of scripting or
programming, but progresses rapidly toward an intermediate/advanced
level of instruction. The exercises and heavily-commented examples
invite active reader participation. Still, it is a work in progress.
The intention is to add much supplementary material in future updates
to this document, as it evolves into a comprehensive book that
matches or surpasses any of the shell scripting manuals in print.
which can be found at http://tldp.org/guides.html. It's an outgrowth
of the original Adv-Bash-Scr-HOWTO from 2000, _very_ much expanded. I'm
recommending it constantly.
Old guy
| |
| Dave Hinz 2007-12-20, 1:40 am |
| On Wed, 19 Dec 2007 04:31:10 -0800 (PST), Faravid <faravid@gmail.com> wrote:
> Sorry, thought you were fooling with me.
> I thought there might have been a program to do this.
Of course there is. AMANDA might be what you're looking for, check it
out.
> I guess I'll have to learn shell scripting then, is it really powerul
> enough for my use? (moving only needed amount of old files with
> directory tree intact and without taking the whole folder when newer
> files exist and only when disk space is running out etc...)
Home-grown scripts are rarely a match for an open-source project that
has 15 years of history behind it. Unless you're REALLY good.
> I need truly automated system, I'm not able to monitor it continuously
> and no one else in the firm got enough skills to do anything manually
> in linux or advanced stuff in windows environment.
Yeah, rolling your won, even if you're up to it, involves more
babysitting than that.
Take a look at amanda and see if that meets your needs. If not, are
your requirements unrealistic, or how does it not get you what you want?
| |
| Dave Hinz 2007-12-20, 1:40 am |
| On Wed, 19 Dec 2007 13:52:14 -0600, Moe Trin <ibuprofin@painkiller.example.tld> wrote:
> On Wed, 19 Dec 2007, in the Usenet newsgroup comp.unix.admin, in article
><760e38a5-2053-41d3-a24c-58dfa9a8d8f7@l1g2000hsa.googlegroups.com>, Faravid
> wrote:
>
> NOTE: Posting from groups.google.com (or some web-forums) dramatically
> reduces the chance of your post being seen. Find a real news server.
Well, I don't care so much about that, as:
> [top posting fixed]
....that. Top-posting makes quoting with context inconvenient, which
limits the number of people who will bother to respond to one's
questions.
| |
| Faravid 2007-12-20, 1:26 pm |
| Thanks to everyone, this helped me alot.
I'm going to take a look at amanda.
I have never used bash scripting as all I've needed to do with linux
in past is use web server via SSH or set my own web server, so it
would require alot of learning and building the system, it's a work
project and I'm not going to waste any of my own time in it + would
take too much time at work.
| |
| Moe Trin 2007-12-20, 7:21 pm |
| On 20 Dec 2007, in the Usenet newsgroup comp.unix.admin, in article
<5stvuvF1b2nqiU4@mid.individual.net>, Dave Hinz wrote:
>Moe Trin <ibuprofin@painkiller.example.tld> wrote:
>
>Well, I don't care so much about that
Because google absolutely refuses to handle abuse problems (you get
a boiler-plate "we can't do anything, but we're thinking" response
from their ignore-bot), people are killfiling anything with a
Message-ID ending in 'googlegroups.com'. At the moment, I'm only
doing so in six of the 84 groups I'm trying to scan daily, but it
has made a dramtic improvement in some groups. For example:
[compton ~]$ grep ^12.*linux.misc.*kill newslog | cut -d' ' -f1,4-6 | column
12/01/2007 58/58 (35 killed), 12/11/2007 77/77 (51 killed),
12/02/2007 60/60 (35 killed), 12/12/2007 75/75 (40 killed),
12/03/2007 104/104 (75 killed), 12/13/2007 116/116 (70 killed),
12/04/2007 81/81 (51 killed), 12/14/2007 93/93 (47 killed),
12/05/2007 70/70 (37 killed), 12/15/2007 106/106 (64 killed),
12/06/2007 51/51 (25 killed), 12/16/2007 48/48 (35 killed),
12/07/2007 54/54 (30 killed), 12/17/2007 94/94 (51 killed),
12/08/2007 40/40 (29 killed), 12/18/2007 78/78 (38 killed),
12/09/2007 79/79 (44 killed), 12/19/2007 60/60 (26 killed),
12/10/2007 82/82 (70 killed), 12/20/2007 40/40 (36 killed),
[compton ~]$
That's comp.os.linux misc. Most of it seems to be spam, or fake
spam advertisements (spammers are incredibly stupid, but I don't
think they are quite as st00pid as the idiots abusing google in
that group).
Old guy
| |
| Dave Hinz 2007-12-21, 1:49 am |
| On Thu, 20 Dec 2007 14:02:06 -0600, Moe Trin <ibuprofin@painkiller.example.tld> wrote:
> On 20 Dec 2007, in the Usenet newsgroup comp.unix.admin, in article
><5stvuvF1b2nqiU4@mid.individual.net>, Dave Hinz wrote:
>
>
[vbcol=seagreen]
[vbcol=seagreen]
> Because google absolutely refuses to handle abuse problems (you get
> a boiler-plate "we can't do anything, but we're thinking" response
> from their ignore-bot), people are killfiling anything with a
> Message-ID ending in 'googlegroups.com'.
Your choice, of course. Maybe it's just that my newsserver
(news.individual.net, 15 bucks a year) is really good at filtering spam
but, I don't block anything from googlegroups, and I wasn't aware of a
spam problem from there.
> At the moment, I'm only
> doing so in six of the 84 groups I'm trying to scan daily, but it
> has made a dramtic improvement in some groups. For example:
(snip)
> That's comp.os.linux misc. Most of it seems to be spam, or fake
> spam advertisements (spammers are incredibly stupid, but I don't
> think they are quite as st00pid as the idiots abusing google in
> that group).
Hm. Maybe a valid point but, with my newsfeed, I don't have to throw
the baby out with the bathwater.
| |
| Moe Trin 2007-12-21, 7:22 pm |
| On 21 Dec 2007, in the Usenet newsgroup comp.unix.admin, in article
<5t0itdF1bb0g9U1@mid.individual.net>, Dave Hinz wrote:
>Your choice, of course. Maybe it's just that my newsserver
>(news.individual.net, 15 bucks a year) is really good at filtering spam
>but, I don't block anything from googlegroups, and I wasn't aware of a
>spam problem from there.
The spam problem kicked up this summer. Before that, there were the
occasional spammer, but nothing difficult to handle. It appeared
initially as some supposedly Chinese manufacturer offering athletic
shoes, and quickly turned to multiple posts of similar ads (it
definitely wasn't a stuck 'P' key). Quicky check showed the posting
hosts were open relays, and it appears to be someone with to much
time on their hand. It's since expanded to other spam products like
pills and tobacco. In August, I reviewed my spool, and found that
there were few useful posts coming through google in that newsgroup
(ham/spam was running worse than 1:25), so I removed roughly 40 rules
from the killfile, and added just one - for that group only. I've
since added five other groups, though for different reasons.
>Hm. Maybe a valid point but, with my newsfeed, I don't have to throw
>the baby out with the bathwater.
Know what you are saying - in this case, I'm not seeing anything in
the trash (I do look, using an on-line reader, rather than what gets
killed off with slrnpull which is run from a cron-job).
Old guy
| |
| Dave Hinz 2007-12-21, 7:22 pm |
| On Fri, 21 Dec 2007 13:59:37 -0600, Moe Trin <ibuprofin@painkiller.example.tld> wrote:
> On 21 Dec 2007, in the Usenet newsgroup comp.unix.admin, in article
><5t0itdF1bb0g9U1@mid.individual.net>, Dave Hinz wrote:
>
[vbcol=seagreen]
> The spam problem kicked up this summer. Before that, there were the
> occasional spammer, but nothing difficult to handle. It appeared
> initially as some supposedly Chinese manufacturer offering athletic
> shoes,
Ahhh, counterfeit chinese goods. Probably complete with lead paint.
Flogging my own site here but, http://www.productrecallwatch.com takes a
bunch of RSS feeds and puts 'em together into one place. 20-some last
week, 30-some the week before. The press reported 2 of 'em. And check
out the tagcloud, see which is biggest. (Hint: China).
i> and quickly turned to multiple posts of similar ads (it
> definitely wasn't a stuck 'P' key). Quicky check showed the posting
> hosts were open relays, and it appears to be someone with to much
> time on their hand. It's since expanded to other spam products like
> pills and tobacco. In August, I reviewed my spool, and found that
> there were few useful posts coming through google in that newsgroup
> (ham/spam was running worse than 1:25), so I removed roughly 40 rules
> from the killfile, and added just one - for that group only. I've
> since added five other groups, though for different reasons.
I'm glad my upstream is dealing with it, so I don't have to.
[vbcol=seagreen]
> Know what you are saying - in this case, I'm not seeing anything in
> the trash (I do look, using an on-line reader, rather than what gets
> killed off with slrnpull which is run from a cron-job).
|
|
|
|
|