|
Home > Archive > Data Storage > December 2004 > Software vs hardware compression in backups
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
Software vs hardware compression in backups
|
|
| Charles Morrall 2004-12-10, 2:45 am |
| Backing up to disk is growing in popularity, and I for one certainly
recommend it in most cases. Another theory of mine is that it's more
favourable to use software compression when configuring the backup job
backing up to disk, and to configure the subsequent staging to tape with no
compression. I base this theory on the assumption that the compression
algorithms are well known, and essentially it's the same process regardless
if the algorithm is implemented in hardware in the tape drive or software in
the backup application. I doubt there's a an ASIC in the tape drive
dedicated to handle compression though. Also, it's probably safe to assume
that processing and memory capacity of a backup server is vastly larger that
of a tape drive even considering the server has to do many other things I've
come to the conclusion that using compression in software has a better
chance of compressing the data stream going to disk than a tape drive has,
when it has to try to stream the tape at the same time. With LTO-3 around
the corner with a native speed of 80MB/s, I doubt you could make the drive
stream with compression anyway in most cases.
Therefore it makes more sense to me to compress the incoming data stream to
disk, and let the tape drive stream the already compressed data using no
compression.
Your comments are, as always, much appreciated.
/charles
| |
| Yura Pismerov 2004-12-10, 5:45 pm |
|
I think it really depends on your particular setup - speed of the network, backup clients, tape drives, capacity of your backup
system, etc.
It also depends on your data. From my personal experience, running software (client side) compression was always better for some
reason - smaller backup frames and less tape media used.
But I should admit that it was not speediest network and tape drives in my case (Legato Networker in mixed gigabit/100MB network and
(4) AIT-3 drives).
Just my $0.02
Charles Morrall wrote:
> Backing up to disk is growing in popularity, and I for one certainly
> recommend it in most cases. Another theory of mine is that it's more
> favourable to use software compression when configuring the backup job
> backing up to disk, and to configure the subsequent staging to tape with no
> compression. I base this theory on the assumption that the compression
> algorithms are well known, and essentially it's the same process regardless
> if the algorithm is implemented in hardware in the tape drive or software in
> the backup application. I doubt there's a an ASIC in the tape drive
> dedicated to handle compression though. Also, it's probably safe to assume
> that processing and memory capacity of a backup server is vastly larger that
> of a tape drive even considering the server has to do many other things I've
> come to the conclusion that using compression in software has a better
> chance of compressing the data stream going to disk than a tape drive has,
> when it has to try to stream the tape at the same time. With LTO-3 around
> the corner with a native speed of 80MB/s, I doubt you could make the drive
> stream with compression anyway in most cases.
> Therefore it makes more sense to me to compress the incoming data stream to
> disk, and let the tape drive stream the already compressed data using no
> compression.
>
> Your comments are, as always, much appreciated.
> /charles
>
>
| |
| Rob Turk 2004-12-10, 5:45 pm |
| "Charles Morrall" <charles.morrall@telia.com> wrote in message
news:h0dud.10631$d5.94554@newsb.telia.net...
> Backing up to disk is growing in popularity, and I for one certainly
> recommend it in most cases. Another theory of mine is that it's more
> favourable to use software compression when configuring the backup job
> backing up to disk, and to configure the subsequent staging to tape with
> no compression.
Host based compression is usually more efficient because there's more buffer
memory for the host to play with and to anticipate data patterns then in the
tape drive itself. Also, when compression happens at the client before being
moved to the backup server and tape drive then it helps to eliminate
bottleneck elsewhere in the chain.
>I base this theory on the assumption that the compression algorithms are
>well known, and essentially it's the same process regardless if the
>algorithm is implemented in hardware in the tape drive or software in the
>backup application. I doubt there's a an ASIC in the tape drive dedicated
>to handle compression though.
That's a wrong assumption. ALDC/IDRC/LZW compression chips have been
available for years and are in common use by tape drive manufacturers. I
don't know of any tape drive doing it's compression in firmware, it's always
a dedicated ASIC or part of a larger buffer management chip.
> Also, it's probably safe to assume that processing and memory capacity of
> a backup server is vastly larger that of a tape drive even considering the
> server has to do many other things I've come to the conclusion that using
> compression in software has a better chance of compressing the data stream
> going to disk than a tape drive has, when it has to try to stream the tape
> at the same time.
Agreed.
>With LTO-3 around the corner with a native speed of 80MB/s, I doubt you
>could make the drive stream with compression anyway in most cases.
Yup. You'll need some really beefy hardware to keep them suckers happy.
> Therefore it makes more sense to me to compress the incoming data stream
> to disk, and let the tape drive stream the already compressed data using
> no compression.
>
> Your comments are, as always, much appreciated.
> /charles
Sounds like a good strategy. Added benefit is that an intermediate staging
disk will allow very fast restores for those "Whoops, I didn't mean to
delete that" incidents of the average PHB...
The only drawback is that you have to make absolutely sure that each next
version of your backup software is backward compatible with the compression
algorithm your software and disaster recovery systems use today.
Rob
| |
| Charles Morrall 2004-12-10, 5:45 pm |
|
"Rob Turk" <_wipe_me_r.turk@chello.nl> skrev i meddelandet
news:41b9e992$0$145$e4fe514c@dreader9.news.xs4all.nl...
> "Charles Morrall" <charles.morrall@telia.com> wrote in message
> news:h0dud.10631$d5.94554@newsb.telia.net...
>
>
> That's a wrong assumption. ALDC/IDRC/LZW compression chips have been
> available for years and are in common use by tape drive manufacturers. I
> don't know of any tape drive doing it's compression in firmware, it's
> always a dedicated ASIC or part of a larger buffer management chip.
>
I see. Thank's for setting that straight.
>
> Agreed.
>
Which is sort of was what I was getting at, a compression chip can't
outperform the spare CPU cycles and perhaps a GB of RAM of even a modest
server.
>
>
> Sounds like a good strategy. Added benefit is that an intermediate staging
> disk will allow very fast restores for those "Whoops, I didn't mean to
> delete that" incidents of the average PHB...
>
> The only drawback is that you have to make absolutely sure that each next
> version of your backup software is backward compatible with the
> compression algorithm your software and disaster recovery systems use
> today.
>
> Rob
>
>
Didn't think of that actually. However, as long as I do incremental upgrades
of the same software package I should be OK, and in case I'm switching
software I'll just make sure I destage the entire disk device to tape. Also,
there's no guarantee that the tape format (compressed or not) is compatible
anyway so it's not really a new problem, just a fresh angle of an old
problem. 
Thanks Rob!
/charles
| |
| Paul Rubin 2004-12-10, 5:45 pm |
| "Rob Turk" <_wipe_me_r.turk@chello.nl> writes:
> The only drawback is that you have to make absolutely sure that each next
> version of your backup software is backward compatible with the compression
> algorithm your software and disaster recovery systems use today.
I used to backup with a modified version of gnu tar. I set my backup
script to do the actual backup in one tape file, then automatically
dump the source code of the modified tar to a second tape file. So
every backup tape had a source code copy of the backup program in its
own tape file where it could be easily recovered by just about anything.
| |
| Maxim S. Shatskih 2004-12-16, 6:26 pm |
| Good compression algorithms like BWT or (especially) PPM have very bad
worst-case speed on uncompressible data, which means using them in streaming
environment a bit problematic. Yes, the host CPU is fast, but the tape drive is
too, and compression speed of the CPU can be slower then the drive speed.
Surely the tape drive's logic uses faster and worse algorithms like LZ or LZH.
They also do not require much memory (BWT requires starting from 1MB up).
Also - loading the server's CPU by compression can be unaffordable for some
scenarios.
--
Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
maxim@storagecraft.com
http://www.storagecraft.com
"Charles Morrall" <charles.morrall@telia.com> wrote in message
news:h0dud.10631$d5.94554@newsb.telia.net...
> Backing up to disk is growing in popularity, and I for one certainly
> recommend it in most cases. Another theory of mine is that it's more
> favourable to use software compression when configuring the backup job
> backing up to disk, and to configure the subsequent staging to tape with no
> compression. I base this theory on the assumption that the compression
> algorithms are well known, and essentially it's the same process regardless
> if the algorithm is implemented in hardware in the tape drive or software in
> the backup application. I doubt there's a an ASIC in the tape drive
> dedicated to handle compression though. Also, it's probably safe to assume
> that processing and memory capacity of a backup server is vastly larger that
> of a tape drive even considering the server has to do many other things I've
> come to the conclusion that using compression in software has a better
> chance of compressing the data stream going to disk than a tape drive has,
> when it has to try to stream the tape at the same time. With LTO-3 around
> the corner with a native speed of 80MB/s, I doubt you could make the drive
> stream with compression anyway in most cases.
> Therefore it makes more sense to me to compress the incoming data stream to
> disk, and let the tape drive stream the already compressed data using no
> compression.
>
> Your comments are, as always, much appreciated.
> /charles
>
>
| |
| prep@prep.synonet.com 2004-12-16, 6:26 pm |
| "Charles Morrall" <charles.morrall@telia.com> writes:
> Backing up to disk is growing in popularity, and I for one certainly
> recommend it in most cases. Another theory of mine is that it's more
> favourable to use software compression when configuring the backup
> job backing up to disk, and to configure the subsequent staging to
> tape with no compression.
see below
> I base this theory on the assumption that the compression algorithms
> are well known, and essentially it's the same process regardless if
> the algorithm is implemented in hardware in the tape drive or
> software in the backup application. I doubt there's a an ASIC in the
> tape drive dedicated to handle compression though.
This is in fact wrong for at least DLTs and 8mm drives. They do use
compression chips in the data path to do the compression.
I would recomend you do NOT compress you backups unless you can be
totally sure that it will not bite in unexpected ways. ie, DB files
that have large amounts of tempory data for there are multi TB of
blocks that are no longer zeros, and way less compressable! Needing
the space for both the compressed and the original data, and the
software! when recovering to a back up system...
--
Paul Repacholi 1 Crescent Rd.,
+61 (08) 9257-1001 Kalamunda.
West Australia 6076
comp.os.vms,- The Older, Grumpier Slashdot
Raw, Cooked or Well-done, it's all half baked.
EPIC, The Architecture of the future, always has been, always will be.
|
|
|
|
|