|
Home > Archive > Debian Developers > April 2004 > Re: Apt and rsync... I know...
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
Re: Apt and rsync... I know...
|
|
| Otto Wyss 2004-01-31, 12:34 am |
| Sorry for being late but don't read debian-devel regularly.
quote:
> Surely. First, arrange for all of the packages in Debian to be compressed
> using gzip --rsyncable. Then, find a server with gobs of disk, network, I/O
> and CPU resources, mirror the Debian archive, and run an anonymous rsync
> server. Then, write an rsync method for apt, or use rproxy, or whatever.
>
Since about 2 years there's a solution for this except for one thing:
"gzip --rsyncable". And with autobuilder not even that should be a
problem (see: "http://sourceforge.net/projects/dpartialmirror").
O. Wyss
--
See "http://wxguide.sourceforge.net/" for ideas how to design your app.
--
To UNSUBSCRIBE, email to debian-devel-request@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
| |
| Florian Weimer 2004-02-01, 5:34 am |
| Doug Holland wrote:
quote:
> I suggest that rsync files be precalculated, so rsync downloads don't
> have to be crunched on the fly. The servers would store .deb files -
> foo-x.y.z.deb, and they would store the rsync diffs between it and the
> previous version - foo.x.y.z-1_x.y.z.rsdeb. That way, if the user
> doing an apt-get upgrade has the previous .deb file in his cache, apt
> would download the rsync diff file instead of the full .deb, saving
> loads of bandwidth, and since the rsyncs are precomputed and cached,
> the servers don't get hosed.
I'm not sure if rsync is the right format for that. VCDIFF (RFC 3284)
should result in smaller deltas and less CPU overhead. However, I don't
know of any free implementation (Subversion has code that does something
very similar, though). The format itself is not encumbered by patents,
AFAIK.
--
To UNSUBSCRIBE, email to debian-devel-request@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
| |
| Florian Weimer 2004-03-16, 6:38 pm |
| Florian Weimer wrote:
> I'm not sure if rsync is the right format for that. VCDIFF (RFC 3284)
> should result in smaller deltas and less CPU overhead. However, I don't
> know of any free implementation (Subversion has code that does something
> very similar, though). The format itself is not encumbered by patents,
> AFAIK.
Just for reference, here are a few numbers. I compared the data.tar
part of libc6_2.3.2.ds1-10 and libc6_2.3.2.ds1-11 (i386 architecture).
Delta sizes are:
rdiff 7302952
rdiff (gzip -9) 2423336
svndelta 1793107
svndelta (gzip -9) 1539271
xdelta (-0) 1751110
xdelta (-9) 935281
svndelta is quite CPU-intensive, compared to the others. (I'm not sure
if it's because of the base64 encoding, which isn't included in the
numbers above.) xdelta and rdiff use comparable amounts of CPU time
(with and without compression). xdelta seems to need memory proportional
to the size of the input files, which might be a problem.
--
Current mail filters: many dial-up/DSL/cable modem hosts, and the
following domains: atlas.cz, bigpond.com, freenet.de, hotmail.com,
libero.it, netscape.net, postino.it, tiscali.co.uk, tiscali.cz,
tiscali.it, voila.fr, wanadoo.fr, yahoo.com.
--
To UNSUBSCRIBE, email to debian-devel-request@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
| |
| Goswin von Brederlow 2004-03-27, 11:33 pm |
| Florian Weimer <fw@deneb.enyo.de> writes:
> Florian Weimer wrote:
>
>
> Just for reference, here are a few numbers. I compared the data.tar
> part of libc6_2.3.2.ds1-10 and libc6_2.3.2.ds1-11 (i386 architecture).
>
> Delta sizes are:
>
> rdiff 7302952
> rdiff (gzip -9) 2423336
> svndelta 1793107
> svndelta (gzip -9) 1539271
> xdelta (-0) 1751110
> xdelta (-9) 935281
>
> svndelta is quite CPU-intensive, compared to the others. (I'm not sure
> if it's because of the base64 encoding, which isn't included in the
> numbers above.) xdelta and rdiff use comparable amounts of CPU time
> (with and without compression). xdelta seems to need memory proportional
> to the size of the input files, which might be a problem.
And what does rsync say? Ucompressed, gzip -9 and gzip with
--rsyncable please.
MfG
Goswin
--
To UNSUBSCRIBE, email to debian-devel-request@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
| |
| Florian Weimer 2004-04-03, 4:36 pm |
| Goswin von Brederlow wrote:
rsync 7654580
rsync (gzip -9) 2487882
Numbers were obtained using the experimental batch mode. Only the delta
file size is included. Not surprisingly, they are quite similar to
rdiff.
[color=darkred]
xdelta needs all data in core, and is suitable for small patches only.
[color=darkred]
> And what does rsync say? Ucompressed, gzip -9 and gzip with
> --rsyncable please.
Why do you want rsyncable rsync deltas?
--
Current mail filters: many dial-up/DSL/cable modem hosts, and the
following domains: postino.it, tiscali.co.uk, tiscali.cz, tiscali.it,
voila.fr.
--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
| |
| Goswin von Brederlow 2004-04-03, 7:33 pm |
| Florian Weimer <fw@deneb.enyo.de> writes:
> Goswin von Brederlow wrote:
>
> rsync 7654580
> rsync (gzip -9) 2487882
>
> Numbers were obtained using the experimental batch mode. Only the delta
> file size is included. Not surprisingly, they are quite similar to
> rdiff.
>
>
> xdelta needs all data in core, and is suitable for small patches only.
>
>
> Why do you want rsyncable rsync deltas?
No, rsync deltas on the data.tar.gz with --rsyncable. I.e. what one
would get on a plain deb if dpkg used --rsyncable.
MfG
Goswin
--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
| |
| Florian Weimer 2004-04-04, 7:34 am |
| Goswin von Brederlow wrote:
> No, rsync deltas on the data.tar.gz with --rsyncable. I.e. what one
> would get on a plain deb if dpkg used --rsyncable.
It's not really worth it in this case. The rsync delta is 3329453 bytes
(3316181 compressed). The new package version is 4716113 bytes large.
--
Current mail filters: many dial-up/DSL/cable modem hosts, and the
following domains: postino.it, tiscali.co.uk, tiscali.cz, tiscali.it,
voila.fr.
--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
|
|
|
|
|