Debian Developers - Bug#278289: ITP: apt-dupdate -- diff-based update of APT's index files

Author

Bug#278289: ITP: apt-dupdate -- diff-based update of APT's index files

Eduard Bloch

2004-10-25, 5:52 pm

Package: wnpp
Severity: wishlist

* Package name : apt-dupdate
Version : 0.0.1
Upstream Author : me
* License : BSD
Description : diff-based update of APT's index files

apt-dupdate is beeing written right now. The server part is almost done,
the client should be not a big deal. apt-dupdate is similar to the
unofficial apt-pupdate package and uses the same base idea: the
Packages/Sources files are updated using diff/patch files to the
previous version which is already downloaded. The advantage is
impressing: daily diffs for Sid are about 35KiB in size (average,
bzip2ed) compared to >=3MiB downloaded by APT (for main, gziped).

The advantages of apt-dupdate compared to apt-pupdate:

- it will actually work. bjb seems to be MIA since the big server
compromise, the diff update scripts on people.d.o are apparently not
restored
- in difference to apt-pupdate, I do not use chains of small diffs,
based on days and managed by the server. Instead, the server provides
the patch for a md5sum which contains the diffs between the version
of the client and the current one (when the server has data for the
old version, of course)
- it will be faster on high-latency links. apt-pupdate AFAICS needed
n*2+2 http requests where n=#days. apt-dupdate will use exactly one
request.
- it will be faster on low-speed links since one chunk of data
compresses much better than n small chunks. apt-dupdate will use
bzip2 instead of gzip.

-- System Information:
Debian Release: 3.1
APT prefers unstable
APT policy: (500, 'unstable'), (500, 'testing'), (1, 'experimental')
Architecture: i386 (i686)
Kernel: Linux 2.6.9
Locale: LANG=de_DE.UTF-8, LC_CTYPE=de_DE.UTF-8

--
To UNSUBSCRIBE, email to debian-bugs-dist-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Adrian 'Dagurashibanipal' von Bidder

2004-10-27, 2:49 am

Eduard Bloch

2004-10-27, 8:47 pm

Moin Adrian!
Adrian 'Dagurashibanipal' von Bidder schrieb am Mittwoch, den 27. Oktober 2004:

> On Monday 25 October 2004 23.47, Eduard Bloch wrote:
>
>
> How much server load do you anticipate?

The prototype implementation [1] is quite stupid and needs a while.
Dominant factor is bzip2 speed, calculate with n^2/2*(number of
arches)*(number of branches)*200kB - that is the estimated ammount of
data to be recompress plus base files to be checksummed. Eg. ~40MiB for
stable+testing+unstable (all+i386). I expect less than 1-2 minutes CPU
time for P4-2Ghz, 30days history, 12 arches, unstable+testing.

However, I think about rewritting the current thingie to keep only
patches between the iterations and not store the final patch on the
server. Instead, a CGI script would merge them on demand (and cache the
resulted file somewhere so it may be reused). This would save HD space
and CPU cycles on long term.

> Looking forward to trying this. Is this a tool you plan to deploy on the
> official mirrors/on a public Debian machine/... or is it a tool for people
> with many machines to use on their internal Debian mirror?

You can install it on a mirror and create the diff-chain yourself. It
should be full configurable.

[1] http://people.debian.org/~blade/dupdate.pl

Regards,
Eduard.
--
<GyrosGeier> doogie, 25 m/s is pretty fast
<doogie> 40m/s from apache
<doogie> 25m/s is from java
<GyrosGeier> doogie, that's about 8 km/h.
* GyrosGeier ducks
<smurfix> GyrosGeier: Wrong. Other way round please.
<smurfix> 93.6 km/h
....
<GyrosGeier> smurfix, okay 10 pm
<GyrosGeier> smurfix, but 93 km/h proves my point that it is pretty fast

--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org