Dear Friends,

the idea to have some sort of incremental update support for the
archive index files (Packages,Sources) in the archive and in apt has
been around for some time now [1].

Anthony Towns analysed the problem in [2] and came up with the idea to
use ed-style diffs to solve the problem.

Andreas Barth implemented the server side of the index diffs
generation and has a test-repository (with only the index-files) at
[3].

I'm happy to tell you that apt is able to use those index files
now. Robert Lemmen and I implmeneted the needed support.

This massively speeds up a "apt-get update". If you e.g. update daily
you will have to get only ~15kb-30kb worth of update information per
day (instead of 2,7mb for the complete Packages.bz2 as it is now).

We believe that the code is now "good" enough for wider testing. I
have setup a repository for the patched version of apt with the pdiff
support.  Just add this line to your sources.list file:
"deb http://people.debian.org/~mvo/apt/pdiffs/ /"

When fully supported by the archive, the diff support will be
completely transparent, no changes on your side necessary. In the
moment however, the archive does not carry the diff files, so a little
hack is needed to test this. The diffs are on merkel.debian.org, so change y
our
sources.list to point there instead of your usual mirror:
"http://merkel.debian.org/~aba/debian/"

E.g. from:
deb http://ftp.de.debian.org/debian sid main
to
deb http://merkel.debian.org/~aba/debian sid main


There are two issues with using merkel:
1. There is no Release/Release.gpg file on merkel so you get "not
authenticated" warnings
2. merkel does not have the actual package files

To work around (2) and make it possible to still download packages
(even with merkel as the only entry in sources.list) a hack was added
to apt called: "APT::URL-Remap::". This allows one to remap a
URI. Example: apt-get install 3dchess -o
APT::URL-Remap::[url]http://merkel.debian.org/~aba/debian/=http://ftp.de.debian.org/d[/
url]
+ebian/"

will ensure that archives pointing to merkel are actually fetched from
ftp.de.debian.org. Please note that this hack will be removed once
there are servers available with both index diffs and packages (it's
just to make testing easier).

To work around the authentication issue you can use the
"--allow-unauthenticated" switch in apt, this will suppress the
authentication checking. Again once we have full support for the diffs
in the archive the authentification will work (and be checked).

Known issues:
- the progress-bar is a bit jumpy
- the total transfered data information (at the end of the update)
is totally incorrect

Please report success/failure/problems directly to me. There are two
debug switches that should be used if you are having trouble:

-o Debug::pkgAcquire::Diffs=true
-o Debug::pkgAcquire::RRed=true

The source code of the patch is availab in my:
michael.vogt@ubuntu.com--2005/apt--pdiff--0 [4]
baz repository.

Cheers,
Michael

[1] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=128818
[2] http://azure.humbug.org.au/~aj/blog/2003/12/02
[3] http://merkel.debian.org/~aba/debian/
[4] at http://people.ubuntu.com/~mvo/arch/ubuntu
--
Linux is not The Answer. Yes is the answer. Linux is The Question. - Neo


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.or
g

[ Post a follow-up to this message ]

Michael Vogt <mvo@debian.org> wrote:
[...]
> 2. merkel does not have the actual package files

> To work around (2) and make it possible to still download packages
> (even with merkel as the only entry in sources.list) a hack was added
> to apt called: "APT::URL-Remap::". This allows one to remap a
> URI. Example: apt-get install 3dchess -o
> APT::URL-Remap::http://merkel.debian.org/~aba/debia...n.org/d

> +ebian/"

Having to specify this at the commandline is messy, is there a way to put
this in /etc/apt.conf.d/? I've tried in vain using

APT::URL-Remap::http://merkel.debian.org/~aba/debian/ {"http://ftp.at.d
ebian.org/debian";};

thanks, cu andreas

--
"See, I told you they'd listen to Reason," [SPOILER] Svfurlr fnlf,
fuhggvat qbja gur juveyvat tha.
Neal Stephenson in "Snow Crash"


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.or
g

[ Post a follow-up to this message ]

[ Post a follow-up to this message ]

Daniel Burrows <dburrows@debian.org> wrote:
> On Saturday 10 September 2005 07:46 am, Andreas Metzler wrote: 
[vbcol=seagreen] 
[vbcol=seagreen]
> I would expect that removing the braces would do the right thing.

> APT::URL-Remap::http://merkel.debian.org/~aba/debian/
> "http://ftp.at.debian.org/debian";

Thanks,
it does not help though; I still get:
(SID)root@argenau:/# apt-cache show apt
E: Syntax error /etc/apt/apt.conf.d/80incremental:2: Extra junk at end of fi
le
cu andreas
PS: I got tht lots-of-curly-braces-idea from
/etc/apt/apt.conf.d/70debconf.
--
"See, I told you they'd listen to Reason," [SPOILER] Svfurlr fnlf,
fuhggvat qbja gur juveyvat tha.
Neal Stephenson in "Snow Crash"


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.or
g

[ Post a follow-up to this message ]

On Sun, 11 Sep 2005 09:18:57 +0200, Andreas Metzler wrote:
> Daniel Burrows <dburrows@debian.org> wrote: 
>  
>  
>  
> 
> Thanks,
> it does not help though; I still get:
> (SID)root@argenau:/# apt-cache show apt
> E: Syntax error /etc/apt/apt.conf.d/80incremental:2: Extra junk at end offile[/vbc
ol]

APT { URL-Remap::http://merkel.debian.org/~aba/debian/ "http://ftp.at.d
ebian.org/debian"; };
at least parses (should the method of scoping make any difference, or is ita
 bug?),
but seems not to have any effect.
[vbcol=seagreen]
> PS: I got tht lots-of-curly-braces-idea from
> /etc/apt/apt.conf.d/70debconf.

DPkg::Pre-Install-Pkgs wants a list, this is the reason for braces, I think.

-- 
Micha³ Politowski
Talking has been known to lead to communication if practiced carelessly.

[ Post a follow-up to this message ]

Hi Andreas,

thanks for your feedback.

On Sat, Sep 10, 2005 at 04:46:09PM +0200, Andreas Metzler wrote:
> Michael Vogt <mvo@debian.org> wrote:
> [...] 
[..][vbcol=seagreen]
> Having to specify this at the commandline is messy, is there a way to put
> this in /etc/apt.conf.d/? I've tried in vain using
>
> APT::URL-Remap::http://merkel.debian.org/~aba/debian/ {"http://ftp.at.debian.
org/debian";};

Please add (in e.g. /etc/apt/apt.conf.d/50remap):
APT::URL-Remap::"http://merkel.debian.org/~aba/debian/" "http://ftp.de.debia
n.org/debian/";

Note the quotes around the uri, if they are omitted apt will consider
the // as the start of a comment.

Cheers,
Michael

--
Linux is not The Answer. Yes is the answer. Linux is The Question. - Neo


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.or
g

[ Post a follow-up to this message ]

On 2005-09-11 Michael Vogt <mvo@debian.org> wrote:
> On Sat, Sep 10, 2005 at 04:46:09PM +0200, Andreas Metzler wrote:
[...] 
[vbcol=seagreen]
> Please add (in e.g. /etc/apt/apt.conf.d/50remap):
> APT::URL-Remap::"http://merkel.debian.org/~aba/debian/" "http://ftp.de.debian.org/
debian/";
[...]

Thank you, this work nicely.

| Get:8 2005-09-11-2032.14.pdiff [8309B]
| Fetched 20.5MB in 7s (2619kB/s)
cu and- 80K downstream -reas


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.or
g

[ Post a follow-up to this message ]

* Michael Vogt:

> Andreas Barth implemented the server side of the index diffs
> generation and has a test-repository (with only the index-files) at
> [3].

By the way, the secure-testing project on alioth contains a
moderately-tested pure-Python implementation of the client side
(without ed/red depedencies).  It's not a speed daemon, but it
significantly cuts down network traffic.

Thanks a lot for providing the necessary infrastructure.


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.or
g

[ Post a follow-up to this message ]

On Mon, Sep 12, 2005 at 10:11:34PM +0200, Florian Weimer wrote:
> * Michael Vogt: 
>
> By the way, the secure-testing project on alioth contains a
> moderately-tested pure-Python implementation of the client side
> (without ed/red depedencies).  It's not a speed daemon, but it
> significantly cuts down network traffic.

Just to avoid confusion (and because it wasn't mentioned in the
original mail), the apt implementation has it's own rred-method (in
methods/rred.cc) and does not need a external ed/red.

Cheers,
Michael
--
Linux is not The Answer. Yes is the answer. Linux is The Question. - Neo


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.or
g

[ Post a follow-up to this message ]

Michael Vogt <mvo@debian.org> wrote:
[...]
> We believe that the code is now "good" enough for wider testing. I
> have setup a repository for the patched version of apt with the pdiff
> support.  Just add this line to your sources.list file:
> "deb http://people.debian.org/~mvo/apt/pdiffs/ /"
[...]
> Example: apt-get install 3dchess -o
> APT::URL-Remap::http://merkel.debian.org/~aba/debia...n.org/d

> +ebian/"

The remapping features seems not to work with deb-src entries.

----------------
(SID)ametzler@argenau:/tmp$ apt-get -o APT::URL-Remap::http://merkel.debian.org/~a
..ian.org/debian/ source efax
Reading package lists... Done
Building dependency tree... Done
Need to get 114kB of source archives.
Err http://merkel.debian.org sid/main efax 1:0.9a-15 (dsc)
404 Not Found
[...]
Failed to fetch http://merkel.debian.org/~aba/debia...-15
.dsc  404 Not Found
[...]
E: Failed to fetch some archives.
----------------

"apt-get update"ing of deb-src entries seems to work well, though.
Just the actual fetching of source packages is broken.
cu andreas

--
"See, I told you they'd listen to Reason," [SPOILER] Svfurlr fnlf,
fuhggvat qbja gur juveyvat tha.
Neal Stephenson in "Snow Crash"


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.or
g

[ Post a follow-up to this message ]