Debian Developers - [RFC] apt-zeroconf 0.1

Author

[RFC] apt-zeroconf 0.1

Franz Pletz

2006-11-19, 1:17 pm

Dear Debian Developers,

we are working on a distributed apt-cacher for local networks
implemented in Python. We call it apt-zeroconf since we use avahi for
automatically finding other apt-zeroconf instances on the LAN, similar
to Apple's Rendezvous/Bonjour/Zeroconf technology.

It is still in an early development stage, so we wouldn't recommend it
for everyday use. This release is solely for the purpose of getting
feedback and further ideas from you. And bugreports, of course.

Please send all comments, ideas or insults either as replies to this
email or to our trac at

http://trac.phidev.org/trac/wiki/AptZeroconf

You can also reach us via IRC in #apt-zeroconf on irc.debian.org.

Features
--------

* implemented in Python
* easy and fast to set up
* share your apt-cache with others on your LAN transparently using
avahi
* automatic file-list-updates using inotify
* acts as a http server and proxy for apt
* no root privileges required
* active development

Concept
-------

When running more than one Debian box in a LAN with slow internet access
the most often used solution to prevent downloading the same Debian
package twice is running apt-cacher or the like on one single machine,
a centralized server.

The problem with this solution is, that every machine in the network
needs to be configured for this proxy. If it is down, no one can update.
If you go to another network, you must tweak your sources.list or
apt.conf in order to upgrade, which is quite common nowadays with
notebooks and wifi technology.

We would therefore be interested in an automatic and peer-to-peer
solution. When there is no apt caching daemon in the LAN, we want to
fetch the packages directly from the internet. But if there is one or
even more available, apt should automatically use it, without any
configuration from the user. This is by the way the reason we called
it apt-zeroconf in the first place.

The next question is: Who runs an apt caching daemon? In our first
implementation, everyone running apt-zeroconf also shares his apt
package cache in /var/cache/apt/archives to everyone in the network.

Now, one might think this could potentially pose a security threat as
everyone can offer and distribute debs without any authentication
whatsoever. This is not the case as we are not yet caching the package
lists or pdiffs, which are PGP-signed and contain MD5, SHA1 and SHA256
checksums of the packages. But due to the trusted PGP signatures,
caching package lists shouldn't be an issue.

Technical
---------

Apt-zeroconf consists of two parts: a http server and the avahi-related
stuff. Avahi is used to let everyone else in the network know that you
have an apt-zeroconf caching service installed and running. The httpd
does act as http server if used in fake-mirror-mode, but also as http
proxy if you're using it as such with apt. It uses port 1618 by
default.

In every mode, apt-zeroconf is provided with a fallback mirror by apt,
so if there are no apt-zeroconf instances on the LAN offering your deb,
your regular Debian mirror will be used to provide apt with the
necessary files. See Configuration for details.

Configuration
-------------

Yes, there is a little configuration, despite the name, but only
because the user needs to be aware that he is actually using it and
that the daemon has to be running when performing an update, upgrade
or install.

There are two ways to use it:

* Proxy Mode
Add the following line to your /etc/apt/apt.conf:

Acquire { http { Proxy "http://127.0.0.1:1618"; } }

This way every mirror listed in your sources.list will be cached.
If you don't want this use the second method.

* Fake Mirror Mode
Change in your /etc/apt/sources.list

deb http://your.mirror.debian.org/debian sid main

to

deb http://127.0.0.1:1618/your.mirror.debian.org/debian sid main

e.g. deb http://127.0.0.1:1618/ftp.de.debian.org/debian sid main

With this method, you can select only specific mirrors to be patched
through apt-zeroconf.

TODO
----

* Configuration
* Support for resuming (HTTP header pass-through)
* Proxy support
* Documentation
* Use debconf for even easier configuration

Where to get it
---------------

Grab it while it's hot from:

* source: http://phidev.org/azc/apt-zeroconf_0.1.tar.gz
* deb: http://phidev.org/azc/apt-zeroconf_0.1_all.deb
* dsc: http://phidev.org/azc/apt-zeroconf_0.1.dsc

Cheers,
Florian Ludwig
Franz Pletz

--
Franz Pletz \ eat Depends: cook | eat-out
www: http://franz-pletz.org/ \ But eat-out is non-free so that's out
email: fpletz@franz-pletz.org \ And cook Recommends: clean-pans

Reinhard Tartler

2006-11-19, 1:17 pm

Franz Pletz

2006-11-19, 7:23 pm

On Sun, Nov 19, 2006 at 07:07:03PM +0100, Reinhard Tartler wrote:
> Why don't you install a line like
>
>
> in /etc/apt/sources.list.d, and be done with it? That's what I'd call
> zero config.
>
> You would have to convince apt to prefer apt-zeroconfig sources over
> 'regular' HTTP sources, so that it only uses the slow line when the
> package couldn't be found in the neighbourhood. I believe this could be
> tweaked be tuning /etc/apt/preferences. If not, you would have to hack
> up apt a bit.

We also pondered with that idea, but with this configuration modifying
apt is definitely required. Apt associates a server to every package
and version where to get the debs from. We are not building package
lists from all hosts in the network because this would compromise the
security of the whole network.

To understand this, let me show you what is happening behind the
curtains if you're adding a line like this

deb http://127.0.0.1:1618/ftp.de.debian.org/debian main

to your sources.list:

1. When you perform an update the package lists and PGP signatures
will be fetched from ftp.de.debian.org. Apt-zeroconf effectively
acts as a proxy to apt.

2. If you install software and this mirror has the current version
and/or is pinned, apt will query apt-zeroconf for the file.
Apt-zeroconf looks in its file-lists if a host in the network has
this file and fetches it in this case. If not, ftp.de.debian.org
is also used by proxying the request.

Now, without a package list, we can't just easily pin our source. When
we generate a package list from all the hosts in the network we can't
authenticate the source of the packages and the list through the
checksums as they might be forged.

If we can get the package list and the corresponding PGP signature from
another host in the network, however, we can check if we are in a secure
environment. This is also a feature which we would like to implement in
future versions.

So, what modification would apt need for this cool feature to work? We
would need to be able to define some kind of master-server where every
request for debs would go to. We try to emulate this by associating the
package list from a mirror to our apt-zeroconf proxy with our scheme of
changing sources.list. Yes, it's a hack, but it works.

What would also be nice is some kind of download-hook where we can
either change the URL before it is downloaded or the download mechanism
itself.

We didn't want to go through the trouble of modifying apt itself
initially, though.

>
> In this scenario, I'd personally don't care at all which mirror I get my
> packages from but from the apt-suite and the archive I get the packages
> from. This can be determined by examining the current apt config, so it
> is possible to create such a config in postinst.

But if we do this in postinst, everytime a user changes his apt config,
our postinst-script needs to be run somehow. I think we can't do this
easily and reliably.

> As you already said, authentication is already ensured by apt itself,
> which is nice.

Unfortunately, this is only the case if we have a correctly
authenticated package list, see my comments on this above.

>
> why proxy support? I thought you want to serve packages from the local
> neighbourhood.

No, currently, apt-zeroconf might also fetch packages from an external
HTTP source itself and therefore act as a proxy to apt. This won't work
if you need a proxy to connect to the internet.

Thanks for your comments.

Cheers,
Franz

--
Franz Pletz \ eat Depends: cook | eat-out
www: http://franz-pletz.org/ \ But eat-out is non-free so that's out
email: fpletz@franz-pletz.org \ And cook Recommends: clean-pans

Andreas Barth

2006-11-20, 7:30 am

* Franz Pletz (fpletz@franz-pletz.org) [061119 21:06]:
> On Sun, Nov 19, 2006 at 07:07:03PM +0100, Reinhard Tartler wrote:
>
> We also pondered with that idea, but with this configuration modifying
> apt is definitely required. Apt associates a server to every package
> and version where to get the debs from. We are not building package
> lists from all hosts in the network because this would compromise the
> security of the whole network.

Two questions:

1. Why don't you add the appropriate proxy-line to to apt via apt.conf.d
(e.g. as conffile, so you don't have to care about that later on)?
2. What security threats do you see?

Cheers,
Andi
--
http://home.arcor.de/andreas-barth/

--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Daniel Jacobowitz

2006-11-21, 1:31 am

On Sun, Nov 19, 2006 at 04:35:21PM +0100, Franz Pletz wrote:
> we are working on a distributed apt-cacher for local networks
> implemented in Python. We call it apt-zeroconf since we use avahi for
> automatically finding other apt-zeroconf instances on the LAN, similar
> to Apple's Rendezvous/Bonjour/Zeroconf technology.

I just want to say "thanks" - this is great. And hopefully, when it
enters Debian, it will truly require zero configuration!

--
Daniel Jacobowitz
CodeSourcery

--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Toni Mueller

2006-11-21, 7:25 am

Hi,

On Sun, 19.11.2006 at 16:35:21 +0100, Franz Pletz <fpletz@franz-pletz.org> wrote:
> * share your apt-cache with others on your LAN transparently using
> avahi
> * automatic file-list-updates using inotify
> * acts as a http server and proxy for apt

would you consider it feasible to add auto-detection features to
apt-proxy, so machines hooking up to a LAN could automatically find and
use instances of apt-proxy if available, and/or allow apt-proxy
instances to mesh with each other (eg. for increased performance)?

Best,
--Toni++

--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Florian Ludwig

2006-11-27, 1:18 pm

On Tue, 2006-11-21 at 09:55 +0100, Toni Mueller wrote:
> Hi,
>
> On Sun, 19.11.2006 at 16:35:21 +0100, Franz Pletz <fpletz@franz-pletz.org> wrote:
>
> would you consider it feasible to add auto-detection features to
> apt-proxy, so machines hooking up to a LAN could automatically find and
> use instances of apt-proxy if available
Therefore we need to patch apt-proxy to announce itself in the network
[through avahi]. Adding avahi-support to apt-proxy is a good idea in
general - I think, since not only apt-zeroconf can make some use of it.
It would be easy to use apt-proxy [with avahi support] as 'fallback'
within apt-zeroconf:
If azc cant find any other azc instance with the needed file it would
use a apt-proxy if available otherwise it fetches the files directly
from the mirror. That would be the best way to go in my opinion.

> and/or allow apt-proxy
> instances to mesh with each other (eg. for increased performance)?

We decided to implement azc from scratch and not expand apt-proxy so we
will not do it now. If you want a more decentralized caching facility
use apt-zeroconf. Its under development and much things are missing that
apt-proxy supports. But this will change hopefully soon.

Thanks for the respond,
i belief we get some new ideas from it.

Florian Ludwig

Florian Ludwig

2006-11-27, 1:18 pm

On Mon, 2006-11-20 at 10:16 +0100, Andreas Barth wrote:
> * Franz Pletz (fpletz@franz-pletz.org) [061119 21:06]:
the[vbcol=seagreen]
could be[vbcol=seagreen]
hack[vbcol=seagreen]
modifying[vbcol=seagreen]
the[vbcol=seagreen]
>
> Two questions:
>
> 1. Why don't you add the appropriate proxy-line to to apt via
apt.conf.d
> (e.g. as conffile, so you don't have to care about that later on)?

This is basically what our proxy mode supports, but we haven't yet
automated the process of adding apt-zeroconf to apt. We should probably
do that, right. We are thinking about using debconf for this to let
users a choice how may like to use azc only for some of there apt
sources.

> 2. What security threats do you see?

Since there is a lot of work done on apt security most cases are handled
right through apt itself. But we maybe should azc let do some checks:
Imagine there is someone in the network who offers corrupted files and
you download through azc. apt will not install them because of
missmatching hashes (as long nobody is able to crack md5, sha at once -
but this is not a azc specific problem). So you fail to install the
stuff you wanted and the you don't know why - and especially not who
is to blame. If you try to install the package again you will fail again
as log as there is only on person who offers the file.

As long as we don't check the package within azc its not possible to
react automatic to an corrupted file since azc doesn't recognizes that
apt fails to install a file. But this would be good - for example for
ignoring the node at further file searches to avoid to download the same
corrupted file again.

This would add some load to azc since the checking cost cpu and hd time
so it should be optional but turned on by default. I hope in the next
release there will be a proper logging system. With that you can figure
out who offered the corrupted file by grep'ing the log files.

Not official reps are always a security risk... especially those that
contain...
* Packages without hashes
* Pkgs without sign
* PkgLists without sign

For now we don't handle package lists (and/or pdiffs) with azc - they
are always fetched from the fallback-server - but may add it this to azc
in future. Since pkg lists are also signed it shouldn't be a security
hole. At all its much more complicated to cache the pkg lists so this
will be done one day in future - perhaps.

Florian Ludwig