Debian Developers - Debian ISOs

Author

Debian ISOs

Anthony L. Bryan

2006-08-17, 1:20 pm

Hi,

Metalinks might be helpful on Debian's download page for CD/DVD images. You
could have a single quick link to your ISOs that contains all the
mirror/p2p/checksum info in it.

Metalinks, a cross platform vendor neutral fortmat, are used by download
managers & contain Mirror & p2p locations for segmented downloads, along
with automatic checksum verification when the download completes. It spreads
the download between multiple servers so its faster for users, more
reliable, & less load on any one server. Metalinks are backward compatible
too.

If you're interested, you can try it out with aria2 (Unix command line)
(http://aria2.sourceforge.net), Speed Download (Mac)
(http://www.yazsoft.com), wxDownload Fast (Mac/Unix/Win), or GetRight
(Windows) (http://www.getright.com), then clicking on these samples:

http://www.metalinker.org/samples.html#isos (There's a metalink for Debian
DVD ISOs there).

Also, OpenOffice.org uses metalinks to distribute their free office suite at
http://distribution.openoffice.org/p2p/magnet.html & a few other Linux
distributions use it for their ISOs.

There's a video of metalink working in GetRight here:
http://www.metalinker.org/implementation.html

Metalinks are just simple XML like this:

<?xml version="1.0" encoding="utf-8"?>
<metalink version="3.0" generator="Metalink Generator v1.00.0034"
xmlns="http://www.metalinker.org/">
<publisher>
<name>debian</name>
<url>http://www.debian.org/</url>
</publisher>
<description>Debian 3.1r2 i386 DVD ISO</description>
<files>
<file name="debian-31r2-i386-binary-1.iso">
<version>3.1r2</version>
<os>Linux-x86</os>
<verification>
<hash type="md5">e467b508185f4fdd8e97c9ee76045288</hash>
</verification>
<resources>
<url type="http" location="us"
preference="100">http://cdimage.debian.org/debian-cd...i386/iso-dvd/de
bian-31r2-i386-binary-1.iso</url>
<url type="http" location="us"
preference="100">http://debian.osuosl.org/debian-cdi...ent/i386/iso-dv
d/debian-31r2-i386-binary-1.iso</url>
<url type="http" location="ro"
preference="100">http://ftp.iasi.roedu.net/mirrors/f....org/debian-cd/
current/i386/iso-dvd/debian-31r2-i386-binary-1.iso</url>
</resources>
</file>
<files>
<file name="debian-31r2-i386-binary-2.iso">
<version>3.1r2</version>
<os>Linux-x86</os>
<verification>
<hash type="md5">34ad0d7a9ff4a44c5cffb6d238ac4b26</hash>
</verification>
<resources>
<url type="http" location="us"
preference="100">http://cdimage.debian.org/debian-cd...i386/iso-dvd/de
bian-31r2-i386-binary-2.iso</url>
<url type="http" location="us"
preference="100">http://debian.osuosl.org/debian-cdi...ent/i386/iso-dv
d/debian-31r2-i386-binary-2.iso</url>
<url type="http" location="ro"
preference="100">http://ftp.iasi.roedu.net/mirrors/f....org/debian-cd/
current/i386/iso-dvd/debian-31r2-i386-binary-2.iso</url>
</resources>
</file>

</files>
</metalink>

thanks,

Anthony Bryan
http://www.metalinker.org/

--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Josselin Mouette

2006-08-22, 7:37 pm

Le jeudi 17 août 2006 à 11:48 -0400, Anthony L. Bryan a écrit :
> Hi,
>
> Metalinks might be helpful on Debian's download page for CD/DVD images. You
> could have a single quick link to your ISOs that contains all the
> mirror/p2p/checksum info in it.
>
> Metalinks, a cross platform vendor neutral fortmat, are used by download
> managers & contain Mirror & p2p locations for segmented downloads, along
> with automatic checksum verification when the download completes. It spreads
> the download between multiple servers so its faster for users, more
> reliable, & less load on any one server. Metalinks are backward compatible
> too.

Given that downloads like Debian ISOs are already putting a heavy
bandwidth load on the servers and that they are already shared among
many servers, I don't think it is a good idea to encourage users to load
several servers at once with one download. We should instead push
bittorrent as the main distribution media for ISOs.
--
.''`. Josselin Mouette /\./\
: :' : josselin.mouette@ens-lyon.org
`. `' joss@debian.org
`- Debian GNU/Linux -- The power of freedom

Bruce Sass

2006-08-22, 7:37 pm

On Tue August 22 2006 13:04, Josselin Mouette wrote:
> Given that downloads like Debian ISOs are already putting a heavy
> bandwidth load on the servers and that they are already shared among
> many servers, I don't think it is a good idea to encourage users to
> load several servers at once with one download. We should instead
> push bittorrent as the main distribution media for ISOs.

or as the main distribution media.

This is user supported software, why not user distributed also.

- Bruce

--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Mgr. Peter Tuharsky

2006-08-23, 1:29 am

If that's intended, then it needs to be done in such a way that even
low-to-moderately-skilled user can set it up with ease.

I know it's silly to even mention that, but unfortunatelly, user
friendliness and good documentation (good for users, not only for
developers!) are still, ehm, not a matter of course.

I'm not being a geek, however, aren't there some better protocols than
bittorent?

Peter

--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Josselin Mouette

2006-08-23, 7:38 am

First of all, please use a MUA that doesn't break threads.

Le mercredi 23 ao=FBt 2006 =E0 08:10 +0200, Mgr. Peter Tuharsky a =E9crit :
> If that's intended, then it needs to be done in such a way that even=20
> low-to-moderately-skilled user can set it up with ease.
>=20
> I know it's silly to even mention that, but unfortunatelly, user=20
> friendliness and good documentation (good for users, not only for=20
> developers!) are still, ehm, not a matter of course.

With proper software installed on the system (whatever the system is),
downloading with bittorrent is just a matter of clicking on the link.

> I'm not being a geek, however, aren't there some better protocols than=20
> bittorent?

Bittorrent is by far the most efficient protocol when it comes to large
file distribution.
--=20
.''`. Josselin Mouette /\./\
: :' : josselin.mouette@ens-lyon.org
`. `' joss@debian.org
`- Debian GNU/Linux -- The power of freedom

Mgr. Peter Tuharsky

2006-08-23, 7:38 am

> First of all, please use a MUA that doesn't break threads.

I'm using Thunderbird and don't intend to switch.

> Bittorrent is by far the most efficient protocol when it comes to large
> file distribution.

OK

Josselin Mouette wrote / napísal(a):
> First of all, please use a MUA that doesn't break threads.
>
> Le mercredi 23 août 2006 à 08:10 +0200, Mgr. Peter Tuharsky a écrit :
>
> With proper software installed on the system (whatever the system is),
> downloading with bittorrent is just a matter of clicking on the link.
>
>
> Bittorrent is by far the most efficient protocol when it comes to large
> file distribution.

--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Christian Perrier

2006-08-23, 7:38 am

> several servers at once with one download. We should instead push
> bittorrent as the main distribution media for ISOs.

I have a few doubts about the knowledge of the average user for
Bittorrent. For sure, having BitTorrent helps reducing the load
because all users that have some know-how with it will use it...but
making it the main distribution mode...ahem.....I think that most of
the users will stick to ISO images downloads.

Gabor Gombas

2006-08-23, 7:38 am

On Wed, Aug 23, 2006 at 11:30:53AM +0200, Christian Perrier wrote:

> I have a few doubts about the knowledge of the average user for
> Bittorrent. For sure, having BitTorrent helps reducing the load
> because all users that have some know-how with it will use it...but
> making it the main distribution mode...ahem.....I think that most of
> the users will stick to ISO images downloads.

Many users do not know what an ISO image is, either. If someone creates
a nice frontend with a big button saying "Push me to download & burn the
latest Debian CD/DVD", users won't care if it uses bittorrent in the
background.

Gabor

--=20
---------------------------------------------------------
MTA SZTAKI Computer and Automation Research Institute
Hungarian Academy of Sciences
---------------------------------------------------------

Christian Perrier

2006-08-23, 7:38 am

> Many users do not know what an ISO image is, either. If someone creates

My knowledge of the average user (one of the few things I claim to
have quite big experience of) is that they do. Especially, the "Bob
User" who's the D-I team favourite user (the user that's not
a complete nerd with computers and is, or thinks (s)he is, a skilled
Windows user).

I agree that Joe Dumbuser doesn't know what is an ISO image but, most
often, (s)he will get the CD from a Bob User..:-)

Christian Perrier

2006-08-23, 7:38 am

Quoting Josselin Mouette (joss@debian.org):

> When a nice bittorrent frontend is installed, the user will only have to
> click on the link to start the download. This is true for Windows and
> Linux.

I don't doubt it..:-)

But I doubt that many users have a nice Bittorrent frontend and, what
I'm really sure of, many users cannot indeed use Bittorrent (which
protocol is quite certainly filtered on many corporate networks).

Again, my point is really that promoting BitTorrent as the only way to
download images of Debian CD/DVD would really be bad...and promiting
it as the very preferred way would be quite unwise if the
"traditional" download of ISO images is too hidden.

Josselin Mouette

2006-08-23, 7:38 am

Le mercredi 23 ao=FBt 2006 =E0 11:30 +0200, Christian Perrier a =E9crit :
> I have a few doubts about the knowledge of the average user for
> Bittorrent. For sure, having BitTorrent helps reducing the load
> because all users that have some know-how with it will use it...but
> making it the main distribution mode...ahem.....I think that most of
> the users will stick to ISO images downloads.

When a nice bittorrent frontend is installed, the user will only have to
click on the link to start the download. This is true for Windows and
Linux.
--=20
.''`. Josselin Mouette /\./\
: :' : josselin.mouette@ens-lyon.org
`. `' joss@debian.org
`- Debian GNU/Linux -- The power of freedom

Hendrik Sattler

2006-08-23, 1:28 pm

Am Mittwoch 23 August 2006 12:41 schrieb Josselin Mouette:
> Le mercredi 23 ao=C3=BBt 2006 =C3=A0 11:30 +0200, Christian Perrier a =C3=
=A9crit :
>
> When a nice bittorrent frontend is installed, the user will only have to
> click on the link to start the download. This is true for Windows and
> Linux.

If, not when. There is no bittorrent client in any Suggests: or Recommends:=
=20
line of any of the browsers in Debian. And I guess that most system do not=
=20
have one intalled.
However, http and ftp will always work as that is the same method as the on=
e=20
used to access the download page.

HS

Bruce Sass

2006-08-23, 1:28 pm

On Wed August 23 2006 05:30, Hendrik Sattler wrote:
> Am Mittwoch 23 August 2006 12:41 schrieb Josselin Mouette:
=C3=A9crit :[vbcol=seagreen]
>
> If, not when. There is no bittorrent client in any Suggests: or
> Recommends: line of any of the browsers in Debian. And I guess that
> most system do not have one intalled.
> However, http and ftp will always work as that is the same method as
> the one used to access the download page.

A browser Suggests: or Recommends: is not really needed as most=20
bittorrent clients appear to be standalone programs, it is also a=20
little silly for browsers to suggest or recommend clients for every=20
mime-type out there.

"http and ftp will always work" is a really good point... someone=20
mentioned `corporate filtering,' I think bandwidth limiting by ISPs=20
would be a bigger problem. Shouldn't be a deal killer though, just=20
don't use bittorrent as the only method.

Best, imo, would be a new torrent-like protocol and some serious PR to=20
minimize the possibility of it being seen and used as just another p2p=20
network the RIAA (whoever) doesn't like.

=2D Bruce

Blars Blarson

2006-08-23, 7:29 pm

In article <1156329705.4961.51.camel@silicium.ccc.cea.fr>
joss@debian.org writes:
>When a nice bittorrent frontend is installed, the user will only have to
>click on the link to start the download. This is true for Windows and
>Linux.

You left out the reconfigure the firewall(s) step. Not only is this
non-trivial, the user may not have the ability to do so.
--
Blars Blarson blarson@blars.org
http://www.blars.org/blars.html
With Microsoft, failure is not an option. It is a standard feature.

--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Bruce Sass

2006-08-23, 7:29 pm

On Wed August 23 2006 12:32, Blars Blarson wrote:
> In article <1156329705.4961.51.camel@silicium.ccc.cea.fr>
>
> joss@debian.org writes:
>
> You left out the reconfigure the firewall(s) step. Not only is this
> non-trivial, the user may not have the ability to do so.

This is a bit of a red herring. Torrents work without re-configuring
firewalls, they just don't work as well. There appears to be two
reasons for that (all of this is, "afaict", and I've only looked at it
superficially so far):

The desire to spread the server load over many peers and foil "leeches"
has resulted in a policy where `the download rate is proportional to
the upload rate.' With an unconfigured firewall the client can only
upload to clients it is downloading from, it is the resulting limited
upload rate which chokes the download rate. Since the policy is set by
the tracker, and Debian would need to manage its own tracker[1], this
problem should be managable without too many hoops.

Another reason for non-trivial reconfiguration is because clients can
have braindead out-of-the box configurations. e.g., last time I looked
at bittornado it was setup to randomly use any of 50,000 ports[2]. The
user ends up needing to configure both the firewall and client to
realize the full potential. So, if the problem above is managable, this
one would be a non-issue.

On the other hand, if it turns out that it is not possible to overcome
mis|unconfigured firewalls with more liberal tracker policy and/or
client configs, it becomes a question of whether there are enough users
able and willing to jump through the hoops to make offering the service
worthwhile (main distribution method or not). <shrug> I dunno how to
determine that, but active torrents of Debian .iso's do exist, see:
http://www.torrentz.com/search_debian

- Bruce

[1] to ensure the network is not used to distribute non-Debian stuff

[2] 50,000 ports... to complicate port based bandwidth limiting or
filtering?

--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Adam Borowski

2006-08-23, 7:29 pm

On Wed, Aug 23, 2006 at 03:34:34PM -0600, Bruce Sass wrote:
> On Wed August 23 2006 12:32, Blars Blarson wrote:
>
> This is a bit of a red herring. Torrents work without re-configuring
> firewalls, they just don't work as well.

They don't work well if there's NAT[1] involved, you wanted to say. Do I
need to point out a wonderful opportunity to push in some IPv6 propaganda?

Too bad, most torrent clients seem to be heavily lagging behind the rest of
network software even though it's them who would benefit the most from
peer-to-peer (duh) addressing. Most but not all, and since you get to
choose both the tracker software and the suggested clients...

The relevant download page can say <<END
* if you have IPv6 connectivity, or can configure it (->howto), download the
ISOs using bittorrent. For this, install package XXX, or, on M$ Windows
systems, download and run ->this.
* if you have only IPv4, you can still use bittorrent. If you're behind
NAT, your download speed will suffer unless you can reconfigure your
firewall (->discussion).
* if you can't run bittorrent at all, you'll have to fall back to ->http or
->ftp. This method should work everywhere, although it puts a strain on our
servers.
END

I hope this is dumbed down enough to handle users who would be capable to
reconfigure their firewalls in the first place. If they have that much
clue, it's a waste to use a bittorrent-specific one-time fix instead of the
Final Solution to the NAT Question.

[1]. In the typical sense of the word, that is.
--
1KB // Microsoft corollary to Hanlon's razor:
// Never attribute to stupidity what can be
// adequately explained by malice.

--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Alexey Feldgendler

2006-08-24, 1:29 am

On Thu, 24 Aug 2006 00:38:59 +0700, Bruce Sass <bmsass@shaw.ca> wrote:

> "http and ftp will always work" is a really good point... someone
> mentioned `corporate filtering,' I think bandwidth limiting by ISPs
> would be a bigger problem. Shouldn't be a deal killer though, just
> don't use bittorrent as the only method.

If only there was some auto-negotiation method like the one HTTP provides for content types: IF the client is able to use bittorrent, THEN use it, ELSE use FTP.

--
Alexey Feldgendler <alexey@feldgendler.ru>
[ICQ: 115226275] http://feldgendler.livejournal.com

--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Bernhard R. Link

2006-08-25, 1:29 pm

* Adam Borowski <kilobyte@angband.pl> [060824 01:46]:
>
> They don't work well if there's NAT[1] involved, you wanted to say.
Blocking incoming connections is a common and good starting points for
every firewall setup. That NAT makes this mandatory does not change the
fact that protocols needing listening ports are a security hole many
people do not like to introduce.

> Do I
> need to point out a wonderful opportunity to push in some IPv6 propaganda?

One of the nice features of NAT is that it adds another layer of
security if your firewall contains a "no incoming connections" part:
If everything fails there still has to be some mechanism to translate
the intern IPs to extern addressable. So I hope someone will still
make it available with IPv6...

Hochachtungsvoll,
Bernhard R. Link

--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Anthony L. Bryan

2006-08-28, 7:33 am

> Le jeudi 17 ao=FBt 2006 =E0 11:48 -0400, Anthony L. Bryan a =E9crit :
> CD/DVD images. You
> used by download
> downloads, along
> completes. It spreads
> backward compatible
>=20
> Given that downloads like Debian ISOs are already putting a heavy
> bandwidth load on the servers and that they are already shared among
> many servers, I don't think it is a good idea to encourage=20
> users to load
> several servers at once with one download. We should instead push
> bittorrent as the main distribution media for ISOs.

That=92s the point of Metalink. You list the bittorrent (with a higher
priority) and mirror sources (lower) so it defaults to p2p but has other
sources for backup. bittorrent isn't an option for everyone. With =
Metalink
you can give priority to certain servers and also make bandwidth more
efficient by limiting downloads to servers in the downloaders same =
country
or geographic region.

Anthony Bryan
metalink [ http://www.metalinker.org/ ]

Subredu Manuel

2006-08-30, 7:35 am

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Josselin Mouette wrote:
> Le jeudi 17 août 2006 à 11:48 -0400, Anthony L. Bryan a écrit :
>
> Given that downloads like Debian ISOs are already putting a heavy
> bandwidth load on the servers and that they are already shared among
> many servers, I don't think it is a good idea to encourage users to load
> several servers at once with one download. We should instead push
> bittorrent as the main distribution media for ISOs.

I don't think you got the picture. Metalink can contain information for
download using the following protocols:
*) http
*) ftp
*) bittorrent
*) e2k

Also, magnet url's can be included. What protocol is used to download
the files, remains to users choice. IMHO, either if you use metalinks or
not, the user can still choose to use http/ftp/torrent/e2k/ without you
to approve that.
The main advantage in using metalinks is that all information about
downloading files using different protocols are included in _one file_
and the user has to choose which one to use.
And related to mirrors. What's the purpose of the mirrors ? To reduce
the load on the main servers and to help in better distribution files.
Agree ? Let's see now. What does metalink does ? Permit the use of _all
mirrors_ and better, permit the average Joe to download the images
faster. How do you know that the bittorrent is the logical choice ? Just
becase the protocol is great for distributing large files ? How do you
known that some area of China (for example) does not use e2k as their
primary download protocol ? Just think that in this moment, metalink
provides the user the most complete information for downloading a file.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFE9VHljGXbUSvc3AsRAngGAJ9N7AxjO/3qbTRZM2alaM81JPN9yQCfUB23
MOEc/gmgtqBrLebg4A+8Nxk=
=DI3k
-----END PGP SIGNATURE-----

--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Bruce Sass

2006-08-30, 7:35 pm

On Wed August 30 2006 02:52, Subredu Manuel wrote:
> Josselin Mouette wrote:
>
> I don't think you got the picture. Metalink can contain information
> for download using the following protocols:
> *) http
> *) ftp
> *) bittorrent
> *) e2k
<...>

You seem to be ignoring that metalinker will use multiple protocols,=20
servers, and connections in an effort to get the fastest download.=20
http://www.metalinker.org/Metalink_3.0_Spec.pdf also indicates it will=20
switch servers if the download rate falls, and perhaps even perpetuate=20
old torrents --- which is all great for the client, but will pretty=20
much guarantee worse than possible performance from the servers.

It is also not clear what will happen when a release is made and=20
hundreds (thousands?) of clients hit the fastest mirror, whose download=20
rate then drops, prompting all the clients to try switching to the new=20
fastest mirror, whose rate then drops, ... At the very least there=20
will be more overhead traffic for the servers, worst case could be some=20
kind of nasty cyclic oscillations with the server loads which=20
undermines transfer efficiency for everyone. Ya, that's kinda FUDish,=20
but given the newness and limited deployment of the technology,=20
speculation is about all we have to work with.

The objective is to ease and spread out the load on the servers;=20
bittorrent and socially engineered access to the less efficient=20
protocols is more likely to do that than bittorrent + ftp + http +=20
multiple servers/client. IMO

I think Metalinks are an interesting idea which deserve investigation,=20
but it seems clear they would result in more server overhead and=20
unclear whether the load spreading would offset that or even happen[1].

=2D Bruce

[1] could clients be attracted to a few fast ftp servers and the often=20
slow-to-start torrents end up being underutilized... the exact opposite=20
of what is best for Debian's servers.

Subredu Manuel

2006-08-31, 7:37 am

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Bruce Sass wrote:
>
> You seem to be ignoring that metalinker will use multiple protocols,
> servers, and connections in an effort to get the fastest download.
> http://www.metalinker.org/Metalink_3.0_Spec.pdf also indicates it will
> switch servers if the download rate falls, and perhaps even perpetuate
> old torrents --- which is all great for the client, but will pretty
> much guarantee worse than possible performance from the servers.
>
> It is also not clear what will happen when a release is made and
> hundreds (thousands?) of clients hit the fastest mirror, whose download
> rate then drops, prompting all the clients to try switching to the new
> fastest mirror, whose rate then drops, ... At the very least there
> will be more overhead traffic for the servers, worst case could be some
> kind of nasty cyclic oscillations with the server loads which
> undermines transfer efficiency for everyone. Ya, that's kinda FUDish,
> but given the newness and limited deployment of the technology,
> speculation is about all we have to work with.

When a new release is made, all servers (and mirrors) are getting hit.
First, the tier 1 (ftp.<country>.debian.org), and then the other
mirrors. So, if you only consider http and ftp, metalink is the logical
choice, since it switches automatically to a new server, when the rate
drops (at least, it should, but this depends on the client).
The only situation in which you can save the servers bandwidth is when
you use download protocols that involves the users bandwidth, like
bittorrent, edonkey and other specifica file sharing protocols.
So, there are 2 situations here:
*) make .torrent files
*) make .metalinks files only with bittorrent, e2k protocols and
magnet urls

You can see that metalink has advantage over bittorrent simply by having
bittorrent _and e2k_

>
> The objective is to ease and spread out the load on the servers;
> bittorrent and socially engineered access to the less efficient
> protocols is more likely to do that than bittorrent + ftp + http +
> multiple servers/client. IMO

ok. then use metalink only with bittorrent and e2k links

I only try to convince you that using this technology is just a very big
advantage for the average joe.

>
> I think Metalinks are an interesting idea which deserve investigation,
> but it seems clear they would result in more server overhead and
> unclear whether the load spreading would offset that or even happen[1].

Sure. I'm not saying that it should be adopted as once, without further
investigation and tests. But is not something that should be ignored
just because is new. I'm only saying that metalink deserves a change.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFE9oFgjGXbUSvc3AsRAgwnAJ4igHthjnvH
4K6MqoiEggwc1levcgCeKsNZ
KpOGY4jRTx40t54xgzGIsrE=
=pChm
-----END PGP SIGNATURE-----

--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Bruce Sass

2006-08-31, 7:37 am

On Thu August 31 2006 00:27, Subredu Manuel wrote:
> Bruce Sass wrote:
>
> When a new release is made, all servers (and mirrors) are getting
> hit. First, the tier 1 (ftp.<country>.debian.org), and then the other
> mirrors.

Would each mirror or region need a unique metalink file to ensure that
happens?

I'm thinking that if all clients see the same metalink file they will
all hit the same servers, and someone wanting to support the local
mirrors could start fetching from 1/2-way around the world (I suppose
there would still need to be non-metalink access just for this reason.)

Hmmm, that brings up another question. Each downloadable file would need
its own metalink file. How much space will that take and at what point
would it become significant.

> So, if you only consider http and ftp, metalink is the
> logical choice, since it switches automatically to a new server, when
> the rate drops (at least, it should, but this depends on the client).

Transfers in progress could change servers, or new clients would get a
different server?

The former is what I would be concerned about (additional overhead,
exacerbated by poorly coded or configured clients); the latter would be
a good thing but could also be done with a smart round-robin setup
(maybe already is via http.us.debian.org.)

> The only situation in which you can save the servers bandwidth is
> when you use download protocols that involves the users bandwidth,
> like bittorrent, edonkey and other specifica file sharing protocols.
> So, there are 2 situations here:
> *) make .torrent files
> *) make .metalinks files only with bittorrent, e2k protocols and
> magnet urls
>
> You can see that metalink has advantage over bittorrent simply by
> having bittorrent _and e2k_

Ya, assuming both are eventually made available from Debian.

>
> ok. then use metalink only with bittorrent and e2k links

> I only try to convince you that using this technology is just a very
> big advantage for the average joe.

Nothing wrong with that. I did much the same thing back in '95(?) with
http when Debian only provided ftp access to the archives (got told
that http was too inefficient. :-)

- Bruce

--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Anthony L. Bryan

2006-08-31, 7:51 pm

On Thu August 31 2006 00:27, Subredu Manuel wrote:
>
>Would each mirror or region need a unique metalink file to ensure that
>happens?
>
>I'm thinking that if all clients see the same metalink file they will
>all hit the same servers, and someone wanting to support the local
>mirrors could start fetching from 1/2-way around the world (I suppose
>there would still need to be non-metalink access just for this reason.)
>
>Hmmm, that brings up another question. Each downloadable file would need
>its own metalink file. How much space will that take and at what point
>would it become significant.
>
>
>Transfers in progress could change servers, or new clients would get a
>different server?
>
>The former is what I would be concerned about (additional overhead,
>exacerbated by poorly coded or configured clients); the latter would be
>a good thing but could also be done with a smart round-robin setup
>(maybe already is via http.us.debian.org.)

Hi Bruce, just wanted to say thanks for investigating Metalink. These are
all valid concerns. For the last few months, the only big user of Metalinks
has been OpenOffice.org, and I haven't heard any complaints from
mirror/server admins. That doesn't mean there aren't any. I'm sure there
will be problems, and I'd like to work them out as soon as possible.

Metalink allows for information on the country location ("location") of a
server along with priority ("preference"):

<url type="ftp"
location="au"
preference="30">

ftp://mirror.pacific.net.au/OpenOff..._LinuxIntel_ins
tall.tar.gz
</url>

The simplest way might be for clients to try local mirrors, in order of
preference, then all other mirrors in order of preference (or, other close
countries). I think this added information of location and priority will
lead to more efficient use of bandwidth over just plain links (hopefully).

Manuel has also made a web interface that generates Metalinks depending on
what country you select.

Metalinks can list multiple or single files.

The size of Metalink files depends on how much information you want to
include and how many mirrors are listed. For example, Metalinks listing
every kernel.org or OOo mirror are around 20k. All mirrors for Fedora ISOs:
54k, Ubuntu ISOs: 58k, SUSE ISOs: 10k.

I'd be interested in working on any things that may be an issue for Debian
as I'm sure they would affect others too.

One thing I'd definitely like to do is get aria2
(http://aria2.sourceforge.net/) included in Debian. It's a BitTorrent and
Metalink command line client. Maybe recommending certain clients over
another and working with the authors of those clients to be correctly coded
and configured by default could help.

Anthony Bryan
Metalink [ www.metalinker.org ]

--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Patrick Ruckstuhl

2006-08-31, 7:51 pm

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,
> One thing I'd definitely like to do is get aria2
> (http://aria2.sourceforge.net/) included in Debian. It's a BitTorrent and
> Metalink command line client. Maybe recommending certain clients over
> another and working with the authors of those clients to be correctly coded
> and configured by default could help.
I'm working on this at the moment

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=383595

Regards,
Patrick
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (MingW32)

iD8DBQFE908HaA/ ofYi4EMoRApb5AJ4m7e5oCoI47OXRhRZirxvUljZ
h1gCeJk2M
sca1ezYl3VFniYAZM7evH28=
=74V1
-----END PGP SIGNATURE-----

--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Bruce Sass

2006-08-31, 7:51 pm

Hello Anthony,

Thanks for the response.

On Thu August 31 2006 12:17, you wrote:
> Hi Bruce, just wanted to say thanks for investigating Metalink. These
> are all valid concerns. For the last few months, the only big user of
> Metalinks has been OpenOffice.org, and I haven't heard any complaints
> from mirror/server admins. That doesn't mean there aren't any. I'm
> sure there will be problems, and I'd like to work them out as soon as
> possible.
>
> Metalink allows for information on the country location ("location")
> of a server along with priority ("preference"):
>
> <url type="ftp"
> location="au"
> preference="30">
>
> ftp://mirror.pacific.net.au/OpenOff...o_2.0.3_LinuxIn
>tel_ins tall.tar.gz
> </url>
>
> The simplest way might be for clients to try local mirrors, in order
> of preference, then all other mirrors in order of preference (or,
> other close countries). I think this added information of location
> and priority will lead to more efficient use of bandwidth over just
> plain links (hopefully).

I think you are correct, provided the clients prefer local mirrors and
don't start casting around all over the 'net looking for a "great"
server without good reason (whatever that may be. :-)

> Manuel has also made a web interface that generates Metalinks
> depending on what country you select.
>
> Metalinks can list multiple or single files.
>
> The size of Metalink files depends on how much information you want
> to include and how many mirrors are listed. For example, Metalinks
> listing every kernel.org or OOo mirror are around 20k. All mirrors
> for Fedora ISOs: 54k, Ubuntu ISOs: 58k, SUSE ISOs: 10k.

Hmmm...

[quick manual counts, therefore approximate numbers]

Ubuntu has ~100 mirrors
(http://www.ubuntu.com/download)

Debian has ~36 primary + ~255 secondary mirrors
(http://www.debian.org/mirror/list)

which indicates that a Metalink file for Debian could be over 150k;
(for CDs) x 12 (11 arches and source) x (> )15 images per release.
Giving ~9M of HDD overhead if each download needed its own Metalink
file, <2M for one Metalink file per release, with reality somewhere in
between.

....so: HDD space shouldn't be a problem. It wouldn't be that simple
though because Debian mirrors are not necessarily identical and Ubuntu
appears to only have 3 or 4 arches with 3 images each on each mirror
(iow, mirror count aside, more complex Metalinks would be needed.) How
well the clients handle large numbers of <resources> is likely to be
significant and could affect how many <resources> can be reasonably
stuffed into a single Metalink; some of Debian's arches are rather
limited wrt RAM and CPU cycles compared to modern standards, and
Debian's minimum system requirements are low compared to others.

A scheme which is only useful to an arbitrary subset of Debian
installations has less chance of being adopted than one which everyone
can use. "Arbitrary" in that there may be no clear dividing line
between `works' and `does not work' based on something other than the
size of the Metalink file, the dependencies it drags in, available RAM,
or other installation dependent variables (or whims). I'm not saying
this is a problem with Metalinks, but it could be a problem unique to
Debian and would need special attention.

> I'd be interested in working on any things that may be an issue for
> Debian as I'm sure they would affect others too.
>
> One thing I'd definitely like to do is get aria2
> (http://aria2.sourceforge.net/) included in Debian. It's a BitTorrent
> and Metalink command line client. Maybe recommending certain clients
> over another and working with the authors of those clients to be
> correctly coded and configured by default could help.

http://lists.debian.org/debian-ment...8/msg00197.html

I'm not sure why it hasn't appeared in Unstable yet.
[but it looks like Patrick does :-]

- Bruce

p.s. - I'm not on debian-cd, CC me if you want.

--
To UNSUBSCRIBE, email to debian-cd-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Anthony L. Bryan

2006-09-01, 7:56 pm

> which indicates that a Metalink file for Debian could be over 150k;
> (for CDs) x 12 (11 arches and source) x (> )15 images per release.
> Giving ~9M of HDD overhead if each download needed its own Metalink
> file, <2M for one Metalink file per release, with reality
> somewhere in
> between.
>
> ...so: HDD space shouldn't be a problem. It wouldn't be that simple
> though because Debian mirrors are not necessarily identical
> and Ubuntu
> appears to only have 3 or 4 arches with 3 images each on each mirror
> (iow, mirror count aside, more complex Metalinks would be
> needed.) How
> well the clients handle large numbers of <resources> is likely to be
> significant and could affect how many <resources> can be reasonably
> stuffed into a single Metalink; some of Debian's arches are rather
> limited wrt RAM and CPU cycles compared to modern standards, and
> Debian's minimum system requirements are low compared to others.

Just did a quick peek and the 54k Fedora metalinks contain 286 links (keep
in mind, some are ftp & http like http://planetmirror.com and
ftp://ftp.planetmirror.com). Hopefully actual Debian metalinks will be
generated at http://metalink.packages.ro/ soon.

One other nice feature is that metalink clients can use partial/chunk
checksums (from torrent info) for mirrors, so if there's a faulty mirror,
info from it will be discarded and only that segment will need to be
re-transferred from somewhere else.

As far as system requirements, I'm sure testing would need to be done but
aria2 is command line only and designed to be slim (from their page):

'Unlike Aria, which has GTK+ interface, aria2 provides command-line
interface only. But GUI-lessness brings lower resource requirement. The
physical memory usage is typically 3MB(normal HTTP/FTP downloads) to
5MB(BitTorrent downloads).'

>
> http://lists.debian.org/debian-ment...8/msg00197.html
>
> I'm not sure why it hasn't appeared in Unstable yet.
> [but it looks like Patrick does :-]

Perfect, looking forward to that.

Anthony Bryan
Metalink [ www.metalinker.org ]

--
To UNSUBSCRIBE, email to debian-cd-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Patrick Ruckstuhl

2006-09-10, 1:27 am

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,
>
> Perfect, looking forward to that.
It's now in unstable

Patrick
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (MingW32)

iD8DBQFFA3T/aA/ofYi4EMoRAkAxAJ0ZGOwMIixEceOIamPVbtp/RJ+CKACfWk7F
vgLEpGQvkAaDF/hAJvR2Dsw=
=kWt3
-----END PGP SIGNATURE-----

--
To UNSUBSCRIBE, email to debian-cd-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Anthony L. Bryan

2006-09-22, 7:30 pm

Here's a brief refresher on metalinks & how it could be useful for Debian
ISO distribution:

aria2 is in unstable and testing so people can install & use it easily,
thanks to Patrick.

I hope some people can try it out!

apt-get install aria2

aria2c
http://www.metalinker.org/samples/d...-1.iso.metalink

--
To UNSUBSCRIBE, email to debian-cd-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Anthony L. Bryan

2006-09-22, 7:30 pm

(Oops, hit the send key combo before finishing, sorry).

Here's a brief refresher on metalinks & how it could be useful for Debian
ISO distribution:

'Metalink makes complex download pages obsolete by replacing long lists of
download mirrors and BitTorrent trackers with a single .metalink file. As
you might have already guessed, a .metalink file is a file that tells a
download manager all the different ways it can download a file. The file
itself takes the form of an open XML standard that can list an unlimited
number of HTTP and FTP sources as well as BitTorrent trackers and ed2k and
magnet links.'

aria2 is in unstable and testing so people can install & use it easily,
thanks to Patrick.

I hope some people can try it out!

apt-get install aria2

aria2c
http://www.metalinker.org/samples/d...-1.iso.metalink

This is a big DVD ISO download, but if you just let it run for a few minutes
you'll see how fast & easy it can be if you have a fast connection. Once it
finishes, the checksum will automatically be verified. Automatically
generated metalinks would list all mirrors, so you could tell the client
you're in de so it should use those mirrors first, then other mirrors
nearby.

Here's what this metalink contains:

<?xml version="1.0" encoding="utf-8"?>
<metalink version="3.0" xmlns="http://www.metalinker.org/">
<publisher>
<name>debian</name>
<url>http://www.debian.org/</url>
</publisher>
<description>Debian 3.1r2 i386 DVD ISO</description>
<files>
<file name="debian-31r2-i386-binary-1.iso">
<version>3.1r2</version>
<os>Linux-x86</os>
<verification>
<hash type="md5">e467b508185f4fdd8e97c9ee76045288</hash>
</verification>
<resources>
<url type="http" location="us"
preference="100">http://cdimage.debian.org/debian-cd...i386/iso-dvd/de
bian-31r2-i386-binary-1.iso</url>
<url type="http" location="us"
preference="100">http://debian.osuosl.org/debian-cdi...ent/i386/iso-dv
d/debian-31r2-i386-binary-1.iso</url>
<url type="http" location="ro"
preference="100">http://ftp.iasi.roedu.net/mirrors/f....org/debian-cd/
current/i386/iso-dvd/debian-31r2-i386-binary-1.iso</url>
<url type="http" location="br"
preference="100">http://linorg.usp.br/iso/debian/3.1...iso-dvd/debian-
31r2-i386-binary-1.iso</url>
<url type="http" location="es"
preference="100">http://ftp.gva.es/mirror/debian-cd/...386/iso-dvd/deb
ian-31r2-i386-binary-1.iso</url>
<url type="http" location="de"
preference="100">http://ftp.de.debian.org/debian-cd/...386/iso-dvd/deb
ian-31r2-i386-binary-1.iso</url>
<url type="http" location="de"
preference="100">http://ftp-stud.fht-esslingen.de/de...urrent/i386/iso
-dvd/debian-31r2-i386-binary-1.iso</url>
<url type="http" location="at"
preference="100">http://debian.inode.at/debian-cd/3..../iso-dvd/debian
-31r2-i386-binary-1.iso</url>
<url type="http" location="fr"
preference="100">ftp://ftp.free.fr/pub/Distributions...bian-cd/3.1_r2/
i386/iso-dvd/debian-31r2-i386-binary-1.iso</url>
<url type="http" location="au"
preference="100">ftp://mirror.aarnet.edu.au/pub/debi..._r2/i386/iso-dv
d/debian-31r2-i386-binary-1.iso</url>
<url type="http" location="au"
preference="100">ftp://ftp.iinet.net.au/pub/debian/d...current/i386/is
o-dvd/debian-31r2-i386-binary-1.iso</url>
<url type="http" location="be"
preference="100">ftp://ftp.scarlet.be/pub/debian-cd/...386/iso-dvd/deb
ian-31r2-i386-binary-1.iso</url>

</resources>
</file>
</file>

</files>
</metalink>

--
To UNSUBSCRIBE, email to debian-cd-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Andrew Donnellan

2006-09-24, 1:37 am

On 9/22/06, Anthony L. Bryan <albryan@comcast.net> wrote:
> (Oops, hit the send key combo before finishing, sorry).
>
> Here's a brief refresher on metalinks & how it could be useful for Debian
> ISO distribution:
>
> 'Metalink makes complex download pages obsolete by replacing long lists of
> download mirrors and BitTorrent trackers with a single .metalink file. As
> you might have already guessed, a .metalink file is a file that tells a
> download manager all the different ways it can download a file. The file
> itself takes the form of an open XML standard that can list an unlimited
> number of HTTP and FTP sources as well as BitTorrent trackers and ed2k and
> magnet links.'

Is it possible to extend it and make a client capable of jigdo or
similar? It might look like:

<url type="jigdo" location="au"
preference="100"
jigdofile="http://cdimage.debian.org/debian-cd/current/i386/jigdo-dvd/debian-31r3-i386-binary-1.jigdo">
ftp://ftp.au.debian.org/debian/</url>

(Sorry, this is hacked up; I don't know how acceptable it would be to
specify the jigdo template as an attribute instead of the content.)
--
Andrew Donnellan
http://andrewdonnellan.com
http://ajdlinux.blogspot.com
Jabber - ajdlinux@jabber.org.au
GPG - hkp://subkeys.pgp.net 0x5D4C0C58
-------------------------------
Member of Linux Australia - http://linux.org.au
Debian user - http://debian.org
Get free rewards - http://ezyrewards.com/?id=23484
OpenNIC user - http://www.opennic.unrated.net

--
To UNSUBSCRIBE, email to debian-cd-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Francois Petillon

2006-09-29, 1:53 pm

[sry, I am coming late and I am breaking all references as I did not
received this thread. Worse, I am answering to different mails at once
;-) I also apologize to debian-cd subscribers as I already sent this
answer there before realizing the initial thread was posted here too.
As there was no reply in debian-cd, I repost this answer only in
debian-devel]

I will start to introduce myself, François Pétillon, working for a small
french ISP (Free) and managing one of the debian CD mirror
(ftp.free.fr/ftp.proxad.net).

From: "Anthony L. Bryan" <albryan@comcast.net>
> Metalinks, a cross platform vendor neutral fortmat, are used by
> download managers & contain Mirror & p2p locations for segmented
> downloads, along with automatic checksum verification when the
> download completes. It spreads the download between multiple servers
> so its faster for users, more reliable, & less load on any one server.

No, that's bullshit. You are just telling that when x users download one
DVD (around 5 GB), doing it with n connections will be more
server-friendly (x*n connections to manage for all servers) than doing
it with a single one (x connections).
A server load depends on several issues :
- CPU (the more you have to manage connections, the more CPU you waste)
- memory (each connection is using a few KB/MB)
- disk IOs

And, as far as I am concerned, disk IO is the greatest problem (disk
capacity & bandwidth is increasing faster than seek time). Thus, we try
to optimize disk IO to get max performance out of that kind of server.
And optimization usually means trying to read the greatest amount of
sectors for each disk seek.

But having to manage more connections means less memory per connection
(thus reading less data per disk seek). And segmented downloads also
mean you will not be able to optimize all disk access (I already have
seen FTP clients requesting a DVD per 8KB segment, thus all disks
request were around 8KB while the server would try to load a few MB at a
time when feasible).

This for these reasons I prefer to see someone to download a full CD
image rather than using jigdo if he has to download many packages one by
one.

I do not have anything against Metalinks by itself, it is just that
people _will_ make stupid things with it and they _will_ degrade servers
performances.

From: Subredu Manuel <diablo@iasi.roedu.net>
> Let's see now. What does metalink does ? Permit the use of _all
> mirrors_ and better, permit the average Joe to download the images
> faster.

Wait a minute. Are you telling us your tool is creating bandwidth ? If
someone is download at a slow rate, there must be a good reason. No ?
The server or the network may be loaded. Fine, do your tool solve this
problem ? No, it just steal others users bandwidth/server ressources.
When everyone will have to use that kind of tool, then you'll be back to
the start point but with degraded performances.

From: Josselin Mouette <joss@debian.org>
> Given that downloads like Debian ISOs are already putting a heavy
> bandwidth load on the servers and that they are already shared among
> many servers, I don't think it is a good idea to encourage users to
> load several servers at once with one download. We should instead push
> bittorrent as the main distribution media for ISOs.

Well, with a network traffic around 120-140 GB a day (less than 15 Mbps,
cf ftp://ftp.free.fr/stats/debiancd.weekly.20060923.txt), I can hardly
consider this mirror as loaded... :-)
Do not forget that network is not free either and P2P may cost more for
an ISP than hosting mirrors (for Free case, 30% of users are connected
through FT IPADSL for which bandwidth cost is around 220 EU per Mbps per
month, 10 times the transit cost, more than 300 times the hardware cost
to output 1 Mbps/month on a FTP server).

François

--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Sylvain Beucler

2006-09-29, 1:53 pm

> From: "Anthony L. Bryan" <albryan@comcast.net>
>
> No, that's bullshit.

This sounds a bit partial.

Anthony's point stays valid when bandwidth is limited on the
server. For example I have a dedicated server from ovh.com with
limited bandwidth: I'd rather have people use bittorrent & co rather
than getting my server on its knees.

However your points are very interesting from the point of view of an
ISP (where one's own bandwidth is not the bottleneck).

More to the point, maybe you could suggest the mirrors maintainers to
separate a list of 'jidgo mirrors' and a list of 'ISO mirrors'.

--
Sylvain
PS: I use Free at home and I d/l from ftp.free.fr ;)

--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Marco d'Itri

2006-09-29, 7:58 pm

On Sep 29, Francois Petillon <fantec@proxad.net> wrote:

> And, as far as I am concerned, disk IO is the greatest problem (disk
> capacity & bandwidth is increasing faster than seek time). Thus, we try
> to optimize disk IO to get max performance out of that kind of server.
> And optimization usually means trying to read the greatest amount of
> sectors for each disk seek.
Thank you for expressing this clearly.
As the administrator of ftp.it.d.o I fully agree with your arguments.

To summarize the issue:
* parallel downloads are bad
* using P2P when local mirrors are available is very bad

--
ciao,
Marco

Marco d'Itri

2006-09-29, 7:58 pm

On Sep 29, Sylvain Beucler <beuc@beuc.net> wrote:

> Anthony's point stays valid when bandwidth is limited on the
> server.
Which is not the case for Debian mirrors except possibly in the few
hours after a major release.

--
ciao,
Marco