|
Home > Archive > Debian Developers > March 2004 > Componentized linux?
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
Componentized linux?
|
|
| Ondøej Surý 2004-03-01, 5:33 am |
| Hello,
have you read about progeny proposal[1,2] on componentized linux
distributions? How do you feel about switching to this approach after
sarge? I think it would speed up releases of Debian if we split distro
into components (say: base, x-windows, gnome, kde, apache and so on...)
[1] http://platform.progeny.com/weblogs/000005.html
[2] http://platform.progeny.com/componentized-linux/
O.
--
Ondřej Surý <ondrej@sury.org>
--
To UNSUBSCRIBE, email to debian-devel-request@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
| |
| Andreas Tille 2004-03-01, 5:33 am |
| On Mon, 1 Mar 2004, [iso-8859-2] Ond?ej Sur=FD wrote:
> have you read about progeny proposal[1,2] on componentized linux
> distributions? How do you feel about switching to this approach after
> sarge? I think it would speed up releases of Debian if we split distro
> into components (say: base, x-windows, gnome, kde, apache and so on...)
If I understand this proposal right you are asking for Custom Debian
Distributions. They only suffer from the documentation that they
exist and thus I could only provide a link to a description which is
currently work in progress but might answer your question
http://people.debian.org/~tille/deb...cdd/debian-cdd=
..html/
Kind regards
Andreas.
--
To UNSUBSCRIBE, email to debian-devel-request@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
| |
| Ondøej Surý 2004-03-01, 7:34 am |
| On Mon, 2004-03-01 at 11:26, Andreas Tille wrote:
> On Mon, 1 Mar 2004, [iso-8859-2] Ond?ej Surý wrote:
>
> If I understand this proposal right you are asking for Custom Debian
> Distributions. They only suffer from the documentation that they
> exist and thus I could only provide a link to a description which is
> currently work in progress but might answer your question
>
> http://people.debian.org/~tille/deb...ebian-cdd.html/
I think this goes farther then CDD, that proposal is about making base
distribution smaller and making releases in each areas dependant only on
"base" debian (lsb-1.3 in progeny). For example if I have desktop with
XF86 4.3 and gnome, then I don't care how much RC bugs has apache or
postfix. And when upstream makes new stable release of gnome 2.6 I
don't have to wait for release of debian as whole, but only for new
release of component 'gnome'.
On other side, if I have server with mail services then I don't really
care which version of gnome is in stable and how many bugs it has, but I
do care if I had to wait several years for new release of Debian to have
new release of my favourite MTA.
I think that something like http://www.backports.org/ is more accurate
to this.
O.
--
Ondřej Surý <ondrej@sury.org>
--
To UNSUBSCRIBE, email to debian-devel-request@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
| |
| Andreas Tille 2004-03-01, 7:34 am |
| On Mon, 1 Mar 2004, [iso-8859-2] Ond?ej Sur=FD wrote:
> I think this goes farther then CDD, that proposal is about making base
> distribution smaller and making releases in each areas dependant only on
> "base" debian (lsb-1.3 in progeny). For example if I have desktop with
OK, thanks for clarification. The only thing what I can say that we
just *talked* about exactly ths issue a OSWC in Malaga ...
Kind regards
Andreas.
--
To UNSUBSCRIBE, email to debian-devel-request@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
| |
| Colin Watson 2004-03-01, 7:34 am |
| On Mon, Mar 01, 2004 at 12:28:17PM +0100, Ondřej Surý wrote:
> I think this goes farther then CDD, that proposal is about making base
> distribution smaller and making releases in each areas dependant only on
> "base" debian (lsb-1.3 in progeny). For example if I have desktop with
> XF86 4.3 and gnome, then I don't care how much RC bugs has apache or
> postfix. And when upstream makes new stable release of gnome 2.6 I
> don't have to wait for release of debian as whole, but only for new
> release of component 'gnome'.
But, actually, you do sometimes. To take one of your examples, both
Apache and GNOME have PERL interfaces, and when the version of PERL in
the base distribution changes (e.g. from 5.6 to 5.8) then all modules
need to be rebuilt and both Apache and GNOME need to be updated in sync
for everything to continue working. That's far from the only example of
this sort of thing: Debian is rife with it.
My feeling is that, with the exception of isolated leaf subsystems,
trying to release components separately in a system as deeply
interlinked as Debian is doomed to combinatorial explosion of developer
support effort (e.g. apache-x.y.z/perl-5.6, apache-x.y.z/perl-5.8,
apache-x.y.z+1/perl-5.6, apache-x.y.z+1/perl-5.8, with an extra set of
combinations for every piece of the distribution that joins two
interfaces together). If we were in a situation where old and new
versions of subsystems were more frequently installable in parallel,
then it might be feasible, but we just aren't there: and often
parallel-installability is a difficult problem in itself, although
policy advocates it for at least libraries.
(Yes, this makes release management in general a very difficult problem,
since you lose most of your degrees of freedom in the versions available
for release.)
Cheers,
--
Colin Watson [cjwatson@flatline.org.uk]
--
To UNSUBSCRIBE, email to debian-devel-request@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
| |
| Gunnar Wolf 2004-03-01, 12:33 pm |
| Ond?ej Surý dijo [Mon, Mar 01, 2004 at 12:28:17PM +0100]:
> I think this goes farther then CDD, that proposal is about making base
> distribution smaller and making releases in each areas dependant only on
> "base" debian (lsb-1.3 in progeny). For example if I have desktop with
> XF86 4.3 and gnome, then I don't care how much RC bugs has apache or
> postfix. And when upstream makes new stable release of gnome 2.6 I
> don't have to wait for release of debian as whole, but only for new
> release of component 'gnome'.
>
> On other side, if I have server with mail services then I don't really
> care which version of gnome is in stable and how many bugs it has, but I
> do care if I had to wait several years for new release of Debian to have
> new release of my favourite MTA.
>
> I think that something like http://www.backports.org/ is more accurate
> to this.
Because we do have a deep and entangled dependency chain for many
packages?
I know that, say, most GNOME components will only depend on base and
on other GNOME components. But when you have GNOME components that are
made for accessing your database (once again, just as an example - I
am not a GNOME user), then you don't have cleanly separated sets
anymore. And then, you will find close-to-the-core GNOME components
depending on Mozilla (which will hopefully be on a different set) -
One more cross-relation. And if you keep searching, you will find many
more examples.
If we were to make componentized releases as soon as we got a fully
working and tested PostgreSQL release, problems will arise with users
of other sets - maybe gda2-postgres will break in the middle of its
stable lifetime. That's not what we are looking for in Debian.
And as soon as you mention desktop technologies sent to the server
side (Mono, Java servlets, whatnot), things will not get prettier.
Greetings,
--
Gunnar Wolf - gwolf@gwolf.cx - (+52-55)5630-9700 ext. 1366
PGP key 1024D/8BB527AF 2001-10-23
Fingerprint: 0C79 D2D1 2C4E 9CE4 5973 F800 D80E F35A 8BB5 27AF
--
To UNSUBSCRIBE, email to debian-devel-request@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
| |
| Chad Walstrom 2004-03-01, 12:33 pm |
| On Mon, Mar 01, 2004 at 12:10:16PM +0000, Colin Watson wrote:
> (Yes, this makes release management in general a very difficult
> problem, since you lose most of your degrees of freedom in the
> versions available for release.)
I would counter that release management may be simpler, in that specific
goals for release are more granular. Each component would have its own
release cycle and release manager. Teams of developers are more focused
on releasing software that interests them.
For this flexibility, you would increase complexity; this fact cannot be
ignored. Mathematically, it looks quite scary, but would it really be
that horrible in a practice? There are already numerous back-port
projects out there, allowing people to install Gnome 2.4 or Apache 2 on
woody. Developers WANT their software to run on multiple releases of
Debian. This is also an undeniable fact. People WANT to run a stable
Core and have newer software.
Implementation, on the other hand, does not match Debian's current
paradigm. It would be a big change internally to allow the use of
components. I'm not sure how Ian is going to make it work for Progeny
with the existing Debian and Fedora archives.
Using our own /etc/apt/sources.list as a template, I think the current
notation would change from this:
deb PROTO://HOST/debian RELEASE main contrib non-free
To this:
deb PROTO://HOST/debian COMPONENT RELEASE_1 RELEASE_2
e.g.
deb http://http.us.debian.org/debian core sarge sid
deb http://http.us.debian.org/debian x11-4.3 sid
deb http://http.us.debian.org/debian gnome-2.4 sarge
deb http://http.us.debian.org/debian python-2.3 sarge
deb http://http.us.debian.org/debian contrib sarge
deb http://http.us.debian.org/debian non-free sarge
*shrug* Just brainstormin'.
--
Chad Walstrom <chewie@wookimus.net> http://www.wookimus.net/
assert(expired(knowledge)); /* core dump */
| |
| Zenaan Harkness 2004-03-01, 6:34 pm |
| On Tue, 2004-03-02 at 03:53, Chad Walstrom wrote:
> On Mon, Mar 01, 2004 at 12:10:16PM +0000, Colin Watson wrote:
>
> I would counter that release management may be simpler, in that specific
> goals for release are more granular. Each component would have its own
> release cycle and release manager. Teams of developers are more focused
> on releasing software that interests them.
Release management for-each-sub-project may be simpler, but as someone
else mentioned, we would have a much greater need for parallel installs
and _that_ has would presumably make it (at least the distribution as a
whole) more complex.
It might be useful to work toward untangling the dependencies as much as
possible, and 'componentizing' would almost certainly assist with
identifying where the inter-component dependencies are, which is a good
thing.
> For this flexibility, you would increase complexity; this fact cannot be
> ignored. Mathematically, it looks quite scary, but would it really be
> that horrible in a practice? There are already numerous back-port
> projects out there, allowing people to install Gnome 2.4 or Apache 2 on
> woody. Developers WANT their software to run on multiple releases of
> Debian. This is also an undeniable fact. People WANT to run a stable
> Core and have newer software.
It's very appealing from a me-as-user point of view. I've been meaning
to set up a base-stable (for net, security, etc) + chroot unstable (for
desktop) but haven't yet gotten there. A more modularized Debian would
provide greater flexibility in a similar way...
> Implementation, on the other hand, does not match Debian's current
> paradigm. It would be a big change internally to allow the use of
> components. I'm not sure how Ian is going to make it work for Progeny
> with the existing Debian and Fedora archives.
Does progeny use both Debian _and_ Fedora as a base? If so I had no
idea.
> deb http://http.us.debian.org/debian x11-4.3 sid
This reminded me - there is talk (and hopefully plenty of action) to
break up the monolithix XFree distribution into base libs, drivers, etc.
This componentization seems to be the theme of the day.
Where I used to work, we ended up with a monolithic code base and at one
point split out "lib". We were using Aegis at the time, and created a
lib project which seemed to make sense. We realised after a long time
that the overhead of adding functions to "lib" was way too high, and so
no one did any more (since they had to create an extra 'change' which
was too much overhead for too little immediate gain for that one
programmer.
Nevertheless, the library became more stable, and after the initial
effort of untangling its dependencies so it didn't depend on anything
higher, this probed to be beneficial.
We wanted to create more layers, each layer depending only on
functionality from the layer below, but "we always had another deadline
and not enough willingness to stop for a few weeks to do it".
cheers
zen
--
To UNSUBSCRIBE, email to debian-devel-request@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
| |
| Daniel Ruoso 2004-03-01, 6:34 pm |
| > If we were to make componentized releases as soon as we got a fully
> working and tested PostgreSQL release, problems will arise with users
> of other sets - maybe gda2-postgres will break in the middle of its
> stable lifetime. That's not what we are looking for in Debian.
Yes, that's why the "stable" word in debian doesn't mean stable
software, but stable distribution. If we have a "componentized" linux we
can have stable software released earlier, but we couldn't get a stable
distribution.
The example of PostgreSQL and gda2-postgres illustrates very well the
problem. I can't see a way to "componentize" the choosing of wich
version of PostgreSQL gda2-postgres should expect except then being
non-componentized. That's what "stable distribution" means.
There are other ways of getting "stable software" released earlier in a
"not-so-stable distribution", and I think componentization is not the
way, but (as I've suggested in other posts) categorizing the "upstream
status" of the software (is the upstream stable? beta? alpha?
vapourware?) and create a dynamic release (just like testing) that tries
to group all the software that have a "stable upstream status". [Yes,
it's a proposal]
--
To UNSUBSCRIBE, email to debian-devel-request@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
| |
| Gunnar Wolf 2004-03-01, 7:34 pm |
| Daniel Ruoso dijo [Mon, Mar 01, 2004 at 07:55:57PM -0300]:
> Yes, that's why the "stable" word in debian doesn't mean stable
> software, but stable distribution. If we have a "componentized" linux we
> can have stable software released earlier, but we couldn't get a stable
> distribution.
>
> The example of PostgreSQL and gda2-postgres illustrates very well the
> problem. I can't see a way to "componentize" the choosing of wich
> version of PostgreSQL gda2-postgres should expect except then being
> non-componentized. That's what "stable distribution" means.
>
> There are other ways of getting "stable software" released earlier in a
> "not-so-stable distribution", and I think componentization is not the
> way, but (as I've suggested in other posts) categorizing the "upstream
> status" of the software (is the upstream stable? beta? alpha?
> vapourware?) and create a dynamic release (just like testing) that tries
> to group all the software that have a "stable upstream status". [Yes,
> it's a proposal]
....Remember that in Debian, even in Unstable, we do not want to
package unstable software - not, at least, without clearly marking it
as such (i.e. we have mozilla-* and mozilla-*-snapshot). Yes, I
understand your point, we should probably try each package to play
nicely with its 'neighbors' and only then with not-so-related
packages... But anyway, every package you upload to the archive should
be releasable should a freeze be declared right away.
Greetings,
--
Gunnar Wolf - gwolf@gwolf.cx - (+52-55)5630-9700 ext. 1366
PGP key 1024D/8BB527AF 2001-10-23
Fingerprint: 0C79 D2D1 2C4E 9CE4 5973 F800 D80E F35A 8BB5 27AF
--
To UNSUBSCRIBE, email to debian-devel-request@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
| |
| Ian Murdock 2004-03-24, 10:38 am |
| [ Sorry for the long delay in weighing in on this. ]
On Mon, 2004-03-01 at 07:10, Colin Watson wrote:
> On Mon, Mar 01, 2004 at 12:28:17PM +0100, Ondøej Surý wrote:
>
> But, actually, you do sometimes. To take one of your examples, both
> Apache and GNOME have PERL interfaces, and when the version of PERL in
> the base distribution changes (e.g. from 5.6 to 5.8) then all modules
> need to be rebuilt and both Apache and GNOME need to be updated in sync
> for everything to continue working. That's far from the only example of
> this sort of thing: Debian is rife with it.
Both of you have good points. Componentized Linux does go beyond
custom distros, although from Progeny's point of view, CL enables us to
build custom distros more efficiently, and that's why it's interesting
(we are after all in the business of building them). (It enables others
to build their custom distros more efficiently too, which is why
we're starting to see the beginnings of a community forming around it.)
In order to build custom distros more efficiently, we have to break
apart the monolithic distros into independent pieces. Our customers
want to be able to mix and match components to their requirements,
and their requirements almost invariably differ from each other.
Some want core bits from woody, with edge bits from sarge, and
perhaps even a few bits from sid. Others want some very
small subset of sarge with a customized kernel and a few of their own
packages thrown in for good measure. And so on (and on).
Previously, we had done this by hand, but that's simply not scalable.
Yes, Debian is modular in the sense that it's compromised of packages
that are theoretically independent from one another, but it's monolithic
from the feature/technology perspective, i.e., you pick a distro (sarge,
sid, etc.), and that implies certain features and technologies and
certain versions of them too. Yes, you can mix and match packages from
different distros, but you often do so at your own peril, e.g.
http://lists.debian.org/debian-deve.../msg01644.html.
Colin is right in that although Componentized Linux does help alleviate
one set of problems, it potentially creates an entirely new set of
problems. One problem, as Colin points out, is that dependencies are not
necessarily one-dimensional. If I install the C compiler, then I
want all the development packages for all the components I have
installed, and if I remove the C compiler, I want them all to be
removed; likewise, if I upgrade the Python component, then
I want Apache, Gnome, and the two dozen other components with Python
interfaces to be upgraded to match the new version. And there's the
potentially combinatorial explosion in QA and testing this implies..
Indeed, we are talking through those problems now (see
http://lists.progeny.com/listinfo/cl-workers if you are interested
in following the discussion or participating). If they can be solved,
would the effort pay off in spades? Undoubtedly. Could the same
work help scale the Debian release management process? Undoubtedly...
As the grand old man, I ought to resist the temptation to wax
historical, but like most grand old men, I cannot resist. ;-)
Debian was the first modular distro by necessity: it was the only way we
could think of to subdivide the work of creating an entire distro
amongst many different folks. At the time, most people said it
wouldn't work because the problems seemed intractable, but we applied
the notion of packages to those problems, came up with the idea of
using explicit dependency relationships and a set of
policies and guidelines to make sure everything worked together, and
I daresay the result has been nothing short of spectacular (can
you think of a single contemporary distro that isn't built this way)?
Componentized Linux is really just an application of the same basic
ideas to solve a new set of problems. How do you reduce the complexity
of managing a distro from thousands or tens of thousands of modules to
a few hundred modules? How do you further decouple those few hundred
modules so they can be more flexibly assembled in a greater number of
ways? How do you solve problems at the lowest layers of the module stack
so they don't have to be solved over and over again by different
parties? Given my experience with Debian, I'm confident the seemingly
intractable problems can be solved. And I'm equally confident that
when we solve them, the result will change the way distros are built.
--
Ian Murdock
317-578-8882 (office)
http://www.progeny.com/
http://ianmurdock.com/
"Don't look back--something might be gaining on you." --Satchel Paige
--
To UNSUBSCRIBE, email to debian-devel-request@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
| |
| Nathanael Nerode 2004-03-24, 7:36 pm |
| Ian Murdock wrote:
> Colin is right in that although Componentized Linux does help alleviate
> one set of problems, it potentially creates an entirely new set of
> problems. One problem, as Colin points out, is that dependencies are not
> necessarily one-dimensional. If I install the C compiler, then I
> want all the development packages for all the components I have
> installed, and if I remove the C compiler, I want them all to be
> removed; likewise, if I upgrade the Python component, then
> I want Apache, Gnome, and the two dozen other components with Python
> interfaces to be upgraded to match the new version.
This can be handled with a combination of a debfoster-like program and
metapackages...
> And there's the
> potentially combinatorial explosion in QA and testing this implies..
Now *that* is very difficult.
> Componentized Linux is really just an application of the same basic
> ideas to solve a new set of problems. How do you reduce the complexity
> of managing a distro from thousands or tens of thousands of modules to
> a few hundred modules?
Metapackages. Then, separating packages into "chosen" and "installed by
dependency" (a la debfoster) makes removal work properly. Also, tags will
help when you *want* to look through tens of thousands of modules.
> How do you further decouple those few hundred
> modules so they can be more flexibly assembled in a greater number of
> ways?
Highly accurate dependencies (which may (or may not) require a richer
"namespace" than the current packagename/version one). And slicker
conventions for libraries; symbol versioning helps eliminate a lot of the
"must... upgrade... everything" problem, as do all other
backward-compatibility features.
> How do you solve problems at the lowest layers of the module stack
> so they don't have to be solved over and over again by different
> parties?
Good question.
> Given my experience with Debian, I'm confident the seemingly
> intractable problems can be solved. And I'm equally confident that
> when we solve them, the result will change the way distros are built.
I think a lot of the tools have been prototyped already. :-)
--
Make sure your vote will count.
http://www.verifiedvoting.org/
--
To UNSUBSCRIBE, email to debian-devel-request@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
|
|
|
|
|