|
Home > Archive > Debian Developers > September 2006 > replicating package compression used by dpkg-deb
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
replicating package compression used by dpkg-deb
|
|
| Ian Bruce 2006-09-28, 7:35 am |
| I'm working on a system for differential transport of deb packages using
zsync (client-side rsync). This involves decompressing the gzip'ed tar
archives in the package and then recompressing them later. In order that
the reconstructed package match the "Size" and "MD5sum" fields in the
Apt packages list, the recompression step must produce exactly the same
result as dpkg-deb did originally. This turns out to be a problem:
$ ar -x /var/cache/apt/archives/fuse-utils_2.5.3-4.1_i386.deb
$
$ ls
control.tar.gz data.tar.gz debian-binary
$
$ for x in -{1,2,3,4,5,6,7,8,9}\ {-n,}\ {--rsyncable,}
> do
> gzip -dc data.tar.gz | gzip $x | cmp -s data.tar.gz - && echo $x
> done
$
So it seems that with the currently shipping versions of gzip (I tried
both "stable/updates" and "testing"), there is actually no way to
exactly replicate the compression produced by the standard package build
system.
It appears that dpkg-deb does not exec gzip, and it's not dynamically
linked with anything except glibc. I suppose that it's statically linked
against zlib1g or something like it. So the question is, how can the
exact compression algorithm used by dpkg-deb be made available for
another piece of software? Is it something that's well-specified, or is
it liable to change at any moment?
I see that the Packages lists have started to include SHA-1 and SHA-256
hashes, but I gather that these are not yet in use by any standard part
of the Debian system. Is there any chance that these could be
respecified to be the hash of the uncompressed package contents, like
this?
$ cat debian-binary control.tar data.tar | sha256sum
In addition to facilitating alternative package distribution systems
such as the one I'm working on, this would have the advantage of
allowing the substitution of other compression algorithms like bzip2,
without requiring any change in the Debian package distribution
infrastructure.
Of course the whole problem would be radically simplified if the
"--rsyncable" option were used when the packages were being built. Does
anyone know what the rationale for not doing that is?
-- Ian Bruce
--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
| |
| Mark Brown 2006-09-28, 7:35 am |
| Followup to the other lists too...
On Thu, Sep 28, 2006 at 12:15:21PM +0100, Mark Brown wrote:
> On Thu, Sep 28, 2006 at 03:45:16AM -0700, Ian Bruce wrote:
>
>
> dpkg is statically linked against zlib (you really don't want to break
> the ability to unpack packages if you can avoid it). I presume dpkg-deb
> is too though I haven't checked. The precise output of zlib can vary
> between versions and depending on the options used when running it.
>
>
> It's not implemented by zlib. There is an open bug report asking for
> it with a link to a patch but I wasn't happy with the patch since it
> is enabled using an environment variable and wanted to rework it to have
> an API before forwarding upstream. I've not yet had sufficient
> enthusiasm to do that; current progress suggests that you shouldn't hold
> your breath on that one.
>
> --
> "You grabbed my hand and we fell into it, like a daydream - or a fever."
>
>
> --
> To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
> with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
>
>
--
"You grabbed my hand and we fell into it, like a daydream - or a fever."
--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
| |
| Mark Brown 2006-09-28, 7:35 am |
| On Thu, Sep 28, 2006 at 03:45:16AM -0700, Ian Bruce wrote:
> It appears that dpkg-deb does not exec gzip, and it's not dynamically
> linked with anything except glibc. I suppose that it's statically linked
> against zlib1g or something like it. So the question is, how can the
> exact compression algorithm used by dpkg-deb be made available for
> another piece of software? Is it something that's well-specified, or is
> it liable to change at any moment?
dpkg is statically linked against zlib (you really don't want to break
the ability to unpack packages if you can avoid it). I presume dpkg-deb
is too though I haven't checked. The precise output of zlib can vary
between versions and depending on the options used when running it.
> Of course the whole problem would be radically simplified if the
> "--rsyncable" option were used when the packages were being built. Does
> anyone know what the rationale for not doing that is?
It's not implemented by zlib. There is an open bug report asking for
it with a link to a patch but I wasn't happy with the patch since it
is enabled using an environment variable and wanted to rework it to have
an API before forwarding upstream. I've not yet had sufficient
enthusiasm to do that; current progress suggests that you shouldn't hold
your breath on that one.
--
"You grabbed my hand and we fell into it, like a daydream - or a fever."
--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
| |
| Ian Bruce 2006-09-28, 1:22 pm |
| On Thu, 28 Sep 2006 03:45:16 -0700
Ian Bruce <ian_bruce@fastmail.fm> wrote:
> So it seems that with the currently shipping versions of gzip (I tried
> both "stable/updates" and "testing"), there is actually no way to
> exactly replicate the compression produced by the standard package
> build system.
>
> It appears that dpkg-deb does not exec gzip, and it's not dynamically
> linked with anything except glibc. I suppose that it's statically
> linked against zlib1g or something like it. So the question is, how
> can the exact compression algorithm used by dpkg-deb be made available
> for another piece of software? Is it something that's well-specified,
> or is it liable to change at any moment?
It turns out that the zlib1g-dev package contains a program called
"minigzip" in source form. This is what's needed; "minigzip -9"
reproduces exactly the compression used by dpkg-deb, unlike regular
gzip.
-- Ian Bruce
--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
| |
| Mark Brown 2006-09-28, 1:22 pm |
| On Thu, Sep 28, 2006 at 06:50:53AM -0700, Ian Bruce wrote:
> It turns out that the zlib1g-dev package contains a program called
> "minigzip" in source form. This is what's needed; "minigzip -9"
> reproduces exactly the compression used by dpkg-deb, unlike regular
> gzip.
This may not produce identical results if the version of zlib used to
produce the package differs from that on your system.
--
"You grabbed my hand and we fell into it, like a daydream - or a fever."
--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
| |
| Goswin von Brederlow 2006-09-30, 1:49 am |
| Ian Bruce <ian_bruce@fastmail.fm> writes:
> On Thu, 28 Sep 2006 03:45:16 -0700
> Ian Bruce <ian_bruce@fastmail.fm> wrote:
>
>
> It turns out that the zlib1g-dev package contains a program called
> "minigzip" in source form. This is what's needed; "minigzip -9"
> reproduces exactly the compression used by dpkg-deb, unlike regular
> gzip.
Have you tried gzip -n?
Anyway, look at the source of dpkg-deb and then use the same (static
linking of zlib I assume).
MfG
Goswin
--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
|
|
|
|
|