|
Home > Archive > Unix Programming > March 2006 > gzip-compressing an in-memory string using zlib 1.2.3
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
gzip-compressing an in-memory string using zlib 1.2.3
|
|
| A. Farber 2006-03-21, 3:17 am |
| Hello,
I've written a small Apache 1.3.x module and now would
like to add gzip-compression to the produced output.
However I seem to be missing something minor.
Using zlib 1.2.3 on OpenBSD -current, with gcc 3.3.5.
I have prepared a short test case listed below and wonder, why
does deflate(&zstr, Z_FINISH) return Z_OK and zstr.avail_out == 0?
I was actually expecting it to return Z_STREAM_END here:
laptop:src {599} cat gztest.c
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <zlib.h>
int main() {
char *src = "Hello world!\n", *dst;
z_stream zstr;
int slen = strlen(src), dlen, rc;
fprintf(stderr, "zlib %s: src=%s", zlibVersion(), src);
zstr.zalloc = Z_NULL;
zstr.zfree = Z_NULL;
zstr.opaque = Z_NULL;
zstr.next_in = src;
zstr.avail_in = slen;
/* add 16 to MAX_WBITS to enforce gzip format */
if (Z_OK ==
deflateInit2(&zstr, Z_DEFAULT_COMPRESSION, Z_DEFLATED,
MAX_WBITS + 16, MAX_MEM_LEVEL, Z_DEFAULT_STRATEGY))
fprintf(stderr, "deflateInit2() = Z_OK\n");
dlen = deflateBound(&zstr, slen);
fprintf(stderr, "deflateBound() = %u\n", dlen);
if ((dst = malloc(dlen)) == NULL)
perror("Out of memory");
zstr.next_out = dst;
zstr.avail_out = dlen;
if (Z_STREAM_END == (rc = deflate(&zstr, Z_FINISH)))
fprintf(stderr,
"deflate() = Z_STREAM_END, avail_out = %d\n",
zstr.avail_out);
else
fprintf(stderr,
"deflate() = %d, avail_out = %d, error = %s\n",
rc, zstr.avail_out, zstr.msg);
write(1, dst, zstr.total_out);
if(Z_OK == deflateEnd(&zstr))
fprintf(stderr, "deflateEnd() = Z_OK\n");
free(dst);
return 0;
}
laptop:src {603} gcc -Wall -ggdb gztest.c -o gztest -lz
laptop:src {604} ./gztest > blah.gz
zlib 1.2.3: src=Hello world!
deflateInit2() = Z_OK
deflateBound() = 27
deflate() = 0, avail_out = 0, error = (null)
deflateEnd() = Z_OK
laptop:src {605} xxd blah.gz
0000000: 1f8b 0800 0000 0000 0003 f348 cdc9 c957 ...........H...W
0000010: 28cf 2fca 4951 e402 0041 e4 (./.IQ...A.
laptop:src {606} gunzip blah.gz
gunzip: blah.gz: crc error
laptop:src {607} gunzip -c < blah.gz
gunzip: stdin: crc error
When I compare the blah.gz content with a file produced by the
real gzip, I see that the GZIP magic header (0x1f 0x8b) is there
which is good, but the real file seems to use few more bytes:
laptop:src {608} echo "Hello world" | gzip -c > real-blah.gz
laptop:src {609} xxd real-blah.gz
0000000: 1f8b 0800 0000 0000 0003 f348 cdc9 c957 ...........H...W
0000010: 28cf 2fca 49e1 0200 d5e0 39b7 0c00 0000 (./.I.....9.....
How can it be? I _did_ use the deflateBound() to allocate output buffer
in my program.
Regards
Alex
| |
| Mark Adler 2006-03-21, 3:17 am |
| A. Farber wrote:
> How can it be? I _did_ use the deflateBound() to allocate output buffer
> in my program.
deflateBound() does not take into account the gzip wrapper option in
zlib 1.2.3. This bug was fixed in the unreleased 1.2.3.1 zlib beta.
Just add 12 to the output of deflateBound(). If you use
deflateSetHeader(), then you'll also need to add the size of the file
name, comment, extra, and header crc fields if present.
mark
| |
| A. Farber 2006-03-21, 3:17 am |
| Wow, "straight from the horse's mouth"! :-)
Thank you very much, I've added 12 bytes and get the identical result
now
(BTW I was missing a "!" in the comparison string in the original post)
dlen = 12 + deflateBound(&zstr, slen);
laptop:src {630} ./gztest > blah.gz
laptop:src {631} xxd blah.gz
0000000: 1f8b 0800 0000 0000 0003 f348 cdc9 c957 ...........H...W
0000010: 28cf 2fca 4951 e402 0041 e4a9 b20d 0000 (./.IQ...A......
0000020: 00
laptop:src {632} echo "Hello world!" | gzip -c | xxd
0000000: 1f8b 0800 0000 0000 0003 f348 cdc9 c957 ...........H...W
0000010: 28cf 2fca 4951 e402 0041 e4a9 b20d 0000 (./.IQ...A......
0000020: 00
| |
| Jasen Betts 2006-03-21, 3:17 am |
| On 2006-03-20, A. Farber <Alexander.Farber@gmail.com> wrote:
> Hello,
>
> I've written a small Apache 1.3.x module and now would
> like to add gzip-compression to the produced output.
> However I seem to be missing something minor.
>
> Using zlib 1.2.3 on OpenBSD -current, with gcc 3.3.5.
>
> I have prepared a short test case listed below and wonder, why
> does deflate(&zstr, Z_FINISH) return Z_OK and zstr.avail_out == 0?
> I was actually expecting it to return Z_STREAM_END here:
>
it seems to be estimating the size incorrectly.
that could be a bug in zlib.
if I change
dlen = deflateBound(&zstr, slen);
to
dlen = deflateBound(&zstr, slen)+10;
It woarks, but I don't understand why.
as I understand the documentation in zlib.h deflateBound should be taking
the gzip header into account
--
Bye.
Jasen
|
|
|
|
|