Unix Programming - eliminating fixed-size buffers

Author

eliminating fixed-size buffers

Mohun Biswas

2004-06-17, 5:55 pm

Does anyone know of an open-source library or similar resource to aid in
Unix/C programming without fixed-length buffers? My code currently uses
a large stack construction buffer for things like strings and then
mallocs the result if it needs to live longer. But I've run into both
buffer overflow problems when the buffer size was set too small and
stack overflow problems when set too big, so I want to rewrite with
something based on malloc/realloc/free from the beginning. I could roll
my own but it feels like something that's been done before with the bugs
shaken out.

--
Thanks,
M.Biswas

Barry Margolin

2004-06-17, 5:55 pm

In article <JOkAc.129284$Ly.26527@attbi_s01>,
Mohun Biswas <m.biswas@invalid.addr> wrote:

> Does anyone know of an open-source library or similar resource to aid in
> Unix/C programming without fixed-length buffers? My code currently uses
> a large stack construction buffer for things like strings and then
> mallocs the result if it needs to live longer. But I've run into both
> buffer overflow problems when the buffer size was set too small and
> stack overflow problems when set too big, so I want to rewrite with
> something based on malloc/realloc/free from the beginning. I could roll
> my own but it feels like something that's been done before with the bugs
> shaken out.

I don't know if there's any library involved, but most GNU utilities
avoid the use of fixed-size buffers to hold the lines of the files
they're reading. So take a look at programs like "tail", "sed", etc.
and see if there's some code you can borrow.

--
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***

William Ahern

2004-06-17, 11:51 pm

Mohun Biswas <m.biswas@invalid.addr> wrote:
> Does anyone know of an open-source library or similar resource to aid in
> Unix/C programming without fixed-length buffers? My code currently uses
> a large stack construction buffer for things like strings and then
> mallocs the result if it needs to live longer. But I've run into both
> buffer overflow problems when the buffer size was set too small and
> stack overflow problems when set too big, so I want to rewrite with
> something based on malloc/realloc/free from the beginning. I could roll
> my own but it feels like something that's been done before with the bugs
> shaken out.

asprintf(3) seems to be widely available, at least in the FOSS world of
Linux and *BSD. It takes a pointer to pointer of type char, and if
asprintf(3) returns successfully it will contain a newly malloc'd string.

Otherwise, you can use snprintf(3), which returns the logical length of the
new string. You can get something similar to asprintf(3) like so:

len = snprintf(NULL,0,fmt,...);
str = malloc(len + 1);
...
(void)snprintf(str,len + 1,fmt,...);

Mohun Biswas

2004-06-20, 10:32 pm

William Ahern wrote:
> asprintf(3) seems to be widely available, at least in the FOSS world of
> Linux and *BSD. It takes a pointer to pointer of type char, and if
> asprintf(3) returns successfully it will contain a newly malloc'd string.
>
> Otherwise, you can use snprintf(3), which returns the logical length of the
> new string. You can get something similar to asprintf(3) like so:
>
> len = snprintf(NULL,0,fmt,...);
> str = malloc(len + 1);
> ...
> (void)snprintf(str,len + 1,fmt,...);

Thanks, this was very helpful. I chose to implement my own for
portability and to do it somewhat differently, to avoid having to format
twice in most cases. Viz:

char *
my_asprintf(const char *fmt, ...)
{
va_list ap;
unsigned len;
char *newstr;
char buf[8192]; // not a hard limit
void *ptr; // hack for C++ compatibility ...

va_start(ap, fmt);

len = vsnprintf(buf, sizeof(buf), fmt, ap);
if ((ptr = malloc(len + 1)) == NULL) {
return NULL;
} else {
newstr = (char *)ptr;
}

if (len < sizeof(buf)) {
strcpy(newstr, buf);
} else {
(void) vsnprintf(newstr, len + 1, fmt, ap);
}

va_end(ap);
return newstr;
}

Also, I see there's a piece of freeware (BSD-ish license) called librock
which offers similar functionality and more.

--
Thanks,
M.Biswas

James Antill

2004-06-26, 10:11 am

On Fri, 18 Jun 2004 16:11:52 +0000, Mohun Biswas wrote:

> Thanks, this was very helpful. I chose to implement my own for
> portability and to do it somewhat differently, to avoid having to format
> twice in most cases. Viz:

You might want to look at, http://www.and.org/vstr/printf_comparison.html
.... you aren't the first person to want a portable allocating sprintf. And
personally I'd recommend against doing some like below, vsnprintf() can
return -1 on error (and also when it runs out of space on older
implementations). You also need to va_copy(), or va_end()va_start() before
the second call to vsnprintf().

As a final 1cent, you originally asked for something to aid in Unix/C
proraming without fixed sized buffers. While an allocating *sprintf()
helps, you really want something that has the other string comparison
operations.
Again, there are more than a few pre-made choices:

http://www.and.org/vstr/comparison.html

> char *
> my_asprintf(const char *fmt, ...)
> {
[...]
> }
> }
> Also, I see there's a piece of freeware (BSD-ish license) called librock
> which offers similar functionality and more.

--
James Antill -- james@and.org
Need an efficient and powerful string library for C?
http://www.and.org/vstr/

Paul Hsieh

2004-06-26, 10:11 am

Mohun Biswas <m.biswas@invalid.addr> wrote in message news:<JOkAc.129284$Ly.26527@attbi_s01>...
> Does anyone know of an open-source library or similar resource to aid in
> Unix/C programming without fixed-length buffers? My code currently uses
> a large stack construction buffer for things like strings and then
> mallocs the result if it needs to live longer. But I've run into both
> buffer overflow problems when the buffer size was set too small and
> stack overflow problems when set too big, so I want to rewrite with
> something based on malloc/realloc/free from the beginning. I could roll
> my own but it feels like something that's been done before with the bugs
> shaken out.

Why don't you give this a try:

http://bstring.sf.net/

Its emphasis is on strings, but can be used for holding generic
buffers without issue.

--
Paul Hsieh
http://www.pobox.com/~qed/