Unix Programming - grouping system headers

Author

grouping system headers

Mohun Biswas

2004-06-16, 5:57 pm

I've juat been looking through Marc Rochkind's new "Advanced Unix
Programming" book
(<http://www.amazon.com/exec/obidos/t...=glance&s=books> )
and noticed that he recommends putting a bunch of commonly-used header
files in a common one, then including in source files. E.g.

% cat defs.h
/* (This leaves out some complexity but you get the idea) */
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/types.h>

% cat foo.h
#include "defs.h"

I've often wondered whether or not this was considered good form but
seeing it recommended in a book made me decide to research it some. On
the one hand, it seems like a lot of work was invested in separating
interfaces cleanly by header and there must be a good reason for doing
so. On the other, there's nothing more annoying than adding "#include
<string.h>" to 500 files, one at a time, as you find the need for strlen
in each (and each instance involves a failed compile, then an edit, then
a successful compile). Also, when porting to non-POSIX platforms such as
Windows you get lots of C/C++ source files sprinkled with constructs like

#ifdef WIN32
#include <malloc.h>
#else /*WIN32*/
#include <stdlib.h>
#endif /*WIN32*/

in order to get malloc() declared. It would certainly be nice to take
care of that all in one place.

One could imagine doing interesting things with header grouping. For
instance a "posix.h" file containing the right mix of includes and
defines to provide a straight POSIX environment, an "SUSv3.h" going
somewhat further, etc.

At first blush the criticism of defs.h would seem to be the same as for
global data. But in this case the data is read-only and the interfaces
are standardized, meaning they won't generally be changing from year to
year or platform to platform. So - what is the downside of the defs.h
approach?

--
Thanks,
M.Biswas

Bjorn Reese

2004-06-16, 5:57 pm

On Wed, 16 Jun 2004 15:22:56 +0000, Mohun Biswas wrote:

> year or platform to platform. So - what is the downside of the defs.h
> approach?

Compilation speed is often used to argue against such an approach.

--
mail1dotstofanetdotdk

Mohun Biswas

2004-06-17, 5:55 pm

Bjorn Reese wrote:
> On Wed, 16 Jun 2004 15:22:56 +0000, Mohun Biswas wrote:
>
>
>
>
> Compilation speed is often used to argue against such an approach.

I could see that being an argument against a file which includes *every*
header on the system "just in case". But in the (common) situation where
the great majority of source modules need stdlib.h, unistd.h, stdio.h,
sys/types.h, and perhaps a dozen more, is there a downside to grouping
these and perhaps exposing a few modules to a few APIs they don't use?
After all, even when you bring in (say) stdlib.h only when needed,
you're dragging in hundreds of lines and dozens of other interfaces you
don't need at the same time. If preprocessing time is so critical, why
don't we have fopen.h, fclose.h, popen.h, etc?

--
Thanks,
M.Biswas

Norm Dresner

2004-06-17, 5:55 pm

"Mohun Biswas" <m.biswas@invalid.addr> wrote in message
news:jyiAc.63285$0y.28923@attbi_s03...
> Bjorn Reese wrote:
>
> I could see that being an argument against a file which includes *every*
> header on the system "just in case". But in the (common) situation where
> the great majority of source modules need stdlib.h, unistd.h, stdio.h,
> sys/types.h, and perhaps a dozen more, is there a downside to grouping
> these and perhaps exposing a few modules to a few APIs they don't use?
> After all, even when you bring in (say) stdlib.h only when needed,
> you're dragging in hundreds of lines and dozens of other interfaces you
> don't need at the same time. If preprocessing time is so critical, why
> don't we have fopen.h, fclose.h, popen.h, etc?

I've been doing this since 1983 using a host of compilers in CP/M, VMS,
MS-DOS, WinXX, Linux, and I'm sure at least a half-dozen more systems I've
gratefully forgotten about with AFAICR not a single problem. Sure it takes
a few milliseconds (probably now uS in systems with substantial disk caches)
more per compile but it also takes zero effort when writing the modules. I
even have it in my "prototype" modules in some systems.

Norm

Alan Balmer

2004-06-17, 5:55 pm

On Wed, 16 Jun 2004 21:55:36 +0200, "Bjorn Reese"
<breese@see.signature> wrote:

>On Wed, 16 Jun 2004 15:22:56 +0000, Mohun Biswas wrote:
>
>
>Compilation speed is often used to argue against such an approach.

In most cases, it's probably not a significant factor, especially if
most of the headers would need to be included anyway - the extra level
won't add to the time significantly. In fact, for compilers which can
pre-compile headers, it may speed up the compile.

--
Al Balmer
Balmer Consulting
removebalmerconsultingthis@att.net

joe@invalid.address

2004-06-17, 5:55 pm

Alan Balmer <albalmer@att.net> writes:

> On Wed, 16 Jun 2004 21:55:36 +0200, "Bjorn Reese"
> <breese@see.signature> wrote:
>
>
> In most cases, it's probably not a significant factor, especially if
> most of the headers would need to be included anyway - the extra level
> won't add to the time significantly. In fact, for compilers which can
> pre-compile headers, it may speed up the compile.

I think in most cases this is right. However, I've talked to one or
two people who were involved in very large projects, in which header
files included other header files overmuch. At some point the systems
wouldn't even build due to lack of memory.

Extreme cases, certainly, but how common is it to have to add a header
file to 500 source files? It seems to me like simple good housekeeping
to not include things in header files unless their needed for the
header file to compile.

If it really is necessary to add an include to 500 source files, I
would say use sed or PERL or something of the kind which was designed
to do things like that.

Joe
--
Folks who don't know why America is the Land of Promise should be here
during an election campaign.
-- Milton Berle

Mohun Biswas

2004-06-17, 11:51 pm

joe@invalid.address wrote:
> Extreme cases, certainly, but how common is it to have to add a header
> file to 500 source files?

I think you're missing my point ... at the least, you missed where I
said "... adding [a header] to 500 files, *one at a time*". You're right
that it's rare to have to add a header to 500 files *at once* and that
if you had to you'd use sed or perl. But that wasn't the point. The
point is that you might write a source module, let's call it foo.c. The
first version doesn't need sys/types.h so you don't list it. Then later
you add something basic like a stat() or a getpid() and when you next
compile you get an error or warning. After a few seconds or minutes of
analysis you realize you need a new header. So then you run "man stat"
to see what the required headers are. Then you compare that list with
what you already have and discover that sys/types.h seems to be the
culprit. You add that and compile again, and it succeeds.

In this model you've had to (a) do two compiles instead of one, (b)
invest a few minutes thinking about header-file semantics, (c) run "man
stat" and read at least the top part, and (d) re-edit the source file
(naturally some of these details may differ, e.g. IDE users may have a
way to get required headers without running "man").

Next: in my experience, a high proportion of C source files are going to
need headers like stdlib.h, stdio.h, and sys/types.h. It's hard to know
the exact percentage but let's say it's 60% ... you'd only need 800
source modules to have to go through the above rigamarole 500 times *per
header* - quite possibly a lot more than 800 times overall. And how many
projects have that many (and far more) source files? Lots.

But anyway, exact numbers aren't important. The point is that having to
discover, by trial and error, the needed header(s) every time a change
involves a new API, has a significant cost. And I'm trying to determine
the cost of doing things the other way in order to weigh the two.

FWIW I just looked at GNU make's source code (happened to have it lying
around) and it uses a make.h file to cluster common headers such as
<string.h> and <stdlib.h>.

--
Thanks,
M.Biswas

joe@invalid.address

2004-06-17, 11:51 pm

Mohun Biswas <m.biswas@invalid.addr> writes:

> joe@invalid.address wrote:
>
> I think you're missing my point ... at the least, you missed where I
> said "... adding [a header] to 500 files, *one at a time*". You're
> right that it's rare to have to add a header to 500 files *at once*
> and that if you had to you'd use sed or perl. But that wasn't the
> point. The point is that you might write a source module, let's call
> it foo.c. The first version doesn't need sys/types.h so you don't
> list it. Then later you add something basic like a stat() or a
> getpid() and when you next compile you get an error or
> warning. After a few seconds or minutes of analysis you realize you
> need a new header. So then you run "man stat" to see what the
> required headers are. Then you compare that list with what you
> already have and discover that sys/types.h seems to be the
> culprit. You add that and compile again, and it succeeds.

Ok, so you're editing the file and adding new stuff to it. Why is
adding a header include worse than adding a function call, and all the
logic that might entail?

> In this model you've had to (a) do two compiles instead of one, (b)
> invest a few minutes thinking about header-file semantics, (c) run
> "man stat" and read at least the top part, and (d) re-edit the source
> file (naturally some of these details may differ, e.g. IDE users may
> have a way to get required headers without running "man").

If you don't know what stat() requires, then of course you have to
read the man page. You probably will also have to spend more than a
few minutes thinking about more than header file semantics, and
obviously you may need to re-edit the source file if you added stat()
without knowing what header file it required. All this time makes it a
trivial matter to have to type #include <stdlib.h> or whatever.

> Next: in my experience, a high proportion of C source files are
> going to need headers like stdlib.h, stdio.h, and sys/types.h. It's
> hard to know the exact percentage but let's say it's 60% ... you'd
> only need 800 source modules to have to go through the above
> rigamarole 500 times *per header* - quite possibly a lot more than
> 800 times overall. And how many projects have that many (and far
> more) source files? Lots.

Again, the amount of time figuring out that you need to either know
what header file is required when adding a new call, or simply reading
the man page to find out, is often small potatoes compared to the time
it takes to figure out the logic of the change which requires it. If
you include headers in the source file, then you don't have to look at
the header file in order to know what's already included. If you have
some standard header that nobody looks at, you'll need to do a compile
to find out that what you need isn't there.

> But anyway, exact numbers aren't important. The point is that having
> to discover, by trial and error, the needed header(s) every time a
> change involves a new API, has a significant cost. And I'm trying to
> determine the cost of doing things the other way in order to weigh
> the two.

It may or may not have a significant cost. In my experience, it
doesn't, but it has a lot to do with how the system is designed as
well.

> FWIW I just looked at GNU make's source code (happened to have it
> lying around) and it uses a make.h file to cluster common headers such
> as <string.h> and <stdlib.h>.

GNU make is a pretty mature program so I'd expect that there isn't a
lot of morphing of the type we're talking about. I expect Paul Smith
could speak to that more intelligently than I could though.

Joe
--
Folks who don't know why America is the Land of Promise should be here
during an election campaign.
-- Milton Berle

Mohun Biswas

2004-06-17, 11:51 pm

ERATHOLE

--
Thanks,
M.Biswas

joe@invalid.address

2004-06-20, 10:32 pm

Mohun Biswas <m.biswas@invalid.addr> writes:

> ERATHOLE

what does this mean?

Joe
--
Folks who don't know why America is the Land of Promise should be here
during an election campaign.
-- Milton Berle

Mohun Biswas

2004-06-20, 10:32 pm

joe@invalid.address wrote:
> Mohun Biswas <m.biswas@invalid.addr> writes:
>
>
>
>
> what does this mean?
>
> Joe

It means this topic is looking like a rathole and I propose we avoid
digging deeper into it. C.f. ENOENT, E2BIG, etc.

--
Thanks,
M.Biswas

Alan Balmer

2004-06-20, 10:32 pm

On Thu, 17 Jun 2004 22:48:07 GMT, Mohun Biswas <m.biswas@invalid.addr>
wrote:

> Then later
>you add something basic like a stat() or a getpid() and when you next
>compile you get an error or warning. After a few seconds or minutes of
>analysis you realize you need a new header.

Sorry, I'm not sympathetic to self-generated troubles of this kind. If
you are introducing a new function to the program, you ensure that the
proper prototype is there as well. It's part of the job. No analysis
is needed at this point - *every* function should be prototyped.

--
Al Balmer
Balmer Consulting
removebalmerconsultingthis@att.net