Hello all,

I am looking for a scheme where a UNIX process listening on a
TCP-socket can pass incoming network requests to existing child
processes (so to not have to fork for each request).

My first try was by letting the child processes also doing accept() on
the listen socket (after finishing their initial job). Then I tried a
variant where all processes (parent and children) do select(), followed
by accept(). On Solaris 2.7 and 2.9 however, neither approach gives an
even work subdivision; on Linux (kernel 2.2.15) it does however. But I
want my program to be portable (and use a documented way of working).

So I looked at file descriptor (fd) passing (parent sends fd returned
by accept() to given child process). A pipe to each child does not
scale well (2n fd's for n child processes), so I'm looking at UNIX
sockets.

Using UNIX sockets, I wonder whether a single socket file would suit
all processes (less files needed), but that would probably give me the
same trouble as I had in my multi-accept() scheme. And: even with
SO_REUSEADDR on, bind() simply fails if the socket file already exists.
A bind()/fork() scheme would circumvent that, but this bind() behavior
suggests that this is maybe not something one would want to use.

So I end up with the one-UNIX-socket-per-process setup. Detailing that,
it looks a bit expensive to create and delete a socket file with each
child being born and exiting (possible slowdown due to disk I/O). So I
guess I should reuse socket files as much as possible?

Advise on this or suggestions for better approaches are very welcome!

Thanks,

Jan Stap
Eindhoven, The Netherlands

[ Post a follow-up to this message ]

Jan Stap wrote:
> Hello all,
>
> I am looking for a scheme where a UNIX process listening on a
> TCP-socket can pass incoming network requests to existing child
> processes (so to not have to fork for each request).
>
> My first try was by letting the child processes also doing accept() on
> the listen socket (after finishing their initial job). Then I tried a
> variant where all processes (parent and children) do select(), followed
> by accept(). On Solaris 2.7 and 2.9 however, neither approach gives an
> even work subdivision; on Linux (kernel 2.2.15) it does however. But I
> want my program to be portable (and use a documented way of working).
>
> So I looked at file descriptor (fd) passing (parent sends fd returned
> by accept() to given child process). A pipe to each child does not
> scale well (2n fd's for n child processes), so I'm looking at UNIX
> sockets.
>
> Using UNIX sockets, I wonder whether a single socket file would suit

[snip]

You could take a look at how apache handles this.
Apache is extremely portable. They seem to have several schemes to do
this, but fd-passing + pre-forked children is certainly one of them.

HTH,
AvK

[ Post a follow-up to this message ]

Jan Stap wrote:
> Hello all,
>
> I am looking for a scheme where a UNIX process listening on a
> TCP-socket can pass incoming network requests to existing child
> processes (so to not have to fork for each request).

If you want to use processes, not threads, you can use sendmsg(2) to
pass the file descriptor to the child.  But I wonder how you'd determine
which child is idle and which one is still busy processing requests.
Overall it seems like you'd need a good deal of IPC.  Maybe forking
wouldn't be that much slower (did you measure the overhead?  is it
really too much?).

(socketpair() before forking should work, I guess)

> My first try was by letting the child processes also doing accept() on
> the listen socket (after finishing their initial job). Then I tried a
> variant where all processes (parent and children) do select(), followed
> by accept(). On Solaris 2.7 and 2.9 however, neither approach gives an
> even work subdivision; on Linux (kernel 2.2.15) it does however. But I
> want my program to be portable (and use a documented way of working).

Maybe the equal workload distribution isn't important (or is it); only
that an available child quickly gets the connection.

> Advise on this or suggestions for better approaches are very welcome!

Hm, pthreads might work, but some people don't like them (I do).  Just
have a structure where child threads can wait and let the parent post a
condition when a connection is ready (setting a slot in the structure).
Fast and simple.  Avoid sharing anything else between threads; use
messaging like in the process case.

--
I believe in Karma.  That means I can do bad things to people
all day long and I assume they deserve it.
Dogbert

[ Post a follow-up to this message ]

Jan Stap wrote:

> My first try was by letting the child processes also doing accept() on
> the listen socket (after finishing their initial job).

This is what Apache prior to 2.x does. Does Apache work on your
Solaris?

[ Post a follow-up to this message ]

moi wrote:
> Jan Stap wrote: 
> [snip]
>
> You could take a look at how apache handles this.
> Apache is extremely portable. They seem to have several schemes to do
> this, but fd-passing + pre-forked children is certainly one of them.
>
> HTH,
> AvK
I'll have a look, thanks!

Jan Stap

[ Post a follow-up to this message ]

Ulrich Hobelmann wrote:
> Jan Stap wrote: 
>
> If you want to use processes, not threads, you can use sendmsg(2) to
> pass the file descriptor to the child.  But I wonder how you'd determine
> which child is idle and which one is still busy processing requests.
> Overall it seems like you'd need a good deal of IPC.  Maybe forking
> wouldn't be that much slower (did you measure the overhead?  is it
> really too much?).
>
> (socketpair() before forking should work, I guess)

Yep, sendmsg() I want to use when using sockets for transporting a file
descriptor. The child status I keep in an array of structures in shared
memory. Initially I planned to pass the file descriptor also via this
array, until I realized that the receiving side can do little with
that... Using socketpair() would give me n file descriptors on the
parent process side for n children, which I don't really like.

The overhead of 1 session per child is little for my application: the
program (an SMTP mail filter) is running on a Sun Fire V240 with 2G of
mem with Solaris 2.9; the filter is in loopback mode with Postfix
("after-queue" setup), which uses one SMTP session per email currently.
Mail load is about 15.000 messages/hour, thus giving about 4
connections/second to the filter. System load is <1.0 and about 1.5G of
memory is free. Ok, it's a big machine, but still.

But for usage on smaller machines it may pay off to pass more work to a
given child process. And I find it more elegant. Using it specifically
with Postfix, one could enable connection caching in Postfix (do >1
mail per SMTP session) and leaving the filter at 1 session per child.
 
>
> Maybe the equal workload distribution isn't important (or is it); only
> that an available child quickly gets the connection.

Maybe a little bit: the more even the workload is spread, the less
fork()'s are needed in general (given a set idle timeout per child).
 
>
> Hm, pthreads might work, but some people don't like them (I do).  Just
> have a structure where child threads can wait and let the parent post a
> condition when a connection is ready (setting a slot in the structure).
>   Fast and simple.  Avoid sharing anything else between threads; use
> messaging like in the process case.

I will try that at some point in my free time, but cannot spend the
amount of hours needed for conversion in my boss'es time :-)

Thanks for your answer!

> --
> I believe in Karma.  That means I can do bad things to people
> all day long and I assume they deserve it.
> 	Dogbert

[ Post a follow-up to this message ]

Maxim Yegorushkin wrote:
> Jan Stap wrote:
> 
>
> This is what Apache prior to 2.x does. Does Apache work on your
> Solaris?

Are you sure? I did an strace (Linux, kernel 2.4.20) on the apache
processes (Apache 1.3.26; hardly any web traffic) and found:

- main process: loop with select() just for a one-second timeout,
followed by
a wait4(-1, .., WNOHANG, NULL); so just wait for a child to die as
it seems
- one child process: select(31, [28 29 30], NULL, NULL, NULL)
handle a connection?
- the other child processes: semop(..) (same semaphore for all)

I need to check the code to get more clarity. Btw. Solaris runs Apache
1.x ok (on a Solaris 2.8 box over here, but for which I don't have root
rights).

But thanks for your answer anyway!

Jan Stap

[ Post a follow-up to this message ]

Jan Stap wrote:
> Maxim Yegorushkin wrote: 
>
> Are you sure? I did an strace (Linux, kernel 2.4.20) on the apache
> processes (Apache 1.3.26; hardly any web traffic) and found:

Pretty sure, although it's been quite a while since I last browsed the
code.

>  - main process: loop with select() just for a one-second timeout,
> followed by
>    a wait4(-1, .., WNOHANG, NULL); so just wait for a child to die as
> it seems
>  - one child process: select(31, [28 29 30], NULL, NULL, NULL)
>    handle a connection?
>  - the other child processes: semop(..) (same semaphore for all)

IIRC, they protected accept() call with a mutex/semaphore lock to avoid
thundreing herd problem on old kernels.

[ Post a follow-up to this message ]

Jan Stap wrote:
> Yep, sendmsg() I want to use when using sockets for transporting a file
> descriptor. The child status I keep in an array of structures in shared
> memory. Initially I planned to pass the file descriptor also via this

shared memory?  I find that much more ugly than pthreads, but YYMV.

> array, until I realized that the receiving side can do little with
> that... Using socketpair() would give me n file descriptors on the
> parent process side for n children, which I don't really like.

Then use pthreads ;)
 
>
> I will try that at some point in my free time, but cannot spend the
> amount of hours needed for conversion in my boss'es time :-)

You just use pthread_create with a function, instead of fork and a
function (or exec).  Instead of the shared memory (aaah!) you can use
something like this:
struct Message {
pthread_mutex_t mutex;
pthread_cond_t messageArrived;
/* and the message */
};

and do simple setmessage and getmessage operations on that.  Much faster
than IPC over pipes or sockets (at least on my system), no
de/serializing, and cleaner (IMHO) than shared memory.

Also, if you use fork without exec right now, you don't have to close
all the parent's file descriptors, you don't have to free the parent's
memory and other resources, because it's only one address space.  If the
parent frees some of its memory, it will go back to the OS, because
there are no other processes holding on to it (after forking you inherit
all the parent's crap).

--
I believe in Karma.  That means I can do bad things to people
all day long and I assume they deserve it.
Dogbert

[ Post a follow-up to this message ]

Ulrich Hobelmann wrote:

> Jan Stap wrote: 
>
> shared memory?  I find that much more ugly than pthreads, but YYMV.

I hear more people speaking out their dislike of shared memory. Of
course one has to set the protection bits to prevent security leaks,
but what is so ugly about it?

[snip]

> You just use pthread_create with a function, instead of fork and a
> function (or exec).  Instead of the shared memory (aaah!) you can use
> something like this:
> struct Message {
> 	pthread_mutex_t mutex;
> 	pthread_cond_t messageArrived;
> 	/* and the message */
> };
>
> and do simple setmessage and getmessage operations on that.  Much faster
> than IPC over pipes or sockets (at least on my system), no
> de/serializing, and cleaner (IMHO) than shared memory.
>
> Also, if you use fork without exec right now, you don't have to close
> all the parent's file descriptors, you don't have to free the parent's
> memory and other resources, because it's only one address space.  If the
> parent frees some of its memory, it will go back to the OS, because
> there are no other processes holding on to it (after forking you inherit
> all the parent's crap).

Threads seemed more complex and more kernel-dependent to me at first,
which led me to use the fork() scheme (I'm fairly new to UNIX
programming). But I want to learn, so I will try a threaded version,
once I find the time to do the conversion. For now, I need to extend
the fork()-based version that's now in production. But complexity with
pthreads seems limited, judging from what you write.

Thanks!

Jan Stap

> --
> I believe in Karma.  That means I can do bad things to people
> all day long and I assume they deserve it.
> 	Dogbert

[ Post a follow-up to this message ]