Hello group,


I'm writing an HTTP server that responds much quicker to serially
sent requests than to parallel requests.  For example using
www.acme.com's http_load, if I send 40 request one by one the
total time to get all responses is 70 msec, but if I send 40 all
at once, it is over three seconds!  I'm stumped and am hoping
someone could help me figure it out and fix it.

Using strace I've found that the cause for the slowdown is that
select() is blocking for almost three seconds before notifying
that the listening socket is ready for reading.  The sequence of
events is this:

1) select for reading of listen_fd.

2) accept in a loop as many incoming connections as
possible (about 30.)

3) read the requests' data

4) form the responses.  This step uses a message queue
using pthread condition variables that worker threads are
blocking on and a pipe that the main thread is selecting
on.

5) write the response then close the socket, in a loop
for all connection sockets.

All of the above happens really quickly, and there should be
about 10 connections still hammering at the listening socket that
haven't been accepted yet.  Here's where select() decides to
delay:

6) select for reading of listen_fd.  Blocks for 3 sec!

7-10) as steps 2-5) without any blocking anywhere.

11) I kill the server.

Misc. bits of info: The listening socket is set to be
non-blocking and ndelay.  Using poll() instead of select() makes
the situation much worse.  The listen backlog is set to 1024. The
client and server are on a relatively quiet 10Mb LAN with direct
routes to each other.  I'm using Debian Linux 3.2r with kernel
2.4.25.

I don't understand why this is happening and appreciate any
insight you may have.


Best Regards,

Joonas Pihlaja

[ Post a follow-up to this message ]