|
Home > Archive > Unix Programming > June 2004 > TCP/IP question
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
|
|
| Alan Shank 2004-06-26, 10:10 am |
| I am having trouble understanding a couple of things that are happening
in the interaction between a TCP/IP client and server. Here is the
expected situation:
server client
------
socket
bind
listen
accept
socket
connect (non-blocking,
3-second T/O)
accept returns write
read
(does SQL query against busy database; read (non-blocking
20-sec timeout)
gets info for client)
write gets info back
close
close
accept (etc)
There are 25 server processes listening at well-known service port. The
client is threaded and sends alternately to each port. Client recieves
up to 50 requests/sec, but more often ~30. Normally, the 25 server
processes take about .5 sec per request, so they keep up. Sometimes,
however, the database gets "hammered" and queries take closer to 1
second, in which case a backlog of established connections builds up, to
20 or more per server process.
When that happens, the client side will see some timeouts on the
non-blocking connect, presumably because the server-side TCP/IP will not
accept any more connections for that port. The client then does a close.
QUESTION: Since the 3-way handshake is not completed, wouldn't this
connection never get to the completed queue? In this case, the server
should never be aware of it, right?
More often, however, the client will get a timeout error on the read, at
which point it does a close. This error might occur at various points
relative to the server process:
1) client times out the read and closes the socket while the connection
is on the completed queue, but has not been "accepted" by the server
process.
QUESTION: Wouldn't the server-side TCP/IP remove the connection from the
completed queue, so the server application would never be aware of it?
2) client times out the read and closes the socket between the time that
the server process calls accept and gets the connection from the
completed queue and when it does its read. In this case, I would expect
the read to get an error; however, this should be rare, because very
little time goes by between accept and read.
3) client times out the read and closes the socket after the server
process has accepted the connect and read the client's request, i.e.
while the server process is querying the database.
QUESTION: In this case, wouldn't one expect the server process to get an
error on the call to write(), sending the info back to the client?
The server processes are getting a lot of read errors after accepting a
connection. They are getting these errors when the load is very light,
starting around 4 AM. I don't understand why they are getting these
errors when there is no backlog at all.
I have a shell script that does a netstat -na, greps for the port
numbers involved and writes the results to a file every ten minutes.
When the load is light and the database access good, we generally see:
ESTABLISHED < 25 (i.e. no backlog)
CLOSE_WAIT 0
TIME_WAIT 1000-2000, depending on load
According to Stevens, TIME_WAIT lasts two MSL, which on our system is 30
seconds, so this number indicates the requests handled in the last minute.
When there is a big backup, we see more like:
ESABLISHED 500+
CLOSE_WAIT 200-775
TIME_WAIT anywhere from 0 to 1200 or so
QUESTION: How long does the CLOSE_WAIT state last?
Even though the TIME_WAIT may be 0, I still see that the server
processes are handling requests, because I see log messages showing how
long the requests have taken. This makes me think that the server
processes are doing queries on requests that have already been timed out
by the client, but I have never seen a single error on the write by the
server back to the client. ?????????????????????????????
I would appreciate any insight any can provide as to what is happening
here.
Thanks in advance,
Alan Shank
| |
| Barry Margolin 2004-06-26, 10:10 am |
| In article < 3f91d$40d75451$45234456$5014@allthenewsg
roups.com>,
Alan Shank <goatcabin@direcway.com> wrote:
> I am having trouble understanding a couple of things that are happening
> in the interaction between a TCP/IP client and server. Here is the
> expected situation:
>
> server client
> ------
> socket
> bind
> listen
> accept
> socket
> connect (non-blocking,
>
> 3-second T/O)
> accept returns write
> read
> (does SQL query against busy database; read (non-blocking
> 20-sec timeout)
> gets info for client)
> write gets info back
> close
> close
> accept (etc)
>
> There are 25 server processes listening at well-known service port. The
> client is threaded and sends alternately to each port. Client recieves
> up to 50 requests/sec, but more often ~30. Normally, the 25 server
> processes take about .5 sec per request, so they keep up. Sometimes,
> however, the database gets "hammered" and queries take closer to 1
> second, in which case a backlog of established connections builds up, to
> 20 or more per server process.
>
> When that happens, the client side will see some timeouts on the
> non-blocking connect, presumably because the server-side TCP/IP will not
> accept any more connections for that port. The client then does a close.
>
> QUESTION: Since the 3-way handshake is not completed, wouldn't this
> connection never get to the completed queue? In this case, the server
> should never be aware of it, right?
Right. Connections are never seen by the application until the 3-way
handshake completes.
> More often, however, the client will get a timeout error on the read, at
> which point it does a close. This error might occur at various points
> relative to the server process:
>
> 1) client times out the read and closes the socket while the connection
> is on the completed queue, but has not been "accepted" by the server
> process.
> QUESTION: Wouldn't the server-side TCP/IP remove the connection from the
> completed queue, so the server application would never be aware of it?
This is likely to be implementation dependent. Your is certainly a
reasonable behavior, but I don't know whether implementations actually
do it.
> 2) client times out the read and closes the socket between the time that
> the server process calls accept and gets the connection from the
> completed queue and when it does its read. In this case, I would expect
> the read to get an error; however, this should be rare, because very
> little time goes by between accept and read.
It shouldn't get an error, it should read EOF immediately.
> 3) client times out the read and closes the socket after the server
> process has accepted the connect and read the client's request, i.e.
> while the server process is querying the database.
> QUESTION: In this case, wouldn't one expect the server process to get an
> error on the call to write(), sending the info back to the client?
The first write should not get any error; when the client closes the
connection, it sends a FIN, which only tells the server that it's done
sending data, it doesn't mean that it's unwilling to receive a reply.
The error isn't detected until the server's TCP transmits the info, and
the client's TCP responds with a RST. Since write() simply queues the
data and doesn't wait for it to be transmitted and acknowledged, it
can't report the error right away. When the RST is received, a SIGPIPE
signal will be sent; if you're ignoring this signal, the socket will be
tagged with an error condition, and the *next* call will result in an
error (ECONNRESET).
>
> The server processes are getting a lot of read errors after accepting a
> connection. They are getting these errors when the load is very light,
> starting around 4 AM. I don't understand why they are getting these
> errors when there is no backlog at all.
Maybe you're being hit by port scanners.
>
> I have a shell script that does a netstat -na, greps for the port
> numbers involved and writes the results to a file every ten minutes.
> When the load is light and the database access good, we generally see:
>
> ESTABLISHED < 25 (i.e. no backlog)
> CLOSE_WAIT 0
> TIME_WAIT 1000-2000, depending on load
>
> According to Stevens, TIME_WAIT lasts two MSL, which on our system is 30
> seconds, so this number indicates the requests handled in the last minute.
>
> When there is a big backup, we see more like:
>
> ESABLISHED 500+
> CLOSE_WAIT 200-775
> TIME_WAIT anywhere from 0 to 1200 or so
>
> QUESTION: How long does the CLOSE_WAIT state last?
There's no timeout on CLOSE_WAIT, and no reason why there should be.
Perhaps the server starts doing its calculations after receiving EOF
from the client, and the calculation takes a day to finish; this would
all be wasted effort if the system closed the connection before it could
send the answer.
>
> Even though the TIME_WAIT may be 0, I still see that the server
> processes are handling requests, because I see log messages showing how
> long the requests have taken. This makes me think that the server
> processes are doing queries on requests that have already been timed out
> by the client, but I have never seen a single error on the write by the
> server back to the client. ?????????????????????????????
If you only do a single write(), you'll never see the error. As I said
above, the error happens on the next write() that occurs *after* the
client has responded to the first one.
--
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
| |
| Pete Brett 2004-06-26, 10:11 am |
| Alan Shank <goatcabin@direcway.com> wrote:
<snip>
Since you do not include any code in your description, it is impossible to be
sure what is causing this.
>QUESTION: Since the 3-way handshake is not completed, wouldn't this
>connection never get to the completed queue? In this case, the server
>should never be aware of it, right?
You describe several iterative servers. If a server is processing a previous
request when a syn packet arrives the OS should send a syn ack and once the
client's ack is received, queue the connection for the next time the server
calls accept(). Your call to listen specifies the backlog for this queue.
>
>More often, however, the client will get a timeout error on the read, at
>which point it does a close. This error might occur at various points
>relative to the server process:
After the client calls connect(), it calls write(). Assuming the amount of data
written is small, this call will return almost immediately. The data written is
usually buffered so that the stack can wait in case any other data is made
available soon which could be sent in the same packet... google for "Nagle
algorithm".
>
>1) client times out the read and closes the socket while the connection
>is on the completed queue, but has not been "accepted" by the server
>process.
>QUESTION: Wouldn't the server-side TCP/IP remove the connection from the
>completed queue, so the server application would never be aware of it?
At the least, there is a race condition here. What if the client closes the
connection, but then the server calls accept() before the fin / rst packet can
be sent from the client to the server.
How are you closing the connection? If you are using shutdown(), it is possible
to close the connection in one direction only. If you use close() then the
implication is "When you've sent all that stuff I asked you to send, send a fin
packet to indicate I'm finished". Perhaps you are trying not to close the
connection using fin, but actually you want to reset the connection? To do this,
call setsockopt() with optname = SO_LINGER and optval set to "onoff = 1" and
"linger=0", before you call close().
What you don't seem to be taking account of is the buffering that the OS will
do. The fact that write() returns Ok indicates that data was successfully
written to the stack... not to the remote host. Are you checking the return
value of close() or shutdown()?
>The server processes are getting a lot of read errors after accepting a
>connection. They are getting these errors when the load is very light,
>starting around 4 AM. I don't understand why they are getting these
>errors when there is no backlog at all.
On many systems, daily and weekly cron jobs are started at just after 4 AM.
Again, I suspect that due to extra system load, your servers are not calling
accept() until after the clients have timed out on read.
Pete
| |
| Alan Shank 2004-06-26, 10:11 am |
| First, thank you for your reply.
Barry Margolin wrote:
> In article < 3f91d$40d75451$45234456$5014@allthenewsg
roups.com>,
> Alan Shank <goatcabin@direcway.com> wrote:
>
>
> It shouldn't get an error, it should read EOF immediately.
OK, that is not happening at all, so this scenario is apparently not
happening.
>
>
>
>
> The first write should not get any error; when the client closes the
> connection, it sends a FIN, which only tells the server that it's done
> sending data, it doesn't mean that it's unwilling to receive a reply.
> The error isn't detected until the server's TCP transmits the info, and
> the client's TCP responds with a RST. Since write() simply queues the
> data and doesn't wait for it to be transmitted and acknowledged, it
> can't report the error right away. When the RST is received, a SIGPIPE
> signal will be sent; if you're ignoring this signal, the socket will be
> tagged with an error condition, and the *next* call will result in an
> error (ECONNRESET).
There is only one write, then the server process does a close. Neither
side is using "shutdown." All client threads do just one request. The
server side close never gets an error. The server processes are not
getting SIGPIPE, either; I know this because our common library emits a
log message for all signals received. ????????/
>
>
>
>
> There's no timeout on CLOSE_WAIT, and no reason why there should be.
> Perhaps the server starts doing its calculations after receiving EOF
> from the client, and the calculation takes a day to finish; this would
> all be wasted effort if the system closed the connection before it could
> send the answer.
No, I just meant how long the CLOSE_WAIT state normally lasts before
LAST_ACK and then nothing, in order to compare with the TIME_WAIT number.
>
>
>
>
> If you only do a single write(), you'll never see the error. As I said
> above, the error happens on the next write() that occurs *after* the
> client has responded to the first one.
>
Yes, this must be what is happening, but I am also not getting SIGPIPE.
When I get this read error after accept, the errno is 232, "Connection
reset by peer." Does that literally mean an RST was received?
Thanks for your time; it's aprreciated,
Alan Shank
| |
| Barry Margolin 2004-06-26, 10:11 am |
| In article < 5f42a$40d864a5$45234a33$12463@allthenews
groups.com>,
Alan Shank <goatcabin@direcway.com> wrote:
>
> No, I just meant how long the CLOSE_WAIT state normally lasts before
> LAST_ACK and then nothing, in order to compare with the TIME_WAIT number.
There's no typical length of time for this, it really depends on the
application. A socket on the server goes into CLOSE_WAIT when the
client closes its socket. It will stay in this state until the server
closes its socket. This depends on how quickly the server notices that
the client has closed, and how long it takes to do whatever it does in
response to noticing that.
> Yes, this must be what is happening, but I am also not getting SIGPIPE.
If the server just does a single write() followed immediately by
close(), the RST may not arrive until after the socket has been closed.
Since no process owns the socket any more, there's nowhere to send the
signal, so no error is produced.
> When I get this read error after accept, the errno is 232, "Connection
> reset by peer." Does that literally mean an RST was received?
Yes.
--
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
| |
| Alan Shank 2004-06-26, 10:11 am |
| Once again, thanks very much for your time and advice.
Barry Margolin wrote:
> In article < 5f42a$40d864a5$45234a33$12463@allthenews
groups.com>,
> Alan Shank <goatcabin@direcway.com> wrote:
>
>
> If the server just does a single write() followed immediately by
> close(), the RST may not arrive until after the socket has been closed.
> Since no process owns the socket any more, there's nowhere to send the
> signal, so no error is produced.
This scenario fits the symptoms.
I have asked the guy who wrote the client code to log all the errors on
his end, so we can try to match them up, but it looks like the errors on
his end will not correspond to errors on the server side. The remaining
mystery is the read errors on the server side. We are going to try to
build tcpdump so we can see the packets.
Thanks again,
Alan Shank
|
|
|
|
|