Unix Programming - Odd TCP client/server throughput problem

This is Interesting: Free IT Magazines  
Home > Archive > Unix Programming > November 2006 > Odd TCP client/server throughput problem





You are viewing an archived Text-only version of the thread. To view this thread in it's original format and/or if you want to reply to this thread please [click here]

Author Odd TCP client/server throughput problem
Richard Eich

2006-11-24, 1:22 pm

Client and Server OS: SuSE 9.3 Pro 2.6.11 default kernel, neither
machine patched after install from CDs.

Communicating over: GigE LAN

1) Server (receiver) is consistently adversting a TCP RWIN of 32K.
2) Server consistently has TCP RECV-Q of 0.
3) Client (sender) consistently shows a TCP SEND-Q of 80K.
4) Socket is up and connection is ESTABLISHED from both sides.
5) No data is transmitted.

To troubleshoot, I've torn down and re-established to connection
countless times. There may be a trickle of data initially, but
within a few seconds the client SEND-Q builds and transmission stops.
Receiver's window size never goes below 32K.

Never seen this kind of behavior before. If the server process was
slow, I'd expect to see a RECV-Q buildup to go with the big SEND-Q.
Maxim Yegorushkin

2006-11-24, 7:19 pm


Richard Eich wrote:
> Client and Server OS: SuSE 9.3 Pro 2.6.11 default kernel, neither
> machine patched after install from CDs.
>
> Communicating over: GigE LAN
>
> 1) Server (receiver) is consistently adversting a TCP RWIN of 32K.
> 2) Server consistently has TCP RECV-Q of 0.
> 3) Client (sender) consistently shows a TCP SEND-Q of 80K.
> 4) Socket is up and connection is ESTABLISHED from both sides.
> 5) No data is transmitted.
>
> To troubleshoot, I've torn down and re-established to connection
> countless times. There may be a trickle of data initially, but
> within a few seconds the client SEND-Q builds and transmission stops.
> Receiver's window size never goes below 32K.
>
> Never seen this kind of behavior before. If the server process was
> slow, I'd expect to see a RECV-Q buildup to go with the big SEND-Q.

I would use tcpdump or wireshark/ethereal to see what is being sent and
received.

Richard Eich

2006-11-24, 7:19 pm

maxim.yegorushkin@gmail.com wrote...
>
> Richard Eich wrote:
>
> I would use tcpdump or wireshark/ethereal to see what is being sent and
> received.

How do you think I know the TCP RWIN size for the server?
Maxim Yegorushkin

2006-11-24, 7:19 pm


Richard Eich wrote:
> maxim.yegorushkin@gmail.com wrote...
>
> How do you think I know the TCP RWIN size for the server?

So, what does tcpdump shows you before the transfer stops?

Maxim Yegorushkin

2006-11-24, 7:19 pm


Richard Eich wrote:
> maxim.yegorushkin@gmail.com wrote...
>
> How do you think I know the TCP RWIN size for the server?

Is this the same problem?
http://groups.google.com/group/comp...555a65936967a36

Simple Simon

2006-11-24, 7:19 pm

maxim.yegorushkin@gmail.com wrote...
>
> Richard Eich wrote:
>
> So, what does tcpdump shows you before the transfer stops?

SYN =>
SYN ACK <=
SYN ACK =>

then either nothing, or

Packet =>
ACK <= RWIN 32K // immediate reply
Packet =>
ACK <= RWIN 32K // immediate reply
....
nothing. No shrinking window, no slowdown in the rate of ACKs.

--
Taxes are not "punishment for success". Nor are they "theft". Taxes
are a royalty paid commensurate to the economic benefit obtained from
a shared socio-economic system.

"Those who gain the benefit should also bear the disadvantage."
- Common Law maxim
Richard Eich

2006-11-24, 7:19 pm

maxim.yegorushkin@gmail.com wrote...
>
> Richard Eich wrote:
>
> Is this the same problem?
> http://groups.google.com/group/comp...555a65936967a36

Yes, except that we now have a minimal case (described above) that
exhibits the behavior without what we have learned since apparently
are red herrings.

Maxim Yegorushkin

2006-11-24, 7:19 pm


Simple Simon wrote:
> maxim.yegorushkin@gmail.com wrote...
>
> SYN =>
> SYN ACK <=
> SYN ACK =>
>
> then either nothing, or
>
> Packet =>
> ACK <= RWIN 32K // immediate reply
> Packet =>
> ACK <= RWIN 32K // immediate reply
> ...
> nothing. No shrinking window, no slowdown in the rate of ACKs.

tcpdump output might provide more details.

strace'ing the sender while reproducing the problem may give you a
clue. You may like to find out what the sending thread is doing or
where it is blocked when the data transfer stops.

Richard Eich

2006-11-25, 7:22 pm

maxim.yegorushkin@gmail.com wrote...
>
> Richard Eich wrote:
>
> So, what does tcpdump shows you before the transfer stops?

Looks like we're getting a lot of retransmissions, which I believe
would clearly explain the full sender tcp send-q, the receiver empty
tcp recv-q, and the receiver's normal receive window size.

(Relative ACKs used for clarity).

|Time | 192.168.75.100 | 192.168.76.15 |
|0.000 | PSH, ACK - Len: 4 |Seq = 0 Ack = 0
| |(43349) ------------------> (9550) |
|0.000 | PSH, ACK - Len: 1448 |Seq = 4 Ack = 0
| |(43349) ------------------> (9550) |
|0.000 | ACK | |Seq = 0 Ack = 4
| |(43349) <------------------ (9550) |
|0.000 | ACK - Len: 1448 |Seq = 1452 Ack = 0
| |(43349) ------------------> (9550) |
|0.000 | ACK - Len: 1448 |Seq = 2900 Ack = 0
| |(43349) ------------------> (9550) |
|0.001 | ACK | |Seq = 0 Ack = 1452
| |(43349) <------------------ (9550) |
|0.001 | ACK - Len: 1448 |Seq = 4348 Ack = 0
| |(43349) ------------------> (9550) |
|0.002 | ACK | |Seq = 0 Ack = 2900
| |(43349) <------------------ (9550) |
|0.002 | PSH, ACK - Len: 1448 |Seq = 5796 Ack = 0
| |(43349) ------------------> (9550) |
|0.002 | ACK | |Seq = 0 Ack = 4348
| |(43349) <------------------ (9550) |
|0.002 | ACK - Len: 1448 |Seq = 7244 Ack = 0
| |(43349) ------------------> (9550) |
|0.002 | ACK | |Seq = 0 Ack = 5796
| |(43349) <------------------ (9550) |
|0.003 | ACK - Len: 1448 |Seq = 8692 Ack = 0
| |(43349) ------------------> (9550) |
|0.003 | ACK - Len: 1448 |Seq = 10140 Ack =
0
| |(43349) ------------------> (9550) |
|0.003 | ACK | |Seq = 0 Ack = 7244
| |(43349) <------------------ (9550) |
|0.003 | ACK - Len: 1448 |Seq = 11588 Ack =
0
| |(43349) ------------------> (9550) |
|0.003 | ACK | |Seq = 0 Ack = 8692
| |(43349) <------------------ (9550) |
|0.003 | ACK - Len: 1448 |Seq = 13036 Ack =
0
| |(43349) ------------------> (9550) |
|0.004 | ACK | |Seq = 0 Ack =
10140
| |(43349) <------------------ (9550) |
|0.004 | PSH, ACK - Len: 1448 |Seq = 14484 Ack =
0
| |(43349) ------------------> (9550) |
|0.004 | ACK | |Seq = 0 Ack =
10140
| |(43349) <------------------ (9550) |
|0.004 | ACK - Len: 1448 |Seq = 15932 Ack =
0
| |(43349) ------------------> (9550) |
|0.004 | ACK | |Seq = 0 Ack =
10140
| |(43349) <------------------ (9550) |
|0.004 | ACK - Len: 1448 |Seq = 17380 Ack =
0
| |(43349) ------------------> (9550) |
|0.005 | ACK | |Seq = 0 Ack =
10140
| |(43349) <------------------ (9550) |
|0.005 | ACK - Len: 1448 |Seq = 10140 Ack =
0
| |(43349) ------------------> (9550) |
|0.005 | ACK | |Seq = 0 Ack =
10140
| |(43349) <------------------ (9550) |
|0.006 | ACK | |Seq = 0 Ack =
14484
| |(43349) <------------------ (9550) |
|0.006 | ACK - Len: 1448 |Seq = 18828 Ack =
0
| |(43349) ------------------> (9550) |
|0.007 | ACK | |Seq = 0 Ack =
14484
| |(43349) <------------------ (9550) |
|0.007 | PSH, ACK - Len: 1448 |Seq = 14484 Ack =
0
| |(43349) ------------------> (9550) |
|0.008 | ACK | |Seq = 0 Ack =
20276
| |(43349) <------------------ (9550) |
|0.008 | ACK - Len: 1448 |Seq = 20276 Ack =
0
| |(43349) ------------------> (9550) |
|0.010 | ACK | |Seq = 0 Ack =
21724
| |(43349) <------------------ (9550) |
|0.010 | ACK - Len: 1448 |Seq = 21724 Ack =
0
| |(43349) ------------------> (9550) |
|0.010 | ACK - Len: 1448 |Seq = 23172 Ack =
0
| |(43349) ------------------> (9550) |
|0.011 | ACK | |Seq = 0 Ack =
23172
| |(43349) <------------------ (9550) |
|0.011 | ACK - Len: 1448 |Seq = 24620 Ack =
0
| |(43349) ------------------> (9550) |
|0.011 | ACK - Len: 1448 |Seq = 26068 Ack =
0
| |(43349) ------------------> (9550) |
|0.011 | ACK | |Seq = 0 Ack =
24620
| |(43349) <------------------ (9550) |
|0.011 | ACK - Len: 1448 |Seq = 27516 Ack =
0
| |(43349) ------------------> (9550) |
|0.012 | ACK | |Seq = 0 Ack =
24620
| |(43349) <------------------ (9550) |
|0.012 | ACK - Len: 1448 |Seq = 28964 Ack =
0
| |(43349) ------------------> (9550) |
|0.012 | ACK | |Seq = 0 Ack =
24620
| |(43349) <------------------ (9550) |
|0.012 | ACK - Len: 1448 |Seq = 30412 Ack =
0
| |(43349) ------------------> (9550) |
|0.013 | ACK | |Seq = 0 Ack =
24620
| |(43349) <------------------ (9550) |
|0.013 | ACK - Len: 1448 |Seq = 24620 Ack =
0
| |(43349) ------------------> (9550) |
|0.014 | ACK | |Seq = 0 Ack =
28964
| |(43349) <------------------ (9550) |
|0.216 | ACK - Len: 1448 |Seq = 28964 Ack =
0
| |(43349) ------------------> (9550) |
|0.218 | ACK | |Seq = 0 Ack =
31860
| |(43349) <------------------ (9550) |
|0.218 | ACK - Len: 1448 |Seq = 31860 Ack =
0
| |(43349) ------------------> (9550) |
|0.218 | ACK - Len: 1448 |Seq = 33308 Ack =
0
| |(43349) ------------------> (9550) |
|0.219 | ACK | |Seq = 0 Ack =
33308
| |(43349) <------------------ (9550) |
|0.219 | ACK - Len: 1448 |Seq = 34756 Ack =
0
| |(43349) ------------------> (9550) |
|0.219 | ACK - Len: 1448 |Seq = 36204 Ack =
0
| |(43349) ------------------> (9550) |
|0.219 | ACK | |Seq = 0 Ack =
34756
| |(43349) <------------------ (9550) |
|0.219 | ACK - Len: 1448 |Seq = 37652 Ack =
0
| |(43349) ------------------> (9550) |
|0.220 | ACK | |Seq = 0 Ack =
34756
| |(43349) <------------------ (9550) |
|0.220 | ACK - Len: 1448 |Seq = 39100 Ack =
0
| |(43349) ------------------> (9550) |
|0.220 | ACK | |Seq = 0 Ack =
34756
| |(43349) <------------------ (9550) |
|0.220 | ACK - Len: 1448 |Seq = 40548 Ack =
0
| |(43349) ------------------> (9550) |
|0.221 | ACK | |Seq = 0 Ack =
34756
| |(43349) <------------------ (9550) |
|0.221 | ACK - Len: 1448 |Seq = 34756 Ack =
0
| |(43349) ------------------> (9550) |
|0.222 | ACK | |Seq = 0 Ack =
39100
| |(43349) <------------------ (9550) |
|0.424 | ACK - Len: 1448 |Seq = 39100 Ack =
0
| |(43349) ------------------> (9550) |
|0.426 | ACK | |Seq = 0 Ack =
41996
| |(43349) <------------------ (9550) |
|0.426 | ACK - Len: 1448 |Seq = 41996 Ack =
0
| |(43349) ------------------> (9550) |
|0.426 | ACK - Len: 1448 |Seq = 43444 Ack =
0
| |(43349) ------------------> (9550) |
|0.427 | ACK | |Seq = 0 Ack =
43444
| |(43349) <------------------ (9550) |
|0.427 | ACK - Len: 1448 |Seq = 44892 Ack =
0
| |(43349) ------------------> (9550) |
|0.427 | ACK - Len: 1448 |Seq = 46340 Ack =
0
| |(43349) ------------------> (9550) |
|0.427 | ACK | |Seq = 0 Ack =
44892
| |(43349) <------------------ (9550) |
|0.427 | ACK - Len: 1448 |Seq = 47788 Ack =
0
| |(43349) ------------------> (9550) |
|0.428 | ACK | |Seq = 0 Ack =
44892
| |(43349) <------------------ (9550) |
|0.428 | PSH, ACK - Len: 1448 |Seq = 49236 Ack =
0
| |(43349) ------------------> (9550) |
|0.428 | ACK | |Seq = 0 Ack =
44892
| |(43349) <------------------ (9550) |
|0.428 | ACK - Len: 1448 |Seq = 50684 Ack =
0
| |(43349) ------------------> (9550) |
|0.429 | ACK | |Seq = 0 Ack =
44892
| |(43349) <------------------ (9550) |
|0.429 | ACK - Len: 1448 |Seq = 44892 Ack =
0
| |(43349) ------------------> (9550) |
|0.430 | ACK | |Seq = 0 Ack =
49236
| |(43349) <------------------ (9550) |
|0.632 | PSH, ACK - Len: 1448 |Seq = 49236 Ack =
0
| |(43349) ------------------> (9550) |
|0.633 | ACK | |Seq = 0 Ack =
52132
| |(43349) <------------------ (9550) |
|0.634 | ACK - Len: 1448 |Seq = 52132 Ack =
0
| |(43349) ------------------> (9550) |
|0.634 | ACK - Len: 1448 |Seq = 53580 Ack =
0
| |(43349) ------------------> (9550) |
|0.635 | ACK | |Seq = 0 Ack =
53580
| |(43349) <------------------ (9550) |
|0.635 | ACK - Len: 1448 |Seq = 55028 Ack =
0
| |(43349) ------------------> (9550) |
|0.635 | ACK - Len: 1448 |Seq = 56476 Ack =
0
| |(43349) ------------------> (9550) |
|0.635 | ACK | |Seq = 0 Ack =
55028
| |(43349) <------------------ (9550) |
|0.635 | ACK - Len: 1448 |Seq = 57924 Ack =
0
| |(43349) ------------------> (9550) |
|0.636 | ACK | |Seq = 0 Ack =
55028
| |(43349) <------------------ (9550) |
|0.636 | ACK - Len: 1448 |Seq = 59372 Ack =
0
| |(43349) ------------------> (9550) |
|0.636 | ACK | |Seq = 0 Ack =
55028
| |(43349) <------------------ (9550) |
|0.636 | ACK - Len: 1448 |Seq = 60820 Ack =
0
| |(43349) ------------------> (9550) |
|0.637 | ACK | |Seq = 0 Ack =
55028
| |(43349) <------------------ (9550) |
|0.637 | ACK - Len: 1448 |Seq = 55028 Ack =
0
| |(43349) ------------------> (9550) |
|0.638 | ACK | |Seq = 0 Ack =
59372
| |(43349) <------------------ (9550) |
|0.840 | ACK - Len: 1448 |Seq = 59372 Ack =
0
| |(43349) ------------------> (9550) |
|0.841 | ACK | |Seq = 0 Ack =
62268
| |(43349) <------------------ (9550) |
|0.841 | ACK - Len: 1448 |Seq = 62268 Ack =
0
| |(43349) ------------------> (9550) |
|0.841 | ACK - Len: 1448 |Seq = 63716 Ack =
0
| |(43349) ------------------> (9550) |
|0.843 | ACK | |Seq = 0 Ack =
63716
| |(43349) <------------------ (9550) |
|0.843 | ACK - Len: 1448 |Seq = 65164 Ack =
0
| |(43349) ------------------> (9550) |
|0.843 | ACK - Len: 1448 |Seq = 66612 Ack =
0
| |(43349) ------------------> (9550) |
|0.843 | ACK | |Seq = 0 Ack =
65164
| |(43349) <------------------ (9550) |
|0.843 | ACK - Len: 1448 |Seq = 68060 Ack =
0
| |(43349) ------------------> (9550) |
|0.844 | ACK | |Seq = 0 Ack =
65164
| |(43349) <------------------ (9550) |
|0.844 | ACK - Len: 1448 |Seq = 69508 Ack =
0
| |(43349) ------------------> (9550) |
|0.844 | ACK | |Seq = 0 Ack =
65164
| |(43349) <------------------ (9550) |
|0.844 | ACK - Len: 1448 |Seq = 70956 Ack =
0
| |(43349) ------------------> (9550) |
|0.845 | ACK | |Seq = 0 Ack =
65164
| |(43349) <------------------ (9550) |
|0.845 | ACK - Len: 1448 |Seq = 65164 Ack =
0
| |(43349) ------------------> (9550) |
|0.847 | ACK | |Seq = 0 Ack =
69508
| |(43349) <------------------ (9550) |
|1.047 | ACK - Len: 1448 |Seq = 69508 Ack =
0
| |(43349) ------------------> (9550) |
|1.048 | ACK | |Seq = 0 Ack =
72404
| |(43349) <------------------ (9550) |
|1.048 | ACK - Len: 1448 |Seq = 72404 Ack =
0
| |(43349) ------------------> (9550) |
|1.048 | ACK - Len: 1448 |Seq = 73852 Ack =
0
| |(43349) ------------------> (9550) |
|1.050 | ACK | |Seq = 0 Ack =
73852
| |(43349) <------------------ (9550) |
|1.050 | ACK - Len: 1448 |Seq = 75300 Ack =
0
| |(43349) ------------------> (9550) |
|1.050 | ACK - Len: 1448 |Seq = 76748 Ack =
0
| |(43349) ------------------> (9550) |
|1.050 | ACK | |Seq = 0 Ack =
75300
| |(43349) <------------------ (9550) |
|1.050 | ACK - Len: 1448 |Seq = 78196 Ack =
0
| |(43349) ------------------> (9550) |
|1.051 | ACK | |Seq = 0 Ack =
75300
| |(43349) <------------------ (9550) |
|1.051 | ACK - Len: 1448 |Seq = 79644 Ack =
0
| |(43349) ------------------> (9550) |
|1.051 | ACK | |Seq = 0 Ack =
75300
| |(43349) <------------------ (9550) |
|1.051 | ACK - Len: 1448 |Seq = 81092 Ack =
0
| |(43349) ------------------> (9550) |
|1.052 | ACK | |Seq = 0 Ack =
75300
| |(43349) <------------------ (9550) |
|1.052 | ACK - Len: 1448 |Seq = 75300 Ack =
0
| |(43349) ------------------> (9550) |
|1.053 | ACK | |Seq = 0 Ack =
79644
| |(43349) <------------------ (9550) |
|1.255 | ACK - Len: 1448 |Seq = 79644 Ack =
0
| |(43349) ------------------> (9550) |
|1.256 | ACK | |Seq = 0 Ack =
82540
| |(43349) <------------------ (9550) |
|1.256 | ACK - Len: 1448 |Seq = 82540 Ack =
0
| |(43349) ------------------> (9550) |
|1.256 | ACK - Len: 1448 |Seq = 83988 Ack =
0
| |(43349) ------------------> (9550) |
|1.258 | ACK | |Seq = 0 Ack =
83988
| |(43349) <------------------ (9550) |
|1.258 | ACK - Len: 1448 |Seq = 85436 Ack =
0
| |(43349) ------------------> (9550) |
|1.258 | ACK - Len: 1448 |Seq = 86884 Ack =
0
| |(43349) ------------------> (9550) |
|1.258 | ACK | |Seq = 0 Ack =
85436
| |(43349) <------------------ (9550) |
|1.258 | ACK - Len: 1448 |Seq = 88332 Ack =
0
| |(43349) ------------------> (9550) |
|1.259 | ACK | |Seq = 0 Ack =
85436
| |(43349) <------------------ (9550) |
|1.259 | ACK - Len: 1448 |Seq = 89780 Ack =
0
| |(43349) ------------------> (9550) |
|1.259 | ACK | |Seq = 0 Ack =
85436
| |(43349) <------------------ (9550) |
|1.259 | ACK - Len: 1448 |Seq = 91228 Ack =
0
| |(43349) ------------------> (9550) |
|1.260 | ACK | |Seq = 0 Ack =
85436
| |(43349) <------------------ (9550) |
|1.260 | ACK - Len: 1448 |Seq = 85436 Ack =
0
| |(43349) ------------------> (9550) |
|1.261 | ACK | |Seq = 0 Ack =
89780
| |(43349) <------------------ (9550) |
|1.463 | ACK - Len: 1448 |Seq = 89780 Ack =
0
| |(43349) ------------------> (9550) |
|1.464 | ACK | |Seq = 0 Ack =
92676
| |(43349) <------------------ (9550) |
|1.464 | PSH, ACK - Len: 1448 |Seq = 92676 Ack =
0
| |(43349) ------------------> (9550) |
|1.464 | ACK - Len: 1448 |Seq = 94124 Ack =
0
| |(43349) ------------------> (9550) |
|1.466 | ACK | |Seq = 0 Ack =
94124
| |(43349) <------------------ (9550) |
|1.466 | ACK - Len: 1448 |Seq = 95572 Ack =
0
| |(43349) ------------------> (9550) |
|1.466 | ACK - Len: 1448 |Seq = 97020 Ack =
0
| |(43349) ------------------> (9550) |
|1.466 | ACK | |Seq = 0 Ack =
95572
| |(43349) <------------------ (9550) |
|1.466 | ACK - Len: 1448 |Seq = 98468 Ack =
0
| |(43349) ------------------> (9550) |
|1.467 | ACK | |Seq = 0 Ack =
95572
| |(43349) <------------------ (9550) |
|1.467 | ACK - Len: 1448 |Seq = 99916 Ack =
0
| |(43349) ------------------> (9550) |
|1.467 | ACK | |Seq = 0 Ack =
95572
| |(43349) <------------------ (9550) |
|1.467 | ACK - Len: 1448 |Seq = 101364 Ack =
0
| |(43349) ------------------> (9550) |
|1.468 | ACK | |Seq = 0 Ack =
95572
| |(43349) <------------------ (9550) |
|1.468 | ACK - Len: 1448 |Seq = 95572 Ack =
0
| |(43349) ------------------> (9550) |
|1.469 | ACK | |Seq = 0 Ack =
99916
| |(43349) <------------------ (9550) |
|1.670 | ACK - Len: 1448 |Seq = 99916 Ack =
0
| |(43349) ------------------> (9550) |
|1.671 | ACK | |Seq = 0 Ack =
102812
| |(43349) <------------------ (9550) |
|1.671 | ACK - Len: 1448 |Seq = 102812 Ack =
0
| |(43349) ------------------> (9550) |
|1.671 | ACK - Len: 1448 |Seq = 104260 Ack =
0
| |(43349) ------------------> (9550) |
|1.672 | ACK | |Seq = 0 Ack =
104260
| |(43349) <------------------ (9550) |
|1.672 | ACK - Len: 1448 |Seq = 105708 Ack =
0
| |(43349) ------------------> (9550) |
|1.673 | ACK - Len: 1448 |Seq = 107156 Ack =
0
| |(43349) ------------------> (9550) |
|1.673 | ACK | |Seq = 0 Ack =
105708
| |(43349) <------------------ (9550) |
|1.673 | ACK - Len: 1448 |Seq = 108604 Ack =
0
| |(43349) ------------------> (9550) |
|1.674 | ACK | |Seq = 0 Ack =
105708
| |(43349) <------------------ (9550) |
|1.674 | ACK - Len: 1448 |Seq = 110052 Ack =
0
| |(43349) ------------------> (9550) |
|1.674 | ACK | |Seq = 0 Ack =
105708
| |(43349) <------------------ (9550) |
|1.674 | PSH, ACK - Len: 1448 |Seq = 111500 Ack =
0
| |(43349) ------------------> (9550) |
|1.675 | ACK | |Seq = 0 Ack =
105708
| |(43349) <------------------ (9550) |
|1.675 | ACK - Len: 1448 |Seq = 105708 Ack =
0
| |(43349) ------------------> (9550) |
|1.676 | ACK | |Seq = 0 Ack =
110052
| |(43349) <------------------ (9550) |
|1.878 | ACK - Len: 1448 |Seq = 110052 Ack =
0
| |(43349) ------------------> (9550) |
|1.879 | ACK | |Seq = 0 Ack =
112948
| |(43349) <------------------ (9550) |
|1.879 | ACK - Len: 1448 |Seq = 112948 Ack =
0
| |(43349) ------------------> (9550) |
|1.879 | ACK - Len: 1448 |Seq = 114396 Ack =
0
| |(43349) ------------------> (9550) |
|1.880 | ACK | |Seq = 0 Ack =
114396
| |(43349) <------------------ (9550) |
|1.881 | ACK - Len: 1448 |Seq = 115844 Ack =
0
| |(43349) ------------------> (9550) |
|1.881 | ACK - Len: 1448 |Seq = 117292 Ack =
0
| |(43349) ------------------> (9550) |
|1.881 | ACK | |Seq = 0 Ack =
115844
| |(43349) <------------------ (9550) |
|1.881 | ACK - Len: 1448 |Seq = 118740 Ack =
0
| |(43349) ------------------> (9550) |
|1.882 | ACK | |Seq = 0 Ack =
115844
| |(43349) <------------------ (9550) |
|1.882 | ACK - Len: 1448 |Seq = 120188 Ack =
0
| |(43349) ------------------> (9550) |
|1.882 | ACK | |Seq = 0 Ack =
115844
| |(43349) <------------------ (9550) |
|1.882 | PSH, ACK - Len: 1448 |Seq = 121636 Ack =
0
| |(43349) ------------------> (9550) |
|1.883 | ACK | |Seq = 0 Ack =
115844
| |(43349) <------------------ (9550) |
|1.883 | ACK - Len: 1448 |Seq = 115844 Ack =
0
| |(43349) ------------------> (9550) |
|1.884 | ACK | |Seq = 0 Ack =
120188
| |(43349) <------------------ (9550) |
|2.086 | ACK - Len: 1448 |Seq = 120188 Ack =
0
| |(43349) ------------------> (9550) |
|2.087 | ACK | |Seq = 0 Ack =
123084
| |(43349) <------------------ (9550) |
|2.087 | ACK - Len: 1448 |Seq = 123084 Ack =
0
| |(43349) ------------------> (9550) |
|2.087 | ACK - Len: 1448 |Seq = 124532 Ack =
0
| |(43349) ------------------> (9550) |
|2.088 | ACK | |Seq = 0 Ack =
124532
| |(43349) <------------------ (9550) |
|2.088 | ACK - Len: 1448 |Seq = 125980 Ack =
0
| |(43349) ------------------> (9550) |
|2.088 | ACK - Len: 1448 |Seq = 127428 Ack =
0
| |(43349) ------------------> (9550) |
|2.089 | ACK | |Seq = 0 Ack =
125980
| |(43349) <------------------ (9550) |
|2.089 | ACK - Len: 1448 |Seq = 128876 Ack =
0
| |(43349) ------------------> (9550) |
|2.090 | ACK | |Seq = 0 Ack =
125980
| |(43349) <------------------ (9550) |
|2.090 | ACK - Len: 1448 |Seq = 130324 Ack =
0
| |(43349) ------------------> (9550) |
|2.090 | ACK | |Seq = 0 Ack =
125980
| |(43349) <------------------ (9550) |
|2.090 | ACK - Len: 1448 |Seq = 131772 Ack =
0
| |(43349) ------------------> (9550) |
|2.091 | ACK | |Seq = 0 Ack =
125980
| |(43349) <------------------ (9550) |
|2.091 | ACK - Len: 1448 |Seq = 125980 Ack =
0
| |(43349) ------------------> (9550) |
|2.092 | ACK | |Seq = 0 Ack =
130324
| |(43349) <------------------ (9550) |
|2.294 | ACK - Len: 1448 |Seq = 130324 Ack =
0
| |(43349) ------------------> (9550) |
|2.295 | ACK | |Seq = 0 Ack =
133220
| |(43349) <------------------ (9550) |
|2.295 | ACK - Len: 1448 |Seq = 133220 Ack =
0
| |(43349) ------------------> (9550) |
|2.295 | ACK - Len: 1448 |Seq = 134668 Ack =
0
| |(43349) ------------------> (9550) |
|2.297 | ACK | |Seq = 0 Ack =
134668
| |(43349) <------------------ (9550) |
|2.297 | ACK - Len: 1448 |Seq = 136116 Ack =
0
| |(43349) ------------------> (9550) |
|2.297 | ACK - Len: 1448 |Seq = 137564 Ack =
0
| |(43349) ------------------> (9550) |
|2.297 | ACK | |Seq = 0 Ack =
136116
| |(43349) <------------------ (9550) |
|2.297 | ACK - Len: 1448 |Seq = 139012 Ack =
0
| |(43349) ------------------> (9550) |
|2.298 | ACK | |Seq = 0 Ack =
136116
| |(43349) <------------------ (9550) |
|2.298 | ACK - Len: 1448 |Seq = 140460 Ack =
0
| |(43349) ------------------> (9550) |
|2.298 | ACK | |Seq = 0 Ack =
136116
| |(43349) <------------------ (9550) |
|2.298 | ACK - Len: 1448 |Seq = 141908 Ack =
0
| |(43349) ------------------> (9550) |
|2.299 | ACK | |Seq = 0 Ack =
136116
| |(43349) <------------------ (9550) |
|2.299 | ACK - Len: 1448 |Seq = 136116 Ack =
0
| |(43349) ------------------> (9550) |
|2.300 | ACK | |Seq = 0 Ack =
140460
| |(43349) <------------------ (9550) |
|2.501 | ACK - Len: 1448 |Seq = 140460 Ack =
0
| |(43349) ------------------> (9550) |
|2.502 | ACK | |Seq = 0 Ack =
143356
| |(43349) <------------------ (9550) |
|2.502 | ACK - Len: 1448 |Seq = 143356 Ack =
0
| |(43349) ------------------> (9550) |
|2.502 | ACK - Len: 1448 |Seq = 144804 Ack =
0
| |(43349) ------------------> (9550) |
|2.503 | ACK | |Seq = 0 Ack =
144804
| |(43349) <------------------ (9550) |
|2.503 | ACK - Len: 1448 |Seq = 146252 Ack =
0
| |(43349) ------------------> (9550) |
|2.503 | PSH, ACK - Len: 1448 |Seq = 147700 Ack =
0
| |(43349) ------------------> (9550) |
|2.503 | ACK | |Seq = 0 Ack =
146252
| |(43349) <------------------ (9550) |
|2.503 | ACK - Len: 1448 |Seq = 149148 Ack =
0
| |(43349) ------------------> (9550) |
|2.505 | ACK | |Seq = 0 Ack =
146252
| |(43349) <------------------ (9550) |
|2.505 | ACK - Len: 1448 |Seq = 150596 Ack =
0
| |(43349) ------------------> (9550) |
|2.505 | ACK | |Seq = 0 Ack =
146252
| |(43349) <------------------ (9550) |
|2.505 | PSH, ACK - Len: 1448 |Seq = 152044 Ack =
0
| |(43349) ------------------> (9550) |
|2.506 | ACK | |Seq = 0 Ack =
146252
| |(43349) <------------------ (9550) |
|2.506 | ACK - Len: 1448 |Seq = 146252 Ack =
0
| |(43349) ------------------> (9550) |
|2.507 | ACK | |Seq = 0 Ack =
150596
| |(43349) <------------------ (9550) |
|2.709 | ACK - Len: 1448 |Seq = 150596 Ack =
0
| |(43349) ------------------> (9550) |
|2.710 | ACK | |Seq = 0 Ack =
153492
| |(43349) <------------------ (9550) |
|2.710 | ACK - Len: 1448 |Seq = 153492 Ack =
0
| |(43349) ------------------> (9550) |
|2.710 | PSH, ACK - Len: 1448 |Seq = 154940 Ack =
0
| |(43349) ------------------> (9550) |
|2.711 | ACK | |Seq = 0 Ack =
154940
| |(43349) <------------------ (9550) |
|2.711 | ACK - Len: 1448 |Seq = 156388 Ack =
0
| |(43349) ------------------> (9550) |
|2.711 | ACK - Len: 1448 |Seq = 157836 Ack =
0
| |(43349) ------------------> (9550) |
|2.711 | ACK | |Seq = 0 Ack =
156388
| |(43349) <------------------ (9550) |
|2.711 | ACK - Len: 1448 |Seq = 159284 Ack =
0
| |(43349) ------------------> (9550) |
|2.713 | ACK | |Seq = 0 Ack =
156388
| |(43349) <------------------ (9550) |
|2.713 | ACK - Len: 1448 |Seq = 160732 Ack =
0
| |(43349) ------------------> (9550) |
|2.713 | ACK | |Seq = 0 Ack =
156388
| |(43349) <------------------ (9550) |
|2.713 | ACK - Len: 1448 |Seq = 162180 Ack =
0
| |(43349) ------------------> (9550) |
|2.714 | ACK | |Seq = 0 Ack =
156388
| |(43349) <------------------ (9550) |
|2.714 | ACK - Len: 1448 |Seq = 156388 Ack =
0
| |(43349) ------------------> (9550) |
|2.715 | ACK | |Seq = 0 Ack =
160732
| |(43349) <------------------ (9550) |
|2.917 | ACK - Len: 1448 |Seq = 160732 Ack =
0
| |(43349) ------------------> (9550) |
|2.918 | ACK | |Seq = 0 Ack =
163628
| |(43349) <------------------ (9550) |
|2.918 | ACK - Len: 1448 |Seq = 163628 Ack =
0
| |(43349) ------------------> (9550) |
|2.918 | ACK - Len: 1448 |Seq = 165076 Ack =
0
| |(43349) ------------------> (9550) |
|2.919 | ACK | |Seq = 0 Ack =
165076
| |(43349) <------------------ (9550) |
|2.919 | ACK - Len: 1448 |Seq = 166524 Ack =
0
| |(43349) ------------------> (9550) |
|2.919 | ACK - Len: 1448 |Seq = 167972 Ack =
0
| |(43349) ------------------> (9550) |
|2.919 | ACK | |Seq = 0 Ack =
166524
| |(43349) <------------------ (9550) |
|2.919 | ACK - Len: 1448 |Seq = 169420 Ack =
0
| |(43349) ------------------> (9550) |
|2.921 | ACK | |Seq = 0 Ack =
166524
| |(43349) <------------------ (9550) |
|2.921 | ACK - Len: 1448 |Seq = 170868 Ack =
0
| |(43349) ------------------> (9550) |
|2.921 | ACK | |Seq = 0 Ack =
166524
| |(43349) <------------------ (9550) |
|2.921 | ACK - Len: 1448 |Seq = 172316 Ack =
0
| |(43349) ------------------> (9550) |
|2.922 | ACK | |Seq = 0 Ack =
166524
| |(43349) <------------------ (9550) |
|2.922 | ACK - Len: 1448 |Seq = 166524 Ack =
0
| |(43349) ------------------> (9550) |
|2.923 | ACK | |Seq = 0 Ack =
170868
| |(43349) <------------------ (9550) |
|3.125 | ACK - Len: 1448 |Seq = 170868 Ack =
0
| |(43349) ------------------> (9550) |
|3.126 | ACK | |Seq = 0 Ack =
173764
| |(43349) <------------------ (9550) |
|3.126 | PSH, ACK - Len: 1448 |Seq = 173764 Ack =
0
| |(43349) ------------------> (9550) |
|3.126 | ACK - Len: 1448 |Seq = 175212 Ack =
0
| |(43349) ------------------> (9550) |
|3.127 | ACK | |Seq = 0 Ack =
175212
| |(43349) <------------------ (9550) |
|3.127 | ACK - Len: 1448 |Seq = 176660 Ack =
0
| |(43349) ------------------> (9550) |
|3.127 | ACK - Len: 1448 |Seq = 178108 Ack =
0
| |(43349) ------------------> (9550) |
|3.127 | ACK | |Seq = 0 Ack =
176660
| |(43349) <------------------ (9550) |
|3.127 | PSH, ACK - Len: 1448 |Seq = 179556 Ack =
0
| |(43349) ------------------> (9550) |
|3.128 | ACK | |Seq = 0 Ack =
176660
| |(43349) <------------------ (9550) |
|3.128 | ACK - Len: 1448 |Seq = 181004 Ack =
0
| |(43349) ------------------> (9550) |
|3.129 | ACK | |Seq = 0 Ack =
176660
| |(43349) <------------------ (9550) |
|3.129 | PSH, ACK - Len: 1448 |Seq = 182452 Ack =
0
| |(43349) ------------------> (9550) |
|3.130 | ACK | |Seq = 0 Ack =
176660
| |(43349) <------------------ (9550) |
|3.130 | ACK - Len: 1448 |Seq = 176660 Ack =
0
| |(43349) ------------------> (9550) |
|3.131 | ACK | |Seq = 0 Ack =
181004
| |(43349) <------------------ (9550) |
Maxim Yegorushkin

2006-11-26, 1:16 pm


Richard Eich wrote:
> maxim.yegorushkin@gmail.com wrote...
>
> Looks like we're getting a lot of retransmissions, which I believe
> would clearly explain the full sender tcp send-q, the receiver empty
> tcp recv-q, and the receiver's normal receive window size.
>
> (Relative ACKs used for clarity).

[]

In the final six tcp segments show that the last two 1448-byte segments
remain unacknowledged. The sender's tcp stack must retransmit the
segments as it happened in the packet dump before. The only explanation
I can think of why we can not see retransmissions in the dump, is that
the retransmissions never get to the network interface egress queue.
The queue is full when the network is congested or when the driver is
misbehaving. You may like to check ifconfig output, particularly if
errors, dropped and collisions fields are non zero for the related
interface.

Does send/write in the client return with an error and if so what is
errno value?

Richard Eich

2006-11-26, 1:16 pm

maxim.yegorushkin@gmail.com wrote...
>
> Richard Eich wrote:
>
> []
>
> In the final six tcp segments show that the last two 1448-byte segments
> remain unacknowledged. The sender's tcp stack must retransmit the
> segments as it happened in the packet dump before. The only explanation
> I can think of why we can not see retransmissions in the dump, is that
> the retransmissions never get to the network interface egress queue.
> The queue is full when the network is congested or when the driver is
> misbehaving. You may like to check ifconfig output, particularly if
> errors, dropped and collisions fields are non zero for the related
> interface.

I've been checking /proc/net/dev regularly already. In sum, hundreds
of thousands of packets out that interface, 34 errors and 0 drops.

> Does send/write in the client return with an error and if so what is
> errno value?

I log if write() to that socket returns <= 0, and haven't seen any
log messages for that event. I'll double-check that to make sure
it's unlikely that a log message is getting lost.

I'll also push to have the patches made current, just in case there's
the driver is a factor and there's already a fix for it.


Richard Eich

2006-11-26, 7:18 pm

richard.eich@domain.invalid wrote...
> maxim.yegorushkin@gmail.com wrote...
>
> I've been checking /proc/net/dev regularly already. In sum, hundreds
> of thousands of packets out that interface, 34 errors and 0 drops.
>
>
> I log if write() to that socket returns <= 0, and haven't seen any
> log messages for that event. I'll double-check that to make sure
> it's unlikely that a log message is getting lost.
>
> I'll also push to have the patches made current, just in case there's
> the driver is a factor and there's already a fix for it.

I've recently noticed that, even when the throughput is fine, that
the senders receive window hovers around 12. I take that to mean
that that TCP input buffer is consistently well-loaded.

Would another explanation for the lack of retransmissions in the dump
be that the ACKs are delayed through the TCP input buffer?

I've had the net.core.rmem_max and net.core.wmem_max set to 16MB for
a few weeks now. Maybe that's too big?
phil-news-nospam@ipal.net

2006-11-27, 1:29 am

In comp.unix.programmer Richard Eich <richard.eich@domain.invalid> wrote:

| Client and Server OS: SuSE 9.3 Pro 2.6.11 default kernel, neither
| machine patched after install from CDs.
|
| Communicating over: GigE LAN
|
| 1) Server (receiver) is consistently adversting a TCP RWIN of 32K.
| 2) Server consistently has TCP RECV-Q of 0.
| 3) Client (sender) consistently shows a TCP SEND-Q of 80K.
| 4) Socket is up and connection is ESTABLISHED from both sides.
| 5) No data is transmitted.
|
| To troubleshoot, I've torn down and re-established to connection
| countless times. There may be a trickle of data initially, but
| within a few seconds the client SEND-Q builds and transmission stops.
| Receiver's window size never goes below 32K.
|
| Never seen this kind of behavior before. If the server process was
| slow, I'd expect to see a RECV-Q buildup to go with the big SEND-Q.

I once had a problem that looked like this when doing a very large file
transfer (a tarball of a entire version of Slackware packages). It
would always stop at the same point, showing a full send queue at the
sender, and nothing at the receiver. Various diagnostics were attempted
and I finally discovered that the packets being resent were arriving at
the receiving machine sligtly corrupt. It seems there was a very strange
data sensitivity in the ethernet interface hardware apparently causing it
to lose sync and shift things off by one byte. I tried this on 3 other
machines with the same ethernet interface hardware and they all had the
very same problem. I installed a separate ethernet card on that machine
and used it instead of the ethernet in the motherboard, and then it was
fine. It all pointed at the ethernet chip on that particular motherboard.
I subsequently get around that issue by compressing and/or encrypting the
data. The point in the data where the corruption took place had many
bytes of binary zero in a row, but not too many. Longer sequences of
zeros did not cause a problem. It seemed to depend on some of the data
before the zeros as well. This problem was on the Intel ISP1100 server
and was reproducible on every ISP1100 I had access to. It did not occur
on any Intel ethernet card, nor any other ethernet I have.

If something in the data is causing data corruption at a hardware layer,
all the retransmissions will be in vain.

--
|---------------------------------------/----------------------------------|
| Phil Howard KA9WGN (ka9wgn.ham.org) / Do not send to the address below |
| first name lower case at ipal.net / spamtrap-2006-11-26-2228@ipal.net |
|------------------------------------/-------------------------------------|
Maxim Yegorushkin

2006-11-27, 7:22 am


Richard Eich wrote:

> richard.eich@domain.invalid wrote...
>
> I've recently noticed that, even when the throughput is fine, that
> the senders receive window hovers around 12. I take that to mean
> that that TCP input buffer is consistently well-loaded.

You probably have window scaling enabled by default, so 12 should be
scaled (12 * 2 ^ wscale). http://tools.ietf.org/html/rfc1323#section-2

The sender does not receive any data but ACKs, I'm not sure if its
receive buffer size has relevance.

> Would another explanation for the lack of retransmissions in the dump
> be that the ACKs are delayed through the TCP input buffer?
>
> I've had the net.core.rmem_max and net.core.wmem_max set to 16MB for
> a few weeks now. Maybe that's too big?

I would try using another network card / driver to see if the hardware
and the driver are good.

Richard Eich

2006-11-27, 1:18 pm

maxim.yegorushkin@gmail.com wrote...
>
> Richard Eich wrote:
>
>
> You probably have window scaling enabled by default, so 12 should be
> scaled (12 * 2 ^ wscale). http://tools.ietf.org/html/rfc1323#section-2
>
> The sender does not receive any data but ACKs, I'm not sure if its
> receive buffer size has relevance.

The sender has two of its four interfaces in promiscuous mode, and is
getting hit pretty hard 24x7 (i.e., 240+ million packets-in on eth0
alone, in the past six hours). Most of that is TCP.

>
> I would try using another network card / driver to see if the hardware
> and the driver are good.

I'm heading to that course of action. It's complicated a little by
the two interfaces in promisc mode being on an optical GiG/E card,
which was supplied after-market by our hardware vendor and
retrofitted by Solaris SAs. I'll have to get a replacement (not
difficult) and updated drivers (if any, not difficult) and then get
into a remote production datacenter and perform surgery (logistically
messy, but not difficult).

I greatly appreciate every single thing you've offered for help.
Sponsored Links






Free braindumps | Software forum | Database administration forum

Copyright 2003 - 2008 webservertalk.com