Hi all!

I have a Linux server (FC3) with 3 application that collects several
data from 3 equipments using sockets.
These 3 application are restarted (killed and start) each 15 minutes.
Several hours later, when restarting the application, I get a
'Connection refused' error (in the 3 application).
After this error if we reboot the server everthing goes OK. After
several hours the same error happens.
Does any one has an ideia how to find what is causing this kind of
problema?

I will be grateful for any help.

Regards,

Oliveira

[ Post a follow-up to this message ]

Oliveira wrote:

> I have a Linux server (FC3) with 3 application that collects several
> data from 3 equipments using sockets.
> These 3 application are restarted (killed and start) each 15 minutes.
> Several hours later, when restarting the application, I get a
> 'Connection refused' error (in the 3 application).
> After this error if we reboot the server everthing goes OK. After
> several hours the same error happens.
> Does any one has an ideia how to find what is causing this kind of
> problema?

When you get a 'connection refused', it usually means that nothing is
listening on the port you're trying to connect to. There are many
reasons this can happen.

First, check to make sure that the program that listens on that port is
actually running. Then check to see if it has a socket bound to that
port (with 'netstat' or a similar tool). Then check to make sure it
hasn't hung or crashed.

Also, make sure the application will not start up if it fails to bind
to its primary port (the one that you're connecting to). At an absolute
minimum, log the inability to bind to the port on startup and see if
that log entry coincides with your problem.

In other words, troubleshoot the error condition and narrow it down. If
I had to take a total guess, I'd guess that it's a combination of not
setting SO_REUSEPORT and not being diligent in attempting to 'bind' on
startup.

DS

[ Post a follow-up to this message ]

David Schwartz wrote:
> Oliveira wrote:
> 
>
> When you get a 'connection refused', it usually means that nothing is
> listening on the port you're trying to connect to. There are many
> reasons this can happen.
>
> First, check to make sure that the program that listens on that port is
> actually running. Then check to see if it has a socket bound to that
> port (with 'netstat' or a similar tool). Then check to make sure it
> hasn't hung or crashed.

In addition to said, it might be a resource leak in the server. On
linux the default limit of open file descriptors may be 1024, the
server might me running out of the them neglecting to close file
descriptors, so that accept fails and RST is sent back to the client,
which causes connect fail with ECONNREFUSED.

[ Post a follow-up to this message ]

Maxim Yegorushkin wrote:
> David Schwartz wrote: 
>
> In addition to said, it might be a resource leak in the server. On
> linux the default limit of open file descriptors may be 1024, the
> server might me running out of the them neglecting to close file
> descriptors, so that accept fails and RST is sent back to the client,
> which causes connect fail with ECONNREFUSED.

A quick test to see if the server is leaking fd's would be something
like:

$ lsof -p `pidof server`

[ Post a follow-up to this message ]

First of all thank you very much (David and Maxim) for the answers.

I have one more information that can be useful.

When I do this
netstat | grep http | wc -l

I have more than 1500 lines! Apache is running as web server.

This (high) number of http connections can cause the problem related
with socket connection?

[ Post a follow-up to this message ]

Oliveira wrote:
> First of all thank you very much (David and Maxim) for the answers.
>
> I have one more information that can be useful.
>
> When I do this
>    netstat | grep http | wc -l
>
> I have more than 1500 lines! Apache is running as web server.
>
> This (high) number of http connections can cause the problem related
> with socket connection?

Do you experience connectivity problems with your apache?

[ Post a follow-up to this message ]

Oliveira wrote:
> First of all thank you very much (David and Maxim) for the answers.
>
> I have one more information that can be useful.
>
> When I do this
>    netstat | grep http | wc -l
>
> I have more than 1500 lines! Apache is running as web server.
>
> This (high) number of http connections can cause the problem related
> with socket connection?

Are these connections to Apache? Are you sure they're to a port Apache
is using?

DS

[ Post a follow-up to this message ]

David Schwartz escreveu:

> Are these connections to Apache? Are you sure they're to a port Apache
> is using?
>
> DS

I think so. The (netstat) lines are similar to the folowing:

tcp        0      0 ::ffff:172.17.11.3:http
::ffff:172.17.11.235:1352   TIME_WAIT

The 172.17.11.3 is the (web) server and the 172.17.11.235 is one client
computer in the plant.

Oliveira

[ Post a follow-up to this message ]

Oliveira wrote:
> David Schwartz escreveu:
> 
>
> I think so. The (netstat) lines are similar to the folowing:
>
> tcp        0      0 ::ffff:172.17.11.3:http
> ::ffff:172.17.11.235:1352   TIME_WAIT
>
> The 172.17.11.3 is the (web) server and the 172.17.11.235 is one client
> computer in the plant.
>
> Oliveira

Is the connection rate supposed to be so high? This indicates a very
high rate of connections, probably 8 per second or so.

DS

[ Post a follow-up to this message ]

David Schwartz escreveu:
> Is the connection rate supposed to be so high? This indicates a very
> high rate of connections, probably 8 per second or so.
>
> DS

Each cliente computer has a web browser opened (IE) and the web page is
self refreshing each 6 seconds (refreshing is done 10 time per minute).

I change the refresh period from 6 seconds to 90 seconds. Now the
command
netstat | grep http | wc -l

gives somothing about 180. Until now I the "connection refused" problem
never happens, but I still have doubts if the increase of the refresh
period solve the problem.

Once again, thank you for your help.

Oliveira

[ Post a follow-up to this message ]