Perlbal - Limiting backend connections per node

Author

Limiting backend connections per node

dormando

2005-11-16, 5:45 pm

Hey all,

I have gotten perlbal up and running without issue, so far. However, now
stands the daunting task of testing it out of and in production ;)

I see perlbal relies on the backend servers MaxClients (or similar)
setting to limit the number of accepting backend connections that each
server has.

This is an issue for me (at the moment), since 35+ of my backend
webservers were set up by hand, use different linux distros, have their
configs in different places, etc. I can't rebuild them for a couple more
weeks, so it'd be much easier if perlbal had a node option to limit the
max number of backend connections it will try to open to each
independent node. It would also make it much simpler to try different
maximums for the backend to see which is most efficient for throughput.
Otherwise (as far as I can tell) I have to change the config and grace
apache on 100+ machines, which even when done in parallel is a bit
obnoxious.

So uhh. Should I try hacking it in, does perlbal not work right with
this kind of setup, or what?

Thanks,
-Alan

Mark Smith

2005-11-16, 5:45 pm

> I see perlbal relies on the backend servers MaxClients (or similar)
> setting to limit the number of accepting backend connections that each
> server has.
>
> This is an issue for me (at the moment), since 35+ of my backend
> webservers were set up by hand, use different linux distros, have their
> configs in different places, etc. I can't rebuild them for a couple more
> weeks, so it'd be much easier if perlbal had a node option to limit the
> max number of backend connections it will try to open to each
> independent node. It would also make it much simpler to try different
> maximums for the backend to see which is most efficient for throughput.
> Otherwise (as far as I can tell) I have to change the config and grace
> apache on 100+ machines, which even when done in parallel is a bit
> obnoxious.

I guess I'm a little confused as to where the problem is. Perlbal will
connect to a node until it stops processing requests, yes, but this means
that Perlbal will only ever have N+1 connections open to each backend,
where N is the number of MaxClients.

This should be ideal, assuming your MaxClients are set to sane values. If
you haven't, though, and they're set really high -- then yes, you're going
to lose performance on those nodes when they get above their limit.

On LiveJournal, if that happens, we just fix the MaxClients for the node
that is underperforming. That may be 'annoying' sure, but it's always
worked for us.

> So uhh. Should I try hacking it in, does perlbal not work right with
> this kind of setup, or what?

Unless I'm totally missing the boat (possible! hit me if so) then you
shouldn't have to do anything. N+1 isn't going to hurt your nodes,
especially since the +1 request.

Oh -- and in case it's not clear: for this behavior to be, you have to
enable the backend verify and use OPTION requests options ... I forget the
names offhand but they're in the example configs.

--
Junior (aka Mark Smith)
junior@danga.com

Software Engineer
Six Apart / Danga Interactive

dormando

2005-11-16, 5:45 pm

No problem, let me clarify;

>
>I guess I'm a little confused as to where the problem is. Perlbal will
>connect to a node until it stops processing requests, yes, but this means
>that Perlbal will only ever have N+1 connections open to each backend,
>where N is the number of MaxClients.
>
>
>
That's exactly how I understand it works.

>This should be ideal, assuming your MaxClients are set to sane values. If
>you haven't, though, and they're set really high -- then yes, you're going
>to lose performance on those nodes when they get above their limit.
>
>
>
They're not set to sane values

I have 50 webservers where httpd.conf
might not even be named httpd.conf right now. Adjusting the MaxClients
setting across all of my webservers (at the moment) is a huge manual
production. I've been here about a month and a half and am still busy
cleaning it up.

>On LiveJournal, if that happens, we just fix the MaxClients for the node
>that is underperforming. That may be 'annoying' sure, but it's always
>worked for us.
>
>
>
That's totally fine, but changing it for me is nontrivial at the moment.
If I have to switch back to the old load balancer (which does no client
buffering), I have to up the maxclients back to 60+ across the backend.

>Oh -- and in case it's not clear: for this behavior to be, you have to
>enable the backend verify and use OPTION requests options ... I forget the
>names offhand but they're in the example configs.
>
>
>
Yup, I have that all set. Only thing I can't set at the moment is the
persist_backend, but that will be enabled if I can switch the whole
service over and enable keepalives on the backend (and that perlbal
supports our backend for persistent connections; I noticed chunked
encoding short-circuits that).

Thanks,
-Alan