Perlbal - node health

Author

node health

J Davis

2007-10-16, 7:11 pm

Hello,

I've just installed a test instance of Perlbal 1.59 from CPAN. I intend to
use it to add load balancing and high availability to an IIS/ColdFusion
site. My initial impression is that it works wonderfully. My question is
about the options available for determining the health of the nodes in a
reverse_proxy pool. I see from the docs that when "verify_backend" is on
Perlbal will perform a quick OPTIONS request on the nodes. Is there any way
to add additional tests so that, for instance, a nodes DB connectivity might
be verified by checking the content of an http request?

If not, is this something that might be accomplished using a secondary app
to monitor the nodes and update Perlbal on the fly?

I have some ideas about how this might be accomplished but advice and
recommendations from folks who may have experience with this would be
greatly appreciated.

Thanks,
-Jake

Brett Hoerner

2007-10-16, 7:11 pm

On Oct 16, 2007, at Oct16, 2:22:36PM, J Davis wrote:
> Is there any way to add additional tests so that, for instance, a
> nodes DB connectivity might be verified by checking the content of
> an http request?

I don't know of a way to do this through normal configuration or
plugins, but you could check out BackendHTTP.pm, the functions
event_write and event_read_waiting_options are the ones that
currently deal with the OPTIONS check. You could modify it hit a
specific URI with a GET, instead... and then check for some magical
"We're all good" in the headers, I guess. Keep in mind that the
OPTIONS check is currently when you first get a request (afaik), and
it's assumed to be very fast. If your DB check takes a while it
could block the first unlucky user's request... if I'm reading it
correctly. Just sort of stabbing here, I'm not a Perlbal pro. I'm
sure you're better off with the next option.

> If not, is this something that might be accomplished using a
> secondary app to monitor the nodes and update Perlbal on the fly?

Sure, your app (daemon or cron, whatever) could check your server for
connectivity then either ADD/REMOVE servers from a POOL via the
telnet management port, or change a nodelist file (which Perlbal will
eventually re-read).

Brett

dormando

2007-10-16, 7:11 pm

> I don't know of a way to do this through normal configuration or
> plugins, but you could check out BackendHTTP.pm, the functions
> event_write and event_read_waiting_options are the ones that currently
> deal with the OPTIONS check. You could modify it hit a specific URI
> with a GET, instead... and then check for some magical "We're all good"
> in the headers, I guess. Keep in mind that the OPTIONS check is
> currently when you first get a request (afaik), and it's assumed to be
> very fast. If your DB check takes a while it could block the first
> unlucky user's request... if I'm reading it correctly. Just sort of
> stabbing here, I'm not a Perlbal pro. I'm sure you're better off with
> the next option.

OPTIONS is only ran when perlbal first connects to an individual apache
process, so it's not used for the client side service. For each
keepalive request it'll just blind forward. It's better to have a health
check service for the (often rarer) case of the code on an individual
machine exploding.

> Sure, your app (daemon or cron, whatever) could check your server for
> connectivity then either ADD/REMOVE servers from a POOL via the telnet
> management port, or change a nodelist file (which Perlbal will
> eventually re-read).

Yup

It'll reload the nodelist pretty fast, even! As a caveat (this at
least _used_ to be true), once you modify a pool with ADD/REMOVE, it
won't check the nodelist anymore.

-Dormando

J Davis

2007-10-17, 1:12 pm

Many thanks. It sounds like using an outside health check to modify the
nodefile is probably the way to go.
I found the code that sets the NODEFILE_RELOAD_FREQ to 3 in Pool.pm so I
guess that means it checks for updates every three seconds as long as I
don't add/remove nodes through the management service.

Thanks!
-Jake

On 10/16/07, dormando <dormando@rydia.net> wrote:
>
>
>
> OPTIONS is only ran when perlbal first connects to an individual apache
> process, so it's not used for the client side service. For each
> keepalive request it'll just blind forward. It's better to have a health
> check service for the (often rarer) case of the code on an individual
> machine exploding.
>
>
> Yup

It'll reload the nodelist pretty fast, even! As a caveat (this at
> least _used_ to be true), once you modify a pool with ADD/REMOVE, it
> won't check the nodelist anymore.
>
> -Dormando
>

Mark Smith

2007-10-17, 1:12 pm

>
> Many thanks. It sounds like using an outside health check to modify the
> nodefile is probably the way to go.
> I found the code that sets the NODEFILE_RELOAD_FREQ to 3 in Pool.pm so I
> guess that means it checks for updates every three seconds as long as I
> don't add/remove nodes through the management service.
>

Yep, that's right.

I agree with the previous posters, this is probably the right solution for
you. I have a bit of a soft spot for writing plugins and would have solved
this by doing that for my own case. (Tie it in with the management console,
ooh!)

--
Mark Smith / xb95
smitty@gmail.com

Brett Hoerner

2007-10-17, 1:12 pm

On Oct 16, 2007, at Oct16, 3:58:26PM, dormando wrote:
> OPTIONS is only ran when perlbal first connects to an individual
> apache process, so it's not used for the client side service. For
> each keepalive request it'll just blind forward. It's better to
> have a health check service for the (often rarer) case of the code
> on an individual machine exploding.

Right, OPTIONS is only run once per backend in the POOL, but if I'm
reading (and testing) it correctly it doesn't run until a user hits
Perlbal and Perlbal looks for a backend to serve the request, so the
first user(s) to request when Perlbal still has unverified backends
will have to wait for the OPTIONS to come back... which is fast, but
his more-intense App-DB check could block for longer.

I still agree that nodelist is the best bet.

Brett