|
Home > Archive > Perlbal > October 2007 > SIGHUP?
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
|
|
| Matthew Pitts 2007-10-09, 7:12 pm |
| I've been running through the archives and found some mentions of using
the telnet interface for changing Perlbal's config on the fly. But I
really want to know if there is a signal I can send to Perlbal that will
tell it to re-read the config file and rebuild the running config.
We're using Perlbal as a single-public-IP frontend to a bunch of
commodity VMs, none of which have any real traffic, so a slight glitch
in service is acceptable, but I'd rather have a single command like kill
-SIGHUP <perlbal pid> than having to kill it completely and start a new
instance.
TIA for input on this one.
--
Matthew W. Pitts
Software Engineer
mpitts@a3its.com
336.202.3913 (mobile)
A3 IT Solutions, LLC
www.a3its.com
| |
| Mark Smith 2007-10-09, 7:12 pm |
| >
> I've been running through the archives and found some mentions of using
> the telnet interface for changing Perlbal's config on the fly. But I
> really want to know if there is a signal I can send to Perlbal that will
> tell it to re-read the config file and rebuild the running config.
>
> We're using Perlbal as a single-public-IP frontend to a bunch of
> commodity VMs, none of which have any real traffic, so a slight glitch
> in service is acceptable, but I'd rather have a single command like kill
> -SIGHUP <perlbal pid> than having to kill it completely and start a new
> instance.
>
There's no command to do that. Your best bet is to do a 'shutdown graceful'
on your existing Perlbal and start up a new one immediately. Old
connections are allowed to dwindle out and the new one can start handling
things with the new configuration.
The graceful shutdown isn't guaranteed to ever exit though (if a socket
stays alive it can stay up for quite a while) so you still want to watch it
and then kill it off after some period if it's still around.
--
Mark Smith / xb95
smitty@gmail.com
| |
| Todd Lipcon 2007-10-09, 7:12 pm |
| On Tue, 9 Oct 2007, Mark Smith wrote:
> There's no command to do that. Your best bet is to do a 'shutdown graceful'
> on your existing Perlbal and start up a new one immediately. Old
> connections are allowed to dwindle out and the new one can start handling
> things with the new configuration.
>
> The graceful shutdown isn't guaranteed to ever exit though (if a socket
> stays alive it can stay up for quite a while) so you still want to watch it
> and then kill it off after some period if it's still around.
Another trick you can do is use iptables to redirect incoming traffic to a
new perlbal. When you need to do a seamless restart, start up a second
perlbal on a new port and redirect from port 80 to that port. Existing
connections will be maintained, but new ones will head to the new
perlbal/configuration. Watch the stats on the first perlbal until
all the connections die out, at which point you can either kill it or do
the switch again in reverse.
-Todd
| |
| Matthew Pitts 2007-10-10, 1:12 pm |
| Thanks for the help...
On Tue, 2007-10-09 at 21:14 +0000, Mark Smith wrote:
> There's no command to do that. Your best bet is to do a 'shutdown
> graceful' on your existing Perlbal and start up a new one immediately.
> Old connections are allowed to dwindle out and the new one can start
> handling things with the new configuration.
Will Perlbal unbind from it's listening ports immediately upon getting
the 'shutdown graceful' command? If not, then would I need to somehow
monitor the old instance once I issue the command and then startup a new
Perlbal once the old one is completely finished?
If I were to write a Perlbal psuedo-init script, would the following
logic work?
- get Perlbal PID (from file?)
- connect to running Perlbal via telnet
- issue 'shutdown graceful' command
- wait until telnet socket is closed (indicating Perlbal has terminated)
- double check that Perlbal PID is no more (i.e. kill -0 $PID)
- start a new Perlbal
- connect to new Perlbal via telnet to confirm it's running
--
Matthew W. Pitts
Software Engineer
mpitts@a3its.com
336.202.3913 (mobile)
A3 IT Solutions, LLC
www.a3its.com
| |
| Matthew Pitts 2007-10-10, 1:12 pm |
| On Tue, 2007-10-09 at 17:18 -0400, Todd Lipcon wrote:
> Another trick you can do is use iptables to redirect incoming traffic
> to a
> new perlbal. When you need to do a seamless restart, start up a
> second
> perlbal on a new port and redirect from port 80 to that port.
> Existing
> connections will be maintained, but new ones will head to the new
> perlbal/configuration. Watch the stats on the first perlbal until
> all the connections die out, at which point you can either kill it or
> do
> the switch again in reverse.
This is a neat trick. Unfortunately, it would mean a bit of work for our
setup because I have a VHOST selector on port 80 that doles out
connections to reverse proxies running on other ports, which then talk
to the real backend servers. We'd bascially have to maintain 2 configs
with different ports for each service for doing the iptables trick on
port 80. Unless of course, I'm missing something.
--
Matthew W. Pitts
Software Engineer
mpitts@a3its.com
336.202.3913 (mobile)
A3 IT Solutions, LLC
www.a3its.com
| |
| Mark Smith 2007-10-10, 1:12 pm |
| >
> Will Perlbal unbind from it's listening ports immediately upon getting
> the 'shutdown graceful' command? If not, then would I need to somehow
> monitor the old instance once I issue the command and then startup a new
> Perlbal once the old one is completely finished?
>
> If I were to write a Perlbal psuedo-init script, would the following
> logic work?
>
> - get Perlbal PID (from file?)
> - connect to running Perlbal via telnet
> - issue 'shutdown graceful' command
> - wait until telnet socket is closed (indicating Perlbal has terminated)
> - double check that Perlbal PID is no more (i.e. kill -0 $PID)
> - start a new Perlbal
> - connect to new Perlbal via telnet to confirm it's running
>
This is the flow I used on LJ:
* connect to existing Perlbal management port
* issue 'shutdown graceful'
* start new Perlbal up immediately
* wait 10 seconds (usually I'd check the new Perlbal to make sure it's
getting traffic here)
* if the socket from the first step is still open, issue 'shutdown' command
If the socket goes away, then Perlbal shut itself down all of the way. If
not, then the final 'shutdown' forces an immediate termination. You can do
the work with the pid to ensure it's gone away if you want, that'd certainly
be safer.
But yes, Perlbal immediately unbinds all listening sockets when you issue a
graceful shutdown command.
As an aside, Perlbal should have a 'shutdown graceful [timeout]' option
where you can specify how many seconds to sit around for, then force a hard
exit. Hmm...
--
Mark Smith / xb95
smitty@gmail.com
| |
|
| Amusingly, I suddenly needed graceful timeout functionality as of this
morning, so cooked up a quick patch to add it.
It's a small five liner and is attached - just specify the number of
seconds when issuing the graceful shutdown command.
HTH.
Jonty Wareing
--
Developer
Last.fm
On 10/10/07, Mark Smith <smitty@gmail.com> wrote:
>
>
> This is the flow I used on LJ:
>
> * connect to existing Perlbal management port
> * issue 'shutdown graceful'
> * start new Perlbal up immediately
> * wait 10 seconds (usually I'd check the new Perlbal to make sure it's
> getting traffic here)
> * if the socket from the first step is still open, issue 'shutdown' command
>
> If the socket goes away, then Perlbal shut itself down all of the way. If
> not, then the final 'shutdown' forces an immediate termination. You can do
> the work with the pid to ensure it's gone away if you want, that'd certainly
> be safer.
>
> But yes, Perlbal immediately unbinds all listening sockets when you issue a
> graceful shutdown command.
>
> As an aside, Perlbal should have a 'shutdown graceful [timeout]' option
> where you can specify how many seconds to sit around for, then force a hard
> exit. Hmm...
>
>
>
> --
> Mark Smith / xb95
> smitty@gmail.com
| |
| Mark Smith 2007-10-12, 1:11 pm |
| >
> Amusingly, I suddenly needed graceful timeout functionality as of this
> morning, so cooked up a quick patch to add it.
>
> It's a small five liner and is attached - just specify the number of
> seconds when issuing the graceful shutdown command.
>
Thanks! This is now in trunk. 
--
Mark Smith / xb95
smitty@gmail.com
|
|
|
|
|