Allo,
 
I'm currently running Perlbal (1.59, reverse_proxy) to load balance a
bunch of mongrels.  Occasionally I get to a point where Perlbal will
serve return a blank white screen instead of returning the app response.
My users have dubbed it the "White Screen Of Death".  I'm not sure where
to begin debugging the problem or sorting out the cause.
 
I see the issue happen when I have a spike in requests or if I kill off
the mongrels.  I have found that adding more mongrels usually leads to
less white screens.
 
Also, it seems that verify_backend doesn't really jive with mongrel.  I
suspect it's because mongrel doesn't respond to an OPTIONS request.  Is
there any way to benefit from verify_backend in my case?

[ Post a follow-up to this message ]

On Oct 31, 2007, at 1:26 PM, TJ Murphy wrote:

> I see the issue happen when I have a spike in requests or if I kill
> off the mongrels.  I have found that adding more mongrels usually
> leads to less white screens.

Could the mongrels be failing with an empty response?

> Also, it seems that verify_backend doesn't really jive with
> mongrel.  I suspect it's because mongrel doesn't respond to an
> OPTIONS request.  Is there any way to benefit from verify_backend in
> my case?


Adding enough support for OPTIONS into mogrel for the perlbal verify
to work should be pretty easy ...


- ask

--
http://develooper.com/ - http://askask.com/

[ Post a follow-up to this message ]

I sent a patch to allow this to be a bit more tuneable, which allows
this to work with Rails.  (If anyone can figure out how to get Rails
to respond to an OPTION * request, you are my hero.)  It apparently
got lost in the shuffle for the last release, so here it is, updated
for the current trunk (should apply directly to 1.60 as well)

It allows you to set a verify_backend_path, which defaults to the
current '*'.  For rails, you can set it to /, or any other url you
would like to route your OPTIONs request to.

Perlbal + Mongrel with verify_backend solves a pretty irritating
behavior on mongrel's part, which is the caching of connections while
it waits on the Rails process to complete.  I'm using this
patched-perlbal in production on several different Rails apps, and it
works great.

Regards,
Adam

On 10/31/07, TJ Murphy <TJ@freewebs.com> wrote:
>
>
> Allo,
>
> I'm currently running Perlbal (1.59, reverse_proxy) to load balance a bunc
h
> of mongrels.  Occasionally I get to a point where Perlbal will serve retur
n
> a blank white screen instead of returning the app response.  My users have
> dubbed it the "White Screen Of Death".  I'm not sure where to begin
> debugging the problem or sorting out the cause.
>
> I see the issue happen when I have a spike in requests or if I kill off th
e
> mongrels.  I have found that adding more mongrels usually leads to less
> white screens.
>
> Also, it seems that verify_backend doesn't really jive with mongrel.  I
> suspect it's because mongrel doesn't respond to an OPTIONS request.  Is
> there any way to benefit from verify_backend in my case?


--
HJK Solutions - We Launch Startups - http://www.hjksolutions.com
Adam Jacob, Senior Partner
T: (206) 508-4759 E: adam@hjksolutions.com

[ Post a follow-up to this message ]

On Nov 1, 2007, at 16:38, Adam Jacob wrote:

> I sent a patch to allow this to be a bit more tuneable, which allows
> this to work with Rails.

Committed r722.


- ask

--
http://develooper.com/ - http://askask.com/

[ Post a follow-up to this message ]

Using your patch, I did a little more digging.  Here's what I found:

Like you noted, Perlbal's verify_backend option defaults to doing an
HTTP OPTIONS request on the backend server at the path *

Mongrel doesn't like -any- request at the path *.  It throws an error
and doesn't respond.

When Perlbal doesn't get a response from the OPTIONS request, it goes
looking for another backend to use.  For my test setup I had only 1
backend, so it decided to reuse the same server.  It flooded Mongrel
with OPTIONS requests indefinitely.  It wasn't pretty.

Next, I set verify_backend_path to /.  Mongrel accepted this happily and
spit back a response as if it were an HTTP GET.

Unfortunately, I can't really afford to double the number of requests
I'm doing,  So I took it a step further.

I created a custom Mongrel handler insired by ErrCount, a fast cgi-style
counter in Rails (http://errtheblog.com/post/8444).  I set
verify_backend_path to /ctr and had the custom Mongrel handler listen on
/ctr.  All the handler does is pick up the request and spit out a blank
200 response (see attached).  It's fast.

To get this script working with a mongrel setup, just put the path to
the script in your mongrel_cluster.yml as config_script.   You can also
pass the script in on the command line using -S.

This setup has been running for over a week under really heavy load, and
it's holding up well.

-TJ



-----Original Message-----
From: Adam Jacob [mailto:adam@hjksolutions.com] 
Sent: Thursday, November 01, 2007 7:39 PM
To: TJ Murphy
Cc: perlbal@lists.danga.com
Subject: Re: Perlbal with Mongrels

I sent a patch to allow this to be a bit more tuneable, which allows
this to work with Rails.  (If anyone can figure out how to get Rails to
respond to an OPTION * request, you are my hero.)  It apparently got
lost in the shuffle for the last release, so here it is, updated for the
current trunk (should apply directly to 1.60 as well)

It allows you to set a verify_backend_path, which defaults to the
current '*'.  For rails, you can set it to /, or any other url you would
like to route your OPTIONs request to.

Perlbal + Mongrel with verify_backend solves a pretty irritating
behavior on mongrel's part, which is the caching of connections while it
waits on the Rails process to complete.  I'm using this patched-perlbal
in production on several different Rails apps, and it works great.

Regards,
Adam

On 10/31/07, TJ Murphy <TJ@freewebs.com> wrote:
>
>
> Allo,
>
> I'm currently running Perlbal (1.59, reverse_proxy) to load balance a 
> bunch of mongrels.  Occasionally I get to a point where Perlbal will 
> serve return a blank white screen instead of returning the app 
> response.  My users have dubbed it the "White Screen Of Death".  I'm 
> not sure where to begin debugging the problem or sorting out the
cause.
>
> I see the issue happen when I have a spike in requests or if I kill 
> off the mongrels.  I have found that adding more mongrels usually 
> leads to less white screens.
>
> Also, it seems that verify_backend doesn't really jive with mongrel.  
> I suspect it's because mongrel doesn't respond to an OPTIONS request.

> Is there any way to benefit from verify_backend in my case?


--
HJK Solutions - We Launch Startups - http://www.hjksolutions.com Adam
Jacob, Senior Partner
T: (206) 508-4759 E: adam@hjksolutions.com

[ Post a follow-up to this message ]

On Nov 5, 2007, at 16:05, TJ Murphy wrote:

> Next, I set verify_backend_path to /.  Mongrel accepted this happily
> and
> spit back a response as if it were an HTTP GET.
>
> Unfortunately, I can't really afford to double the number of requests
> I'm doing,  So I took it a step further.

*double* ?

Don't mogrel keep Keep-Alive connections?  It should only be a "check"
request per maybe 50, 500 or 50000 "real" requests ....


- ask

--
http://develooper.com/ - http://askask.com/

[ Post a follow-up to this message ]

On 11/5/07, TJ Murphy <TJ@freewebs.com> wrote:
> Using your patch, I did a little more digging.  Here's what I found:
>
> Like you noted, Perlbal's verify_backend option defaults to doing an
> HTTP OPTIONS request on the backend server at the path *
>
> Mongrel doesn't like -any- request at the path *.  It throws an error
> and doesn't respond.

Yeah, that's what I have seen as well.  I can't get Webrick to answer
one either.

> Next, I set verify_backend_path to /.  Mongrel accepted this happily and
> spit back a response as if it were an HTTP GET.

You can set a Rails route to answer this on an Options request.

map.connect '/check',
:controller => 'options',
:action => 'options',
:conditions => { :method => :options }

Then, in app/controllers/options_controller.rb:

class OptionsController < ApplicationController
def options
render :text => "OK"
end
end

That same method can be expanded to provide a canary-style check for
use with load balancers, etc.

In case you didn't want mongrel answering it.

> To get this script working with a mongrel setup, just put the path to
> the script in your mongrel_cluster.yml as config_script.   You can also
> pass the script in on the command line using -S.
>
> This setup has been running for over a week under really heavy load, and
> it's holding up well.

Yeah, it's probably the right way to go.  Since Rails locks the
mongrel, having it not even enter the Rails app should be no harm/no
foul.  But the above will do it in Rails for you, if you care. 

Adam

--
HJK Solutions - We Launch Startups - http://www.hjksolutions.com
Adam Jacob, Senior Partner
T: (206) 508-4759 E: adam@hjksolutions.com

[ Post a follow-up to this message ]