Today at the MogileFS summit Alan (from http://www.gaiaonline.com/)
mentioned that if he enables persistent backend connections, users get
mismatched requests to responses.... but not when he accesses it.  And it
takes awhile to happen during low load.

All this pointed to errors handling malformed client requests, and sure
enough, I think this is it....

We don't disconnect users who do pipelining.  I was able to sneak through
multiple requests to the backend and then two requests would come back,
but they'd be assigned to different users.

This fix doesn't break the test suite, but before I commit, Alan --- can
you verify it fixes things for you?



Index: lib/Perlbal/ClientProxy.pm
 ========================================
===========================
--- lib/Perlbal/ClientProxy.pm  (revision 565)
+++ lib/Perlbal/ClientProxy.pm  (working copy)
@@ -584,13 +584,11 @@
# (see: Danga::Socket::read)
return $self->client_disconnected unless defined $bref;

-    # if we got data that we weren't expecting, something's bogus with
-    # our state machine (internal error)
-    if (defined $remain && ! $remain) {
-        my $blen = length($$bref);
-        my $content = substr($$bref, 0, 80 < $blen ? 80 : $blen);
-        Carp::cluck("INTERNAL ERROR: event_read called on when we're expect
ing no more bytes.  len=$blen, content=[$content]\n");
-        $self->close;
+    # if they didn't declare a content body length and we just got a
+    # readable event that's not a disconnect, something's messed up.
+    # they're overflowing us.  disconnect!
+    if (! $remain) {
+        $self->close("over_wrote");
return;
}


- Brad

[ Post a follow-up to this message ]

I can't screw with the site past 4:30pm, so first thing tomorrow morning
I'll patch (hopefully just one of?) the LB's and try it out.

Thanks!
-Alan

Brad Fitzpatrick wrote:
> Today at the MogileFS summit Alan (from http://www.gaiaonline.com/)
> mentioned that if he enables persistent backend connections, users get
> mismatched requests to responses.... but not when he accesses it.  And it
> takes awhile to happen during low load.
>
> All this pointed to errors handling malformed client requests, and sure
> enough, I think this is it....
>
> We don't disconnect users who do pipelining.  I was able to sneak through
> multiple requests to the backend and then two requests would come back,
> but they'd be assigned to different users.
>
> This fix doesn't break the test suite, but before I commit, Alan --- can
> you verify it fixes things for you?
>
>
>
> Index: lib/Perlbal/ClientProxy.pm
>  ========================================
===========================
> --- lib/Perlbal/ClientProxy.pm  (revision 565)
> +++ lib/Perlbal/ClientProxy.pm  (working copy)
> @@ -584,13 +584,11 @@
>      # (see: Danga::Socket::read)
>      return $self->client_disconnected unless defined $bref;
>
> -    # if we got data that we weren't expecting, something's bogus with
> -    # our state machine (internal error)
> -    if (defined $remain && ! $remain) {
> -        my $blen = length($$bref);
> -        my $content = substr($$bref, 0, 80 < $blen ? 80 : $blen);
> -        Carp::cluck("INTERNAL ERROR: event_read called on when we're expe
cting no more bytes.  len=$blen, content=[$content]\n");
> -        $self->close;
> +    # if they didn't declare a content body length and we just got a
> +    # readable event that's not a disconnect, something's messed up.
> +    # they're overflowing us.  disconnect!
> +    if (! $remain) {
> +        $self->close("over_wrote");
>          return;
>      }
>
>
> - Brad

[ Post a follow-up to this message ]

Thanks!

On Tue, 19 Sep 2006, dormando wrote:

> I can't screw with the site past 4:30pm, so first thing tomorrow morning
> I'll patch (hopefully just one of?) the LB's and try it out.
>
> Thanks!
> -Alan
>
> Brad Fitzpatrick wrote: 
>
>

[ Post a follow-up to this message ]

Looks like that fixed it! We've had zero reports of page swapping /
image swapping since applying the patch. Site's running much snappier
during peak load with backend keepalives working.

I'll probably follow up with some information on exactly what kind of
client is hitting the disconnect call in the patch... Not going to have
time until next week.

-Alan / Dormando / whatever.

dormando wrote:[vbcol=seagreen]
> I can't screw with the site past 4:30pm, so first thing tomorrow morning
> I'll patch (hopefully just one of?) the LB's and try it out.
>
> Thanks!
> -Alan
>
> Brad Fitzpatrick wrote: 

[ Post a follow-up to this message ]

On Fri, 22 Sep 2006, Jacques Marneweck wrote:

> On Thu Sep 21 16:43:16 UTC 2006, dormando <dormando@rydia.net> wrote: 
>
> Alan, that's great news 
>
> Brad what would the ETA for the next minor release of perbal be?

It's in svn, along with some new tests, but I'm not entirely happy with
its level of paranoia, safety-in-face-of-stupid-clients, and logging.
This week, though, if not later today.


>
> Regards
> --jm 
>
>

[ Post a follow-up to this message ]