Perlbal - Bizarre perlbal problem

Author

Bizarre perlbal problem

Dormando

2005-11-18, 5:45 pm

(sorry, couldn't resist :P)

We got HTTP 1.0 keepalives working (for the most part) late in the day
yesterday. Overnight we witnessed a pretty bad glitch where our users
would randomly get other user's site cookies and become logged in as
someone else.

It happened in a small percentage of users, but turning off the backend
keepalives seems to have removed the issue. We're still investigating on
our end, but I'm having a hard time even speculating how that happened;
was perlbal sending back responses from clients other than the
requestor? Any insight?

Other things that came to mind:
- It's possible sometimes we return an improper content-length.
- If a client connection closes before reading any data back from its
connection, does perlbal always junk the backend request before reusing it?

I do have a weird routing setup in order to have two perlbal processes
running on the same IP address, but since that's just on the frontend
and only for incoming requests, I can't see how the responses would get
switched... If they were, it'd happen a lot more often than with what we
saw.

Thanks,
-Alan

Brad Fitzpatrick

2005-11-18, 5:45 pm

Does your backend's session management assuming that for a given
connection, the same user will always be on that connection? What backend
do you use, btw?

If so, that's your problem, since Perlbal mixes up requests and
connections. (on purpose... it's a huge optimization)

- Brad

On Fri, 18 Nov 2005, Dormando wrote:

> (sorry, couldn't resist :P)
>
> We got HTTP 1.0 keepalives working (for the most part) late in the day
> yesterday. Overnight we witnessed a pretty bad glitch where our users
> would randomly get other user's site cookies and become logged in as
> someone else.
>
> It happened in a small percentage of users, but turning off the backend
> keepalives seems to have removed the issue. We're still investigating on
> our end, but I'm having a hard time even speculating how that happened;
> was perlbal sending back responses from clients other than the
> requestor? Any insight?
>
> Other things that came to mind:
> - It's possible sometimes we return an improper content-length.
> - If a client connection closes before reading any data back from its
> connection, does perlbal always junk the backend request before reusing it?
>
> I do have a weird routing setup in order to have two perlbal processes
> running on the same IP address, but since that's just on the frontend
> and only for incoming requests, I can't see how the responses would get
> switched... If they were, it'd happen a lot more often than with what we
> saw.
>
> Thanks,
> -Alan
>
>

Brad Fitzpatrick

2005-11-18, 5:45 pm

I hear your concern, but I'm not worried about Perlbal mixing up requests
and responses.

However, what I'm asking about is different: not that each request can go
to different servers, but that php isn't caching any session info data on
the CONNECTION object, not the REQUEST object.

So you got a TCP connection with a bunch of HTTP requests inside it:

<tcp>
<http_req />
<http_req />
<http_req />
<http_req />
<http_req />
</tcp>

If the php is making the assumption that each http request is from the
same user (which is usually the case!) and caching on the TCP connection
object, then Perlbal could be confusing it, because Perlbal's persistent
connections to php aren't aligned with users at all.

On Fri, 18 Nov 2005, Dormando wrote:

> I don't think so...
>
> Our backend is just direct connections to php running behind either
> apache1 or apache2. As long as the remote user's independent requests go
> down and come back from the same server it should work. IE; If user A
> sends a request down, it just needs to get the reply to that request
> back. If user A sends a request down, but gets the response headers from
> user B, we'd get what we're seeing now.
>
> There's no reliance on the same user hitting the same exact backend
> connection for multiple requests. Before we put the LB in there was no
> keepalive at all, and users would hit one of 80 servers at random.
>
> -Alan
>
> Brad Fitzpatrick wrote:
>
>

Dormando

2005-11-18, 5:45 pm

Ok, all I had was speculation. This sounds more grounding.

Now, you might know that I'm not a php haxx0r. You might say I barely
know the first thing about the language; The fact that php seems to mix
up some stuff per connection instead of per request was news to me. Our
backend is also running a dozen different versions of php at the moment
(which I'm working on fixing!), so it's possible there's a glitch or two
involved here.

I can reason the glitch happening sometimes and being related to PHP
persisting some data either because of a glitch in an old version of PHP
or because of a code bug in a backend I've never looked at myself. It
doesn't feel like a widespread glitch though, given how few of our users
are hit by it. If the first request to a keep-alive connection had all
the login data for the rest of the requests, every one of our users
would be logged out within a couple minutes.

I'll go back to working with one of the programmers in hunting it down,
but for the record; are there any other reasons why this might happen?
Nothing related to connections blowing up or dying or getting confused?
What happens when a bad content-length is fed back to perlbal on accident?

(yes, we're checking X-Forwarded-For and such, I made sure of that at
least).

Thanks,
-Alan

Brad Fitzpatrick wrote:[vbcol=seagreen]
> I hear your concern, but I'm not worried about Perlbal mixing up requests
> and responses.
>
> However, what I'm asking about is different: not that each request can go
> to different servers, but that php isn't caching any session info data on
> the CONNECTION object, not the REQUEST object.
>
> So you got a TCP connection with a bunch of HTTP requests inside it:
>
> <tcp>
> <http_req />
> <http_req />
> <http_req />
> <http_req />
> <http_req />
> </tcp>
>
> If the php is making the assumption that each http request is from the
> same user (which is usually the case!) and caching on the TCP connection
> object, then Perlbal could be confusing it, because Perlbal's persistent
> connections to php aren't aligned with users at all.
>
>
>
> On Fri, 18 Nov 2005, Dormando wrote:
>
>

Brad Fitzpatrick

2005-11-18, 5:45 pm

Perlbal closes connections whenever it gets the slightly hint of somebody
being confused. This was less true in the past, but the past 6 months or
so of development has been adding paranoia/robustness at the cost of
slight performance in some areas.

On Fri, 18 Nov 2005, Dormando wrote:

> Ok, all I had was speculation. This sounds more grounding.
>
> Now, you might know that I'm not a php haxx0r. You might say I barely
> know the first thing about the language; The fact that php seems to mix
> up some stuff per connection instead of per request was news to me. Our
> backend is also running a dozen different versions of php at the moment
> (which I'm working on fixing!), so it's possible there's a glitch or two
> involved here.
>
> I can reason the glitch happening sometimes and being related to PHP
> persisting some data either because of a glitch in an old version of PHP
> or because of a code bug in a backend I've never looked at myself. It
> doesn't feel like a widespread glitch though, given how few of our users
> are hit by it. If the first request to a keep-alive connection had all
> the login data for the rest of the requests, every one of our users
> would be logged out within a couple minutes.
>
> I'll go back to working with one of the programmers in hunting it down,
> but for the record; are there any other reasons why this might happen?
> Nothing related to connections blowing up or dying or getting confused?
> What happens when a bad content-length is fed back to perlbal on accident?
>
> (yes, we're checking X-Forwarded-For and such, I made sure of that at
> least).
>
> Thanks,
> -Alan
>
> Brad Fitzpatrick wrote:
>
>

Dormando

2005-11-18, 5:45 pm

I don't think so...

Our backend is just direct connections to php running behind either
apache1 or apache2. As long as the remote user's independent requests go
down and come back from the same server it should work. IE; If user A
sends a request down, it just needs to get the reply to that request
back. If user A sends a request down, but gets the response headers from
user B, we'd get what we're seeing now.

There's no reliance on the same user hitting the same exact backend
connection for multiple requests. Before we put the LB in there was no
keepalive at all, and users would hit one of 80 servers at random.

-Alan

Brad Fitzpatrick wrote:[vbcol=seagreen]
> Does your backend's session management assuming that for a given
> connection, the same user will always be on that connection? What backend
> do you use, btw?
>
> If so, that's your problem, since Perlbal mixes up requests and
> connections. (on purpose... it's a huge optimization)
>
> - Brad
>
>
> On Fri, 18 Nov 2005, Dormando wrote:
>
>

Dormando

2005-11-19, 2:45 am

Jacques Marneweck wrote:
> Brad Fitzpatrick wrote:
>
>
> I'll do a little digging with your theory ;)
>
> With php everything is stateless from my experience with php over the
> past 8 years. Each request is treated in a stateless manner but one can
> get session data based on the session cookie / session identifier
> specified as ?mysession=sessionid style of URL's or both depending on
> the scenario.
>
> I'm currently serving doc.php.net up from two servers with one perlbal
> instance running on one of the boxes without any issues atm. I' haven't
> upgraded to 1.39 yet. I'll most likely get round to doing the upgrade
> this weekend.
>
> Regards
> --jm

We couldn't find anything in our codebase that uses variables which
persist between requests on the same connection. We had a hard time
figuring out what can persist for the whole connection at all...

We spent some time opening a Keep-Alive connection to a test apache
server, then sending it requests with different user login cookies (or
none at all) each time. We weren't able to convince two versions of PHP
to send us back the wrong cookies.

Further, in any scenario we could think of our users would be getting
their logins swapped instantly, not as small of a percentage as we were
seeing. If a single keep-alive conncetion can service up to 500 requests
in our case, there'd be a lot of room for something to get poisoned and
a lot of requests to get poisoned.

At this point we're leaning toward the idea that one or two of our
ancient webservers is running an ancient broken php install that returns
bunkus data occasionally, and that the Content-Length injection the devs
had tried wasn't 100% perfect either. In a couple weeks our webserver
backend will be an array of shiney new debian servers (new OS, anyway).
We'll try it again then and send some updates.

have fun,
-Alan