Hi all,

I've been doing some experimentation with proxies and load balancers
over the last couple of days after a site I work on
(peopleaggregator.net) had some performance problems due to spammers
occuping all the server's Apache child processes by connecting then
taking forever to send the payload for their POST requests.

I ended up using a custom proxy for it that I wrote some years back, but
was interested whether Perlbal handled this situation, as it generally
seems to be the load balancing swiss army knife of web 2.0, so I gave it
a go.

The good news: it buffered the POST requests fine, and the site didn't
have any trouble staying responsive while my test script created
hundreds of fake connections.

The bad news: after creating about 600 connections, Perlbal said "Out of
memory!" and exited.  It was using about 3G of virtual memory right
before it died, so I'm assuming it ran out of address space.

I'm not sure if this is just because I'm a Perlbal newbie and don't know
how to configure it properly, or if it's a generic Perlbal problem.
Does anyone have a site running Perlbal that they wouldn't mind me
running my script against, to see if I can reproduce the problem there?
Or if you'd like to try it on your own server in private, e-mail me for
a copy (I'd prefer not to encourage script kiddies by making it public,
even though it is a pretty trivial bit of code).

I'm running Perlbal 1.60 from Subversion on PERL 5.8.8 on Debian, kernel
2.6.17, under Colinux on a Vista laptop.  512M RAM, no swap.

Cheers,
Phil

[ Post a follow-up to this message ]

I'm glad it survived your slow POST attack, as that was a specific design
goal.  I'm not so glad about the OOM.  In any case.  Which kernel and
version of Perl?  Using XS HTTPHeaders or not?


On Tue, 1 Jan 2008, Phillip Pearson wrote:

> Hi all,
>
> I've been doing some experimentation with proxies and load balancers
> over the last couple of days after a site I work on
> (peopleaggregator.net) had some performance problems due to spammers
> occuping all the server's Apache child processes by connecting then
> taking forever to send the payload for their POST requests.
>
> I ended up using a custom proxy for it that I wrote some years back, but
> was interested whether Perlbal handled this situation, as it generally
> seems to be the load balancing swiss army knife of web 2.0, so I gave it
> a go.
>
> The good news: it buffered the POST requests fine, and the site didn't
> have any trouble staying responsive while my test script created
> hundreds of fake connections.
>
> The bad news: after creating about 600 connections, Perlbal said "Out of
> memory!" and exited.  It was using about 3G of virtual memory right
> before it died, so I'm assuming it ran out of address space.
>
> I'm not sure if this is just because I'm a Perlbal newbie and don't know
> how to configure it properly, or if it's a generic Perlbal problem.
> Does anyone have a site running Perlbal that they wouldn't mind me
> running my script against, to see if I can reproduce the problem there?
> Or if you'd like to try it on your own server in private, e-mail me for
> a copy (I'd prefer not to encourage script kiddies by making it public,
> even though it is a pretty trivial bit of code).
>
> I'm running Perlbal 1.60 from Subversion on PERL 5.8.8 on Debian, kernel
> 2.6.17, under Colinux on a Vista laptop.  512M RAM, no swap.
>
> Cheers,
> Phil
>

[ Post a follow-up to this message ]

He mentioned this,

On Dec 31, 2007 11:31 PM, Phillip Pearson <pp@myelin.co.nz> wrote:
> I'm running Perlbal 1.60 from Subversion on PERL 5.8.8 on Debian, kernel
> 2.6.17, under Colinux on a Vista laptop.  512M RAM, no swap.

Although I've never used Colinux, I'm not sure if there are any
gotchas because of how it works (haven't looked into it).  I assume
this is a dev/testing setup and not a deployment one? 

I can test it out on a 64-bit Ubuntu 7.10 machine with 4G RAM if
needed.  (I'd need the script I guess)

Brett

[ Post a follow-up to this message ]

Exactly... this is just my dev box, so it's quite possible that Perlbal
on a proper server would handle this quite easily.

Will send a copy of the script to you, Jonty and Brad in a minute so you
can try it out on real hardware 

Cheers,
Phil

Brett Hoerner wrote:
> He mentioned this,
>
> On Dec 31, 2007 11:31 PM, Phillip Pearson <pp@myelin.co.nz> wrote:
> 
>
> Although I've never used Colinux, I'm not sure if there are any
> gotchas because of how it works (haven't looked into it).  I assume
> this is a dev/testing setup and not a deployment one? 
>
> I can test it out on a 64-bit Ubuntu 7.10 machine with 4G RAM if
> needed.  (I'd need the script I guess)
>
> Brett
>

[ Post a follow-up to this message ]