| John Berthels 2007-09-04, 1:12 pm |
| Hi folks,
We're serving quite big files (many in the hundreds of megabyte range,
some >1Gig) and under load perlbal would suddenly shoot up in memory
(but only if we had reproxying enabled).
It looks as though the backend was effectively dumping the file
contents into the write buffer of the ClientProxy and the following
patch has seemingly cured the problem for us.
This may be affecting many people, but you'll see much less of an
effect with smaller file sizes (both because the memory increase is
less and because the client can fetch them and free the ClientProxy
more quickly, so the impact probably goes up as the square of the file
size or similar).
Given that the patch appears to work, it is clear that the client is
losing it's backend somehow, but I've not yet determined how.
I'd obviously prefer to find the root cause for this, but I can't
reproduce the problem except under live load and I don't really want
to do much experimenting there because each perlbal restart can reset
a bunch of long-lived client connections.
Is the patch below suitable for committing? It makes
'too_far_behind_backend' more robust, imho.
regards,
jb
========================================
===========================
--- lib/Perlbal/ClientProxy.pm (revision 697)
+++ lib/Perlbal/ClientProxy.pm (working copy)
@@ -192,7 +192,7 @@
# returns true if this ClientProxy is too many bytes behind the backend
sub too_far_behind_backend {
my Perlbal::ClientProxy $self = $_[0];
- my Perlbal::BackendHTTP $backend = $self->{backend} or return 0;
+ my Perlbal::BackendHTTP $backend = $self->{backend};
# if a backend doesn't have a service, it's a
# ReproxyManager-created backend, and thus it should use the
@@ -202,7 +202,7 @@
# being that reproxied-to webservers are event-based and it's okay
# to tie the up longer in favor of using less buffer memory in
# perlbal)
- my $max_buffer = defined $backend->{service} ?
+ my $max_buffer = ($backend && defined $backend->{service}) ?
$self->{service}->{buffer_size} :
$self->{service}->{buffer_size_reproxy_url};
|