Perlbal - crash with Perlbal-1.41

Author

crash with Perlbal-1.41

Cal Henderson

2006-05-30, 7:11 pm

hi,

i'm testing out perlbal-1.41 with the following config:

---------------------------------------------------------------
CREATE POOL my_apaches
POOL my_apaches ADD 127.0.0.1:81

CREATE SERVICE balancer
SET listen =3D 0.0.0.0:80
SET role =3D reverse_proxy
SET pool =3D my_apaches
SET persist_client =3D on
SET persist_backend =3D on
SET verify_backend =3D on
SET buffer_uploads =3D on
SET buffer_uploads_path =3D /export/home/tmp
ENABLE balancer

---------------------------------------------------------------

it tripped out after about 5 minutes of moderate load with this=20
console message:

---------------------------------------------------------------
beginning run
Undef client_ip (Perlbal::ClientProxy=3DARRAY(0xa1a5504)
) in=20
assign_client. Closing. at=20
/home/calh/perlbal/lib/perl5/site_perl/5.8.6/Perlbal/BackendHTTP.pm=20
line 179.
Undef client_ip (Perlbal::ClientProxy=3DARRAY(0xaefbe58)
) in=20
assign_client. Closing. at=20
/home/calh/perlbal/lib/perl5/site_perl/5.8.6/Perlbal/BackendHTTP.pm=20
line 179.
crash log: Can't use string ("=EF=BF=BD") as a SCALAR ref while "strict=20
refs" in use at=20
/home/calh/perlbal/lib/perl5/site_perl/5.8.6/Perlbal/ClientProxy.pm=20
line 923.
ending run
---------------------------------------------------------------

the subsequent attempt (still running as of writing this email) is=20
giving out periodic error messages about 'Undef client_ip' roughly=20
once every 2 minutes / 5000 requests. host is load balanced, with no=20
DSR and no NAT'ing.

any ideas?

thanks,

--cal

my PERL looks like this:

$ /usr/local/bin/perl -V
Summary of my perl5 (revision 5 version 8 subversion 6) configuration:
Platform:
osname=3Dlinux, osvers=3D2.6.9-5.elsmp,=20
archname=3Di686-linux-thread-multi
uname=3D'linux deadrat4.inktomisearch.com 2.6.9-5.elsmp #1 smp wed=20
jan 5 19:30:39 est 2005 i686 i686 i386 gnulinux '
config_args=3D'-Duselargefiles -de -Dperladmin=3Dprod-eng@yahoo-inc.c=
om=20
-Dusethreads=3Dyes -A=20
define:optimize=3D-O2 -march=3Dpentium3 -fprefetch-loop-arrays -funroll-l=
oops=20
-pipe -A define:cc=3Dgcc -A define:ld=3Dgcc'
hint=3Drecommended, useposix=3Dtrue, d_sigaction=3Ddefine
usethreads=3Ddefine use5005threads=3Dundef useithreads=3Ddefine=20
usemultiplicity=3Ddefine
useperlio=3Ddefine d_sfio=3Dundef uselargefiles=3Ddefine usesocks=3Du=
ndef
use64bitint=3Dundef use64bitall=3Dundef uselongdouble=3Dundef
usemymalloc=3Dn, bincompat5005=3Dundef
Compiler:
cc=3D'gcc', ccflags=20
=3D'-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -fno-strict-aliasing =20
-pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=3D64',
optimize=3D'-O2 -march=3Dpentium3 -fprefetch-loop-arrays -funroll-loo=
ps=20
-pipe',
cppflags=3D'-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -fno-stric=
t-aliasing=20
-pipe -I/usr/local/include'
ccversion=3D'', gccversion=3D'3.4.3 20041212 (Red Hat 3.4.3-9.EL4)',=20
gccosandvers=3D''
intsize=3D4, longsize=3D4, ptrsize=3D4, doublesize=3D8, byteorder=3D1=
234
d_longlong=3Ddefine, longlongsize=3D8, d_longdbl=3Ddefine,=20
longdblsize=3D12
ivtype=3D'long', ivsize=3D4, nvtype=3D'double', nvsize=3D8, Off_t=3D'=
off_t',=20
lseeksize=3D8
alignbytes=3D4, prototype=3Ddefine
Linker and Libraries:
ld=3D'gcc', ldflags =3D' -L/usr/local/lib'
libpth=3D/usr/local/lib /lib /usr/lib
libs=3D-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
perllibs=3D-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
libc=3D/lib/libc-2.3.4.so, so=3Dso, useshrplib=3Dfalse,=20
libperl=3Dlibperl.a
gnulibc_version=3D'2.3.4'
Dynamic Linking:
dlsrc=3Ddl_dlopen.xs, dlext=3Dso, d_dlsymun=3Dundef, ccdlflags=3D'-Wl=
,-E'
cccdlflags=3D'-fpic', lddlflags=3D'-shared -L/usr/local/lib'

Characteristics of this binary (from libperl):
Compile-time options: MULTIPLICITY USE_ITHREADS USE_LARGE_FILES=20
PERL_IMPLICIT_CONTEXT
Built under linux
Compiled at Jul 29 2005 11:21:08
@INC:
/usr/local/lib/perl5/5.8.6/i686-linux-thread-multi
/usr/local/lib/perl5/5.8.6
/usr/local/lib/perl5/site_perl/5.8.6/i686-linux-thread-multi
/usr/local/lib/perl5/site_perl/5.8.6
/usr/local/lib/perl5/site_perl

Jay Buffington

2006-05-31, 1:11 am

I've never tried to bind anything to 0.0.0.0. Maybe that's the source
of the problem?

I would have the balancer listen on 128.206.9.125:80 (or whatever your
external IP is) and the apaches listen on 127.0.0.1:80.

Jay

On 5/30/06, Cal Henderson <calh@yahoo-inc.com> wrote:
> CREATE POOL my_apaches
> POOL my_apaches ADD 127.0.0.1:81
>
> CREATE SERVICE balancer
> SET listen = 0.0.0.0:80
> SET role = reverse_proxy
> SET pool = my_apaches
> SET persist_client = on
> SET persist_backend = on
> SET verify_backend = on
> SET buffer_uploads = on
> SET buffer_uploads_path = /export/home/tmp
> ENABLE balancer
>

dormando

2006-06-02, 7:11 am

Hey,

> ---------------------------------------------------------------
> beginning run
> Undef client_ip (Perlbal::ClientProxy=3DARRAY(0xa1a5504)
) in=20
> assign_client. Closing. at=20
> /home/calh/perlbal/lib/perl5/site_perl/5.8.6/Perlbal/BackendHTTP.pm=20
> line 179.
> Undef client_ip (Perlbal::ClientProxy=3DARRAY(0xaefbe58)
) in=20
> assign_client. Closing. at=20
> /home/calh/perlbal/lib/perl5/site_perl/5.8.6/Perlbal/BackendHTTP.pm=20
> line 179.
> crash log: Can't use string ("=EF=BF=BD") as a SCALAR ref while "strict=
=20
> refs" in use at=20
> /home/calh/perlbal/lib/perl5/site_perl/5.8.6/Perlbal/ClientProxy.pm=20
> line 923.
> ending run

I started getting this a ton depending on client connectios being weird.=20
Haven't had a moment to track down exactly why it's happening, but=20
there's a trivial fix for the problem. Your error's slightly different=20
though. In my case an empty IP address was being passed along, but in=20
your case it looks like 3 bytes of garbage. Just a case of an unhandled=20
error in the source code.

A badly formed trivial patch is as follows (and works, my perlbal=20
instances have been up for months at this point):

--- BackendHTTP.pm.1.39 2006-02-28 20:55:13.634135000 -0800
+++ BackendHTTP.pm 2006-02-28 20:52:59.805445000 -0800
@@ -70,7 +70,14 @@
}

IO::Handle::blocking($sock, 0);
- connect $sock, Socket::sockaddr_in($port, Socket::inet_aton($ip));
+
+ # Sometimes this dies. Unsure of the circumstances.
+ eval {
+ connect $sock, Socket::sockaddr_in($port, Socket::inet_aton($ip)=
);
+ };
+ if ($@) { warn "Socket connect failed (probably due to sockaddr_in)=20
$@";
+ return undef;
+ }

my $self =3D fields::new($class);
$self->SUPER::new($sock);