Perlbal - perlbal crashes

Author

perlbal crashes

Kevin Lewandowski

2005-08-14, 2:45 am

Hello,

I'm running Perlbal in reverse proxy mode and it is crashing quite
frequently. I get the crash message:

crash log: Can't call method "request_method" on an undefined value
at /usr/local/share/perl/5.8.4/Perlbal/HTTPHeaders.pm line 367.

My Perlbal config is:

--
CREATE POOL apache
POOL apache ADD 10.10.10.23:81

CREATE SERVICE proxy
SET proxy.listen = 10.10.10.23:80
SET proxy.role = reverse_proxy
SET proxy.pool = apache
SET proxy.persist_client = on
SET proxy.persist_backend = on
SET proxy.verify_backend = on
ENABLE proxy
--

This Perlbal is feeding Apache behind it. I looked into the code and
it has something to do with Keepalive. The crash happens after a few
hours when keepalive is disabled in apache, and almost immediately
when keepalive is enabled in Apache.

Any ideas on what is wrong? thanks

Kevin

Brad Fitzpatrick

2005-08-14, 2:45 am

What version? 1.2, 1.3, cvs?

I'm pretty sure that bug was both introduced and fixed in cvs. If it's
present in an actual release, uh, that'd suck. You might try updating to
the latest cvs? cvs should be stable right now, except for
buffered-to-disk uploads which are undergoing some refactoring right
now.

- Brad

On Sat, 13 Aug 2005, Kevin Lewandowski wrote:

> Hello,
>
> I'm running Perlbal in reverse proxy mode and it is crashing quite
> frequently. I get the crash message:
>
> crash log: Can't call method "request_method" on an undefined value
> at /usr/local/share/perl/5.8.4/Perlbal/HTTPHeaders.pm line 367.
>
> My Perlbal config is:
>
> --
> CREATE POOL apache
> POOL apache ADD 10.10.10.23:81
>
> CREATE SERVICE proxy
> SET proxy.listen = 10.10.10.23:80
> SET proxy.role = reverse_proxy
> SET proxy.pool = apache
> SET proxy.persist_client = on
> SET proxy.persist_backend = on
> SET proxy.verify_backend = on
> ENABLE proxy
> --
>
> This Perlbal is feeding Apache behind it. I looked into the code and
> it has something to do with Keepalive. The crash happens after a few
> hours when keepalive is disabled in apache, and almost immediately
> when keepalive is enabled in Apache.
>
> Any ideas on what is wrong? thanks
>
> Kevin
>
>

Kevin Lewandowski

2005-08-14, 2:45 am

Okay, I was running cvs from about a week ago. Tried 1.3 and had the
same problem. Now I'm running the latest cvs and there is no crash so
far but I see this message every few seconds:

Use of uninitialized value in numeric lt (< ) at /usr/local/share/perl/
5.8.4/Perlbal/ClientProxy.pm line 487.
Use of uninitialized value in sysread at /usr/local/share/perl/5.8.4/
Danga/Socket.pm line 981.
Use of uninitialized value in numeric lt (< ) at /usr/local/share/perl/
5.8.4/Perlbal/ClientProxy.pm line 487.
Use of uninitialized value in sysread at /usr/local/share/perl/5.8.4/
Danga/Socket.pm line 981.

Kevin

On Aug 13, 2005, at 7:17 PM, Brad Fitzpatrick wrote:

> What version? 1.2, 1.3, cvs?
>
> I'm pretty sure that bug was both introduced and fixed in cvs. If
> it's
> present in an actual release, uh, that'd suck. You might try
> updating to
> the latest cvs? cvs should be stable right now, except for
> buffered-to-disk uploads which are undergoing some refactoring right
> now.
>
> - Brad
>
> On Sat, 13 Aug 2005, Kevin Lewandowski wrote:
>
>
>
>

Kevin Lewandowski

2005-08-14, 2:45 am

Update on this:

now perlbal crashes after a few minutes with out of memory messages:

Out of Memory: Killed process 24339 (mogstored).
Out of Memory: Killed process 24338 (mogstored).
Out of Memory: Killed process 24340 (mogstored).
Out of Memory: Killed process 24341 (mogstored).

(note, I'm running mogstored but modified the config to add a
revserse proxy. That is what's causing the crash).

On Aug 13, 2005, at 9:05 PM, Kevin Lewandowski wrote:

> Okay, I was running cvs from about a week ago. Tried 1.3 and had
> the same problem. Now I'm running the latest cvs and there is no
> crash so far but I see this message every few seconds:
>
> Use of uninitialized value in numeric lt (< ) at /usr/local/share/
> perl/5.8.4/Perlbal/ClientProxy.pm line 487.
> Use of uninitialized value in sysread at /usr/local/share/perl/
> 5.8.4/Danga/Socket.pm line 981.
> Use of uninitialized value in numeric lt (< ) at /usr/local/share/
> perl/5.8.4/Perlbal/ClientProxy.pm line 487.
> Use of uninitialized value in sysread at /usr/local/share/perl/
> 5.8.4/Danga/Socket.pm line 981.
>
> Kevin
>
> On Aug 13, 2005, at 7:17 PM, Brad Fitzpatrick wrote:
>
>
>
>
>

Brad Fitzpatrick

2005-08-14, 2:45 am

To be honest, we haven't used Perlbal in production since version 1.2 and
until we upgrade to the latest version (soon here), I can't really promise
the quality of it.

However, the past few weeks have seen a massive amount of clean-up,
refactoring, strictness, features, and added tests/documentation. So the
end result should be way better than version 1.2 but you picked up cvs
right in the middle of all this work. It's unfortunate that bug you saw
was in 1.3. I'll try to get a new release out here soon.

All that said, I just commited a massive overhaul/cleanup of
ClientProxy.pm about 2 minutes ago that fixes several recently-introduced
bugs. (and also includes a bunch more tests, as always....)

So let me know what you think of current cvs. I'll be gone offline until
Sunday night.

- Brad

On Sat, 13 Aug 2005, Kevin Lewandowski wrote:

> Okay, I was running cvs from about a week ago. Tried 1.3 and had the
> same problem. Now I'm running the latest cvs and there is no crash so
> far but I see this message every few seconds:
>
> Use of uninitialized value in numeric lt (< ) at /usr/local/share/perl/
> 5.8.4/Perlbal/ClientProxy.pm line 487.
> Use of uninitialized value in sysread at /usr/local/share/perl/5.8.4/
> Danga/Socket.pm line 981.
> Use of uninitialized value in numeric lt (< ) at /usr/local/share/perl/
> 5.8.4/Perlbal/ClientProxy.pm line 487.
> Use of uninitialized value in sysread at /usr/local/share/perl/5.8.4/
> Danga/Socket.pm line 981.
>
> Kevin
>
> On Aug 13, 2005, at 7:17 PM, Brad Fitzpatrick wrote:
>
>
>

Brad Fitzpatrick

2005-08-14, 2:45 am

Again,

You're kinda on your own if you're following cvs. Did you also notice
100% CPU usage during the memory usage? I remember fixing a CPU spin
recently (but only related to buffered uploads) and maybe that CPU spin
was also a memory leak and I just killed it before it was a problem.

All I can say is try tonight's cvs. Things have been changing incredibly
quickly the past week.

Sorry about all this.... I won't be making another official release of
Perlbal until we run it in production for a couple days ourselves.
Releasing 1.3 without field testing was my mistake.

- Brad

On Sat, 13 Aug 2005, Kevin Lewandowski wrote:

> Update on this:
>
> now perlbal crashes after a few minutes with out of memory messages:
>
> Out of Memory: Killed process 24339 (mogstored).
> Out of Memory: Killed process 24338 (mogstored).
> Out of Memory: Killed process 24340 (mogstored).
> Out of Memory: Killed process 24341 (mogstored).
>
> (note, I'm running mogstored but modified the config to add a
> revserse proxy. That is what's causing the crash).
>
> On Aug 13, 2005, at 9:05 PM, Kevin Lewandowski wrote:
>
>
>

Brad Fitzpatrick

2005-08-14, 7:45 am

Things I remembered after I sent this email:

-- you must use PERL 5.8. PERL 5.6 is known to leak memory like crazy.

-- the latest cvs version of Perlbal requires the latest Danga::Socket,
also in cvs. Without it, it'll kinda work, but you'll get weird
errors/crashes/warnings.

-- I consider any PERL warning a bug. Please report.

On Sat, 13 Aug 2005, Brad Fitzpatrick wrote:

> Again,
>
> You're kinda on your own if you're following cvs. Did you also notice
> 100% CPU usage during the memory usage? I remember fixing a CPU spin
> recently (but only related to buffered uploads) and maybe that CPU spin
> was also a memory leak and I just killed it before it was a problem.
>
> All I can say is try tonight's cvs. Things have been changing incredibly
> quickly the past week.
>
> Sorry about all this.... I won't be making another official release of
> Perlbal until we run it in production for a couple days ourselves.
> Releasing 1.3 without field testing was my mistake.
>
> - Brad
>
>
> On Sat, 13 Aug 2005, Kevin Lewandowski wrote:
>
>
>

Kevin Lewandowski

2005-08-14, 8:45 pm

Thanks for your help Brad. It's looking better now. btw, it did get
100% cpu usage before crashing. And I had been running PERL 5.8 and
the latest Danga::Socket from cvs. This morning I upgraded perlbal to
the latest cvs and it's been running for ~ 6 hours with no warnings
or crashes.

thanks

Kevin

On Aug 14, 2005, at 4:29 AM, Brad Fitzpatrick wrote:

> Things I remembered after I sent this email:
>
> -- you must use PERL 5.8. PERL 5.6 is known to leak memory like
> crazy.
>
> -- the latest cvs version of Perlbal requires the latest
> Danga::Socket,
> also in cvs. Without it, it'll kinda work, but you'll get weird
> errors/crashes/warnings.
>
> -- I consider any PERL warning a bug. Please report.
>
>
>
> On Sat, 13 Aug 2005, Brad Fitzpatrick wrote:
>
>
>
>

Brad Fitzpatrick

2005-08-15, 5:45 pm

There should be no warnings or crashes, ever.

For now, everybody should stop using Perlbal 1.3. It's apparently screwed
in more ways than one. I was probably overconfident in its then-fledgling
test suite when I released it. Future releases will go back to 1.2's
quality.

- Brad

On Mon, 15 Aug 2005, Jacques Marneweck wrote:

> Brad Fitzpatrick wrote:
>
> That bug is in 1.3.1 from the site, I got bit by that one as well.
>
> Also is it normal for perlbal to crash when connection is reset by peer?
>
> prometheus# tail -10 current | tai64nlocal
> 2005-08-11 10:13:49.457163500 WARNING: AIO mode disabled or not available.
> 2005-08-11 10:13:49.457168500 Perlbal will run slowly under
> load if you're doing any
> 2005-08-11 10:13:49.457170500 disk operations. (e.g.
> web_server mode).
> 2005-08-11 23:31:36.978209500 Running.
> 2005-08-11 23:31:36.978214500 beginning run
> 2005-08-11 23:31:36.978216500 crash log: setsockopt: Connection reset by
> peer at /usr/local/lib/perl5/site_perl/5.8.7/Danga/Socket.pm line 641.
> 2005-08-11 23:31:36.978218500 ending run
> 2005-08-11 23:31:37.453415500 WARNING: AIO mode disabled or not available.
> 2005-08-11 23:31:37.453420500 Perlbal will run slowly under
> load if you're doing any
> 2005-08-11 23:31:37.453422500 disk operations. (e.g.
> web_server mode).
>
> Regards
> --jm
>
>
>
> --
> Jacques Marneweck
> http://www.powertrip.co.za/blog/
>
>
>