| jkiss@sandorlabs.com 2008-01-03, 1:19 pm |
| Hello, I'm trying to solve a problem we're having where some of our
long-
running streaming CGI scripts are causing Apache to grow
continuously,
eventually consuming all system memory and starving Apache and other
processes
(manifested as Apache returning HTTP 500 (internal server error)
responses to
subsequent requests to other resources, and other weirdness). I've
reduced the
test to a really simple CGI script which reproduces the memory growth
reliably
for me (see steps to reproduce below).
I'm wondering if this is normal behaviour (is continuous streaming
not
supported?), or perhaps I'm just mangling my headers or missing
something
really obvious. Any insight you guys have would be great. If it's a
bug in
Apache, my apologies for darkening your door at Christmastime. ;-)
I've observed the behaviour on both of our setups:
Linux: 2.6.13.4.D400-LEAF (a custom (i.e. suspect) roll-our-own Linux
- gcc
version 4.0.2 20051125, Red Hat 4.0.2-8).
Apache: 2.0.55
Linux: Ubuntu Fiesty (standard, nothing special)
Apache: 2.2.3
Steps to reproduce:
-------------------
1. Restart Apache.
2. Point a browser at the CGI script (see offending cgi script (1)
below)
running either runPlain() or runMultipart().
3. Run ps every 10 seconds for 10 minutes (see ps log script (2)
below).
My observations:
-----------------
1. In both the plain/text and multipart case, I observed Apache's
memory
growing linearly at a rate of ~19.6kB/min (typically in 4kB
increments
depending on your Linux/MMU page size).
2. When I stopped the script after 5 minutes, Apache did not appear to
free its
memory. Then, when I restarted the script, Apache ran flat for 5
minutes before
starting to grab more memory. This suggests that when a CGI script
stops,
Apache frees some of its internal memory pool, but does not release it
to
system memory, instead reusing it for future CGI scripts. I found I
had to
restart Apache after each test to avoid masking the memory growth of
subsequent
tests.
3. I tried various data rates in my below scripts, by adjusting the
sleep time,
chars per line and lines per cycle, and found the memory growth rate
was
roughly a linear function of the CGI's data output rate (with the
function
flattening at high data rates probably due to CPU/network saturation).
4. I tried various other stabs in the dark, like flushing stdout on
each cycle
(before the sleep). No change to the growth rate.
1. Offending cgi script:
------------------------
#include <time.h>
#include <dirent.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/time.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <unistd.h>
/*
main.
*/
int main(int argc, char *argv[])
{
runPlain();
}
/*
Run a text/plain test.
*/
int runPlain()
{
int i,j;
printf("Content-type: text/plain; charset=us-ascii\n\n");
for (i=0; ; i++)
{
for (j=0; j<5; j++)
{
printf("%d,%d: wakka wakka wakka wakka wakka wakka wakka
\n",i,j);
}
printf("\n");
usleep(5000);
}
}
/*
Run a multipart/mixed-replace test
per http://www.w3.org/Protocols/rfc1341/7_2_Multipart.html section
7.2.1.
*/
int runMultipart()
{
int i,j,num;
printf("Content-type: multipart/mixed-replace; boundary=myboundary
\n\n");
printf("Preamble\n");
for (i=0; ; i++)
{
printf("--myboundary\n");
printf("Content-type: text/plain; charset=us-ascii\n\n");
for (j=0; j<5; j++)
{
printf("%d,%d: wakka wakka wakka wakka wakka wakka wakka
\n",i,j);
}
printf("\n");
usleep(5000);
}
}
2. ps log:
----------
#!/bin/sh
interval=$1
while :
do
timestamp=`date +%Y.%m.%d-%H.%M.%S`
ps -o pid,pcpu,mem,cmd | sed "s/^/$timestamp /"
sleep $interval
done
|