|
Home > Archive > Macromedia Flash Server > April 2005 > Server crashing - no messages or clues
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
Server crashing - no messages or clues
|
|
| Kevin Day 2005-04-07, 5:56 pm |
|
I'm having a problem where after 2-16 hours under heavy load our
FlashComm server just crashes.
Nothing happens on the console when it crashes. I've got trace()
commands everywhere on the SSAS to see if it was processing something
in particular in my code when it crashes, but it seems random.
I've lowered the FLV cache to 10, reduced IO threads, and haven't
helped any. A script to monitor fcserver's memory use every 10 seconds
doesn't show any memory spikes before it crashes, so I don't think it's
running out of memory. Sometimes it crashes with 500MB resident,
sometimes 700MB... It doesn't look like an out of memory issue.
I've also only got around 300 connections active, so I don't think it's
an issue with too many connections.
Has anyone run into this before?
Are there different libraries (librt, libpthread, libc, etc) or
different kernels that have caused problems before?
=-----------------------------------------------------------
Supported by Fig Leaf Software - http://www.figleaf.com
=-----------------------------------------------------------
To change your subscription options or search the archive:
http://chattyfig.figleaf.com/mailman/listinfo/flashcomm
| |
| Simon Lord 2005-04-07, 5:56 pm |
| First step, check your RAM. This was happening to us on our server
until we swapped out the ram for newer ram - problem went away
instantly. We were crashing on daily intervals as well. It was almost
predictable.
Second, make sure you're running 1.5.2
My 2 cents.
On Mar 25, 2005, at 6:32 PM, Kevin Day wrote:
>
>
> I'm having a problem where after 2-16 hours under heavy load our
> FlashComm server just crashes.
>
> Nothing happens on the console when it crashes. I've got trace()
> commands everywhere on the SSAS to see if it was processing something
> in particular in my code when it crashes, but it seems random.
>
> I've lowered the FLV cache to 10, reduced IO threads, and haven't
> helped any. A script to monitor fcserver's memory use every 10 seconds
> doesn't show any memory spikes before it crashes, so I don't think
> it's running out of memory. Sometimes it crashes with 500MB resident,
> sometimes 700MB... It doesn't look like an out of memory issue.
>
> I've also only got around 300 connections active, so I don't think
> it's an issue with too many connections.
>
>
> Has anyone run into this before?
>
> Are there different libraries (librt, libpthread, libc, etc) or
> different kernels that have caused problems before?
>
>
> =-----------------------------------------------------------
> Supported by Fig Leaf Software - http://www.figleaf.com
> =-----------------------------------------------------------
>
> To change your subscription options or search the archive:
> http://chattyfig.figleaf.com/mailman/listinfo/flashcomm
>
>
Sincerely,
Simon
=-----------------------------------------------------------
Supported by Fig Leaf Software - http://www.figleaf.com
=-----------------------------------------------------------
To change your subscription options or search the archive:
http://chattyfig.figleaf.com/mailman/listinfo/flashcomm
| |
| Kevin Day 2005-04-07, 5:56 pm |
|
On Mar 25, 2005, at 6:28 PM, Simon Lord wrote:
> First step, check your RAM. This was happening to us on our server
> until we swapped out the ram for newer ram - problem went away
> instantly. We were crashing on daily intervals as well. It was
> almost predictable.
>
> Second, make sure you're running 1.5.2
>
> My 2 cents.
>
>
>
Yeah, getting consistent crashes on two separate servers. The ram in
them is also ECC, and I'm not getting any bit errors on it.
Also running 1.5.2... Thanks for the ideas though!
=-----------------------------------------------------------
Supported by Fig Leaf Software - http://www.figleaf.com
=-----------------------------------------------------------
To change your subscription options or search the archive:
http://chattyfig.figleaf.com/mailman/listinfo/flashcomm
| |
| Simon Lord 2005-04-07, 5:56 pm |
| No problem. And I doubt it's your ram if two machines are crashing,
but I just wanted to add that my host did bit tests for me on two
occasions within the span of a week and came up empty. As far as they
were concerned it wasn't the ram. But the minute they changed it the
machine never went down again. Been solid for nearly a year now.
>
> Yeah, getting consistent crashes on two separate servers. The ram in
> them is also ECC, and I'm not getting any bit errors on it.
>
> Also running 1.5.2... Thanks for the ideas though!
>
>
> =-----------------------------------------------------------
> Supported by Fig Leaf Software - http://www.figleaf.com
> =-----------------------------------------------------------
>
> To change your subscription options or search the archive:
> http://chattyfig.figleaf.com/mailman/listinfo/flashcomm
>
>
Sincerely,
Simon
=-----------------------------------------------------------
Supported by Fig Leaf Software - http://www.figleaf.com
=-----------------------------------------------------------
To change your subscription options or search the archive:
http://chattyfig.figleaf.com/mailman/listinfo/flashcomm
| |
| Kevin Day 2005-04-07, 5:56 pm |
|
On Mar 25, 2005, at 5:32 PM, Kevin Day wrote:
>
>
> I'm having a problem where after 2-16 hours under heavy load our
> FlashComm server just crashes.
>
> Nothing happens on the console when it crashes. I've got trace()
> commands everywhere on the SSAS to see if it was processing something
> in particular in my code when it crashes, but it seems random.
>
>
Ok, further troubleshooting has narrowed it down to this...
If I turn off "EnhancedSeek" the crashing goes away.
If I turn it back on and add lots of debugging, I can find that in
every case that it did crash, a client had recently requested a seek to
one specific location in the file. Out of 12 crashes, all 12 had a seek
to that position within the last second before death.
I've recompressed the video with Squeeze three times and ended up with
the same output file, so I don't think it's a case of a random
corrupted .flv. Playing the entire file back shows no graphical
glitches or audio dropouts or anything. However, if I seek to that
exact location, I get a rather corrupted looking bit of video until the
next keyframe. It doesn't crash the server every time, but apparently
after enough clients seek there it does.
Has anyone run into THIS before?
=-----------------------------------------------------------
Supported by Fig Leaf Software - http://www.figleaf.com
=-----------------------------------------------------------
To change your subscription options or search the archive:
http://chattyfig.figleaf.com/mailman/listinfo/flashcomm
| |
| Edward Chan 2005-04-07, 5:56 pm |
| That is great that you have narrowed it down to enhanced seeking. Can you
give us the flv and a simple test app to produce this so we can investigate
further?
Thanks,
Ed
> -----Original Message-----
> From: flashcomm-bounces-1Ss2GqJETD3yZ38Mhd3e/9ZfFG6BLHNm@public.gmane.org
> [mailto:flashcomm-bounces-1Ss2GqJETD3yZ38Mhd3e/9ZfFG6BLHNm@public.gmane.org]
> Sent: Sunday, March 27, 2005 3:43 PM
> To: FlashComm Mailing List
> Subject: Re: [FlashComm] Server crashing - no messages or clues
>
>
> On Mar 25, 2005, at 5:32 PM, Kevin Day wrote:
>
> something
>
> Ok, further troubleshooting has narrowed it down to this...
>
> If I turn off "EnhancedSeek" the crashing goes away.
>
> If I turn it back on and add lots of debugging, I can find
> that in every case that it did crash, a client had recently
> requested a seek to one specific location in the file. Out of
> 12 crashes, all 12 had a seek to that position within the
> last second before death.
>
> I've recompressed the video with Squeeze three times and
> ended up with the same output file, so I don't think it's a
> case of a random corrupted .flv. Playing the entire file
> back shows no graphical glitches or audio dropouts or
> anything. However, if I seek to that exact location, I get a
> rather corrupted looking bit of video until the next
> keyframe. It doesn't crash the server every time, but
> apparently after enough clients seek there it does.
>
>
> Has anyone run into THIS before?
>
>
> =-----------------------------------------------------------
> Supported by Fig Leaf Software - http://www.figleaf.com
> =-----------------------------------------------------------
>
> To change your subscription options or search the archive:
> http://chattyfig.figleaf.com/mailman/listinfo/flashcomm
>
=-----------------------------------------------------------
Supported by Fig Leaf Software - http://www.figleaf.com
=-----------------------------------------------------------
To change your subscription options or search the archive:
http://chattyfig.figleaf.com/mailman/listinfo/flashcomm
|
|
|
|
|