IIS Server - Help interpreting DebugDiag reports

Author

Help interpreting DebugDiag reports

Sam Chrisp

2005-09-27, 2:52 am

Hi,

Below is a description of issues we have been experiencing with our
application running on IIS 6.
We have installed DebugDiag and done full userdumps of all the IIS
processes.

The Crash/Hang report is here: http://202.93.169.84/reports/CrashHang.html
And the Memory Pressure report is here:
http://202.93.169.84/MemoryPressure.html

The worker process which has been crashing is 'FooBAR' with a PID of 5012.

We can see some obvious areas of concern:
- Total reserved memory is 2.87GB, but only 148MB of that is committed.
- ADO components appear to be holding onto large amounts of memory

But cannot draw any proper conclusions from the reports.

Any comments or advice anyone can offer in interpreting these reports and
identifying the issue(s) would be appreciated.

==== Background
We have a web-based application which is predominantly ASP but with some
ASP.NET in areas.
SQL Server 2000 is used as the database backend.

For a few months now we have had issues with the IIS worker process which is
serving the application 'crashing'. Other worker processes running
webservices related to the application do not crash.

For a while this was occuring roughly once a week.
After a recent update to our application (which was quite large, and covered
many areas) it is now occuring on average once a day, and up to 3 times a
day.

Symptoms:
1. The system will slow down and take 3-4 minutes to load a page.
2. Page timeouts may occur.
3. VBScript compilation 'Out of Memory' errors will occur when trying to
load up any ASP page.
4. The worker process may crash and terminate (ref. IIS Errors below).

These issues have always been 'fixed' by an IIS reset.

==== Environment
Webserver config
Processor: 4 * 2.8GHz Xeons
Memory: 4GB DDR
Disk: 36GB U320 SCSI
Software: Windows 2003 w/SP1, .NET Framework 1.1, IIS 6 in Native mode.

Database server config
Shared SQL Instance
Processor: 4 * 2.8GHz Xeons
Memory: 4GB DDR
Disk: Fibre SAN
Software: Windows 2003 w/SP1, SQL Server 2000 w/SP3a.

===== IIS Errors
Event Type: Warning
Event Source: W3SVC
Event Category: None
Event ID: 1011
Date: 8/06/2005
Time: 4:08:27 p.m.
User: N/A
Computer: LNL11T01
Description:
A process serving application pool 'Foobar' suffered a fatal communication
error with the World Wide Web Publishing Service. The process id was '3508'.
The data field contains the error number.

For more information, see Help and Support Center at
http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 6d 00 07 80 m..?

===== Reports
http://202.93.169.84/reports/CrashHang.html
http://202.93.169.84/MemoryPressure.html

Wei-Dong XU [MSFT]

2005-09-27, 7:58 am

Hi Sam,

Based on my research on this issue, these information is very attractive:
Thread 49 - System ID 2504
Entry point msvcrt!_endthreadex+2f
Create time 27/09/2005 11:02:43 a.m.
Time spent in user mode 0 Days 0:1:7.515
Time spent in kernel mode 0 Days 0:0:11.671

Function Source
ntdll!KiFastSystemCallRet
ntdll!NtWaitForMultipleObjects+c
kernel32!WaitForMultipleObjectsEx+11a
user32!RealMsgWaitForMultipleObjectsEx+1
41
user32!MsgWaitForMultipleObjects+1f
comsvcs!CSTAThread::WorkerLoop+1e5
msvcrt!_endthreadex+a3
kernel32!BaseThreadStart+34

This means one/more ASP component(s) at your web application are calling
the COM+ componenet which is hang for a very long time. I am sure this is
not caused by Microsoft jet component and from the log, it appears several
sites are pointing to the default web site located at c:\inetpub\wwwroot.
Please have a checking on this to ensure only one site in IIS pointing at
the default web site.

Furthermore, for locating the problem component, please stop all other
sites and only run default web site. Then review the asp code to see which
component used in the web appliation hanging the execution.

Another way, please configure the recycle time of the default web site
worker process. This is one new feature of IIS6.0 to close the worker
process and start a new one. This way, we could free the occupied system
resource and avoid some high-memory using issues. This operation guide will
introduce the detailed step for you on the configuration:
Configuring Worker Processes for Recycling (IIS 6.0)
http://www.microsoft.com/technet/pr...3/Library/IIS/9
89a6b5c-289c-4a7c-95b5-175ee4c27159.mspx

Please feel free to let me know if you have any further question on this
matter.

Best Regards,
Wei-Dong XU
Microsoft Product Support Services
This posting is provided "AS IS" with no warranties, and confers no rights.
It is my pleasure to be of assistance.

Sam Chrisp

2005-09-27, 7:58 am

Hi Wei-Dong,

Thanks for your reply.

We would like to enable recycling, however if the worker process
recycles, doesn't this mean all users will lose their current session
information?

*** Sent via Developersdex http://www.codecomments.com ***

Wei-Dong XU [MSFT]

2005-09-28, 2:50 am

Hi Sam,

Yes, this mean all users will lose their current session. Because the
recycle indicates that the current worker process used for the site is
eliminiated, IIS will try to create one new worker process on the service.
So all the session data are all cleared. This may be one problem for your
scenario. So I think that is the best for you to find the root cause on the
slow response and remove it from the system. If it is very hidden, the
quickest and most convenient method is to raise one incident to your
Microsoft locat product support service. They will perform the debugging
for you and terminate that culprit. Your local MS PSS contact information
is available from:
http://support.microsoft.com/common/international.aspx

Please feel free to let me know if you have any further question on this
matter.

Best Regards,
Wei-Dong XU
Microsoft Product Support Services
This posting is provided "AS IS" with no warranties, and confers no rights.
It is my pleasure to be of assistance.