HostOnNet.com

Annie · Site Admin Joined: 19 Jun 2005 Posts: 23

Often the most effective way to diagnose the root cause of an IIS Crash or Hang situation is to use a debugging tool that allows you to look inside the various processes that IIS uses (such as dllhost.exe or w3wp.exe). Whilst diagnostic tools like AuthDiag and SSLDiag can help solve configuration issues which are preventing your IIS Server from running properly, a crash or hang usually is caused by faulty code, hardware issues, or problems connecting to remote resources (e.g. a remote database server), and these problems may not deterministically manifest themselves.

The two most common debugging tools for IIS are IISState and ADPlus. ADPlus is available as part of the Windows Debugging Toolkit. IISState was released with the IIS6 Resource Kit Tools, however an updated version is available for download from the IISFaq.com website. There are two downloads - the plain IISState executable, and an MSI installer than includes the necessary debugging components.

If you download the bare executable file (contained in the .zip file), you will need to manually copy across some components from the Windows Deugging Toolkit. In particular you need dbgeng.dll (the Windows Symbolic Debugger Engine) and a number of helper components.

Running IISState will also create two folders: Output and Symbols. Output is where IISState will store its logfiles. The symbols folder is where you will need to place symbols (if you have them) for 3rd party or custom code, and for Windows dlls. If you don't have the Windows symbols, IISState has the ability to connect to the Microsoft Symbol Server and download symbols. However this does require that your IIS server have outbound connectivity to port 80 (HTTP) on Microsoft's Symbol Server, which may not be possible in some production environments.

IISState will download symbols from the Microsoft Symbol Server the first time you run IISState and attach it to a running IIS process. To attach IISState to a running process you require the process' PID (Process ID). You can get this from Task Manager

In IIS 6.0 you will want to attach to InetInfo.exe if IIS itself is crashing, or one of the services that run inside InetInfo (SMTP, NNTP, FTP) is crashing. For web applications, you will want to attach to the relevant w3wp.exe process if you are running in Worker Process Isolation Mode (the native IIS 6.0 mode), or dllhost.exe (if you are running in IIS 5.0 Emulation Mode, and the site is in "medium (pooled)" or "high" isolation)

To determine which Web Application Pool defined in IIS is currently being serviced by which w3wp.exe process you can run iisapp.vbs located in %systemroot%\system32\

To attach IISState to a particular w3wp.exe process execute the following command (where <PID> is the Process ID). This will do an immediate dump of the current process.

iisstate -p <PID>

IISState also supports the following optional switches:

-sc (waits for a "soft crash" such as an ASP 0115 Trappable Error Occured in an External Object)
-hc (waits for a "hard crash" where the process terminates unexpectedly)
-d (write out a dump file, which can be used for further analysis, e.g. by WinDBG)

IISState outputs a logfile containing the stacks of all the threads in the process. This logfile can be fairly lengthy, depending on how much IIS is going when the crash/hang occurs. This is an example of such a logfile, culled from the IIS public newsgroup. Analysing the log requires some knowledge of how Windows works (usually), I won't pretend to know anything about that (at this stage), but a little educated guesswork, use of the Platform SDK and copious reading of Microsoft Windows Internals can give you a pretty good idea of what's going on. I hope to have some analysis pages up soon. But in the mean time, you can have a look at this analysis of a single thread on my blog.

Be aware that IISState attaches invasively to processes, and exiting IISState manually will cause the attached process to terminate. That's usually not a problem if the process is terminating anyway because it's crashed, or isn't responsive due to a "hang" scenario.