Task

Commands/Approach

What to look for at this stage

Get a dump of wspsrv.exe process while the issue is happening

Use the approach from this post

• Make sure that you also collect Perfmon during the time that the issue is not happening until the time that the issue happens.
• Use the counters from this post (item 2)

Load the wspsrv.exe dump and quick review all the threads within this process

On WinDBG type:

~kb*

• Look for patterns
• Check if there are multiple threads using similar stack.
• Take notes of your findings.

Check for critical sections

On WinDBG type:

!cs –l

• Check if there are locked threads.

Dump the information related to the critical section that is locked

On WinDBG type:

!cs -o <owning thread>

• Compare the threads that are in critical section with the ones that appeared as result of ~kb* command.
• Do we have a pattern here?
• Do we have a suspicious activity in one of those threads?

Task

Commands/Approach

What to look for at this stage

If critical section doesn’t show any result but there are still suspicious patterns on the user mode dump, start to review perfmon.

Use TMG PAL while reviewing Perfmon Data.

• Review PAL Report.
• Make sure to address the recommendations from PAL’s report.

Don’t focus only on TMG counters, also review Windows core counters

Memory/*, Processor/*, Network Interface/*, Process/*, Physical Disk/*, Threads/*

Enable netlogon logging to review potential authentication issues:

nltest /dbflag:0x2080ffff

• Review best practices for each core component.
• Make sure to understand the side effects of disk bottleneck on TMG by reading this post.
• Authentication and name resolution can also cause issues of this nature.

Task

Commands/Approach

What to look for at this stage

Make sure to attach a debugger to wspsrv.exe process in order to collect the dump when service crashes

Use Adplus or  DebugDiag

• Make sure to correctly install and configure one of those tools in order to collect the dump.

Load the crash dump on WinDBG

On WinDBG type:

!analyze -v

• Careful read the command output
• Review the exception record
• Review the faulting module

If the faulting module is not a TMG component, review who is the owner

On WinDBG type:

lmvm <faulting module>

• Review the module’s timestamp, sometimes there is a newer version that already fix the issue
• If the issue is caused by a third party, make sure to refer to third party site for more support info.

Task

Commands/Approach

What to look for at this stage

First make sure that is TMG that stops responding or the Windows OS that stops responding

Get answer for the following questions:

1) What do you do in order to put the server in production again?

2) Does the server get back in production if you restart Firewall Service?

3) How frequent does this issue happens?

• The goal here is really to understand if this is a hang on TMG or if the OS is the one giving up and as a side effect TMG stops answering.
• Review TMG Alerts; check if there are flood mitigation events.

If the whole server stops answering, get complete (or kernel) memory dump

Use the approach from this article to configure the server.

• Make sure to review each step of this article when preparing to collect kernel memory dump

Load the kernel dump on WinDBG

On WinDBG type:

!locks

• This command will list the kernel ERESOURCE locks
• The goal is to see if there is something locked in kernel mode
• Review the output to see if there are shared resources with a high contention count
• Review if there are threads waiting for that resource

Once you find a thread that might be waiting for a resource, dump the thread

On WinDBG type:

!thread <thread number>

• Review the thread information
• Look for potential IRPs

If the thread has IRPs, dump the IRP

On WinDBG type:

!irp <irp number>

• Check if the IRP is pending on a lower level driver.
• Check if there is third party holding that resource.
• Use lmvm to dump info about the module