Symptoms

  • Client requests that traverse an AD FS 2.x Proxy server intermittently fail
  • The AD FS/Admin Event Log contains event ID 230 showing that the proxy is experiencing congestion


Possible Cause 1

  • The internal AD FS 2.x Federation Server is overloaded with requests

Possible Cause 2

  • The AD FS 2.x Proxy server is overloaded with requests

Possible Cause 3

  • There is a network issue occurring between the AD FS 2.x Proxy server and the internal AD FS 2.x Federation Server


Troubleshooting Action Plan


On any involved load balaner(s) and firewall(s):

 

  1. No affinity (stickiness) on load balancers - Affinity won’t necessarily break anything, but it is not a requirement, thus Microsoft does not recommend it to be enabled
  2. Ensure the AD FS DNS name /* paths are allowed through any firewalls so that any AD FS 2.x Proxy can talk to any internal AD FS 2.x Federation Server

 

On all AD FS 2.x Proxy servers:

 

1. If you have SCOM/MOM monitoring capabilities, add a new rule to notify you if AD FS 2.x Proxy event ID 230 is logged in the Admin event log

 2. Collecting data while the issue is happening: PerfMon BLG collector set with frequent sampling interval containing:

a. AD FS > Rejected token requests – if this is >= 1, you have a congestion issue - Event ID 230 will be logged

b. AD FS > Token request latency – default threshold is 2000ms - If this number is greater than 2000 for any period of time, rejected token requests begin and event ID 230 is logged

c.  AD FS > Outstanding token requests – default threshold is 16 outstanding requests - If this number is greater than 16 for any period of time, rejected token requests begin and event ID 230 is logged

d. IPv4 > Segments retransmitted/second – look for this to increase when you are experiencing the issue, as this may indicate a transient network issue

 

3. Providing relief while the issue is happening: Set the congestion algorithm thresholds higher temporarily to lessen impact to the environment

 

An AD FS 2.x administrator can adjust the congestion algorithm in the federation server proxy’s config (located in %programfiles%\Active Directory Federation Services 2.0 in Windows Server 2008 R2 and 2008. Located in %windir%\ADFS in Windows Server 2012) file. Here is the line for setting the congestion algorithm in the federation server proxy’s config file which is located in the <microsoft.identityServer.proxy> section:

<congestionControl latencyThresholdInMSec="2000" minCongestionWindowSize="16" />

Explanation of the configurable parameters in this setting: 

Parameters

Allowed Values

Description

latencyThresholdInMsec 

Min Value = 1000
Max Value=60000
Default value = 2000

Controls the sensitivity of the congestion algorithm trigger. When the average latency time rises above the latencyThresholdInMsec value, congestion control comes into effect.

minCongestionWindowsSize

Min Value = 1
Max Value = 10000
Default value = 16

Controls the severity of the congestion algorithm. Defines the smallest number of concurrent requests from the federation server proxy to federation server if persistent congestion occurs.

 

4. ( Optional ) – Collect a circular network trace while monitoring Token request latency - This might help clue you in to TCP retransmissions

 

 

On all internal AD FS 2.x servers:

 

  1. When the issue is happening: PerfMon BLG collector set containing:
    1. AD FS > Token requests/sec – We expect to see this number at or below 60 token requests per second per server - If this is greater than 60, you may simply be overloading the internal server, which causes the proxy to wait

 

Note: Microsoft typically does not see this manifest as a true load issue on the internal servers, assuming that the infrastructure has been sized appropriately for your use scenario. The majority of the time, we find that something unexpected is happening on the network when this issue occurs.