Troubleshooting FIM Performance: Slow or Hanging Full Synchronization

Overview / Problem Statement
Cause
Resolution
See Also

Overview / Problem Statement

Recently worked an issue where we were experiencing an extremely slow Full Synchronization. The Full Synchronization was so slow, that it appeared as if it was hanging and not processing. In our troubleshooting, we executed the following actions:

PSSDIAG (SQL Server Diagnostics Tool)
1. This displayed that we were processing records, but at an extremely slow rate. PSSDIAG did indicate that there may be a problem with the growth of the TempDB. In investigation, we found multiple TempDBs for the multi-core processor. The first TempDB looked ok, but the other TempDBs had a 1 MB auto growth.
Review of Statistics
1. In the Synchronization Service Engine, we reviewed the Statistics Dialog. ( Tools > Statistics )
2. Here we noticed that the FIM Service Management Agent contained 200,000+ Pending Exports.

Cause

The first problem here is the TempDB auto grow being at 1 MB. The auto grow being so small, can provide issues with a slow synchronization.

The next problem here, is that having over 200,000 Pending Exports triggers the by design Object Whacking issue. This is where a Pending Export is deleted and re-added. Object Whacking must touch any object mentioned in a reference attribute on objects that are being deleted and re-added.

Additionally, our SQL Server Support Team was able to identify the contributor to the high number of Lock:Timeouts to be the GhostCleanUpTask Process.

SELECT
OBJECT_NAME(object_id) as Name,
record_count,
GHOST_RECORD_COUNT,
Version_ghost_record_count,
INDEX_TYPE_DESC,
ALLOC_UNIT_TYPE_DESC
FROM sys.dm_db_index_physical_stats (DB_ID(N'FIMSynchronizationService'), object_id('mms_connectorspace'), NULL, NULL , 'DETAILED')

Resolution

1. Set the Auto Grow for the TempDB to be between 300 MB and 500 MB
Clear the Pending Exports
1. Export on FIM Service Management Agent
2. Delta Import (Stage Only) and Delta Synchronization to confirm Export
Full Synchronization on Active Directory Management Agent