Hyper-V: Troubleshoot Pending Snapshot Merges

Hyper-V: Troubleshoot Pending Snapshot Merges

This article is based on a guest blog post by Jeremy Hagan at: http://blogs.technet.com/b/tonyso/archive/2010/11/25/the-case-of-the-pending-vm-snapshot-merge.aspx#3371639. Please help improve it; is is the wiki way!

When a virtual machine stops responding, frequently the underlying cause is that the hard-disk has run out of space. If you are using fixed-size VHD files, (so that there should be nothing that should grow to fill up the disk), check to ensure that the machine is NOT actually running on a differencing disk (AVHD) that had been a snapshot in the past, which changes still waiting to merge into the parent VHD file.

When a snapshot is deleted, the virtual machine must be powered off, not merely rebooted, in order for the data in the differencing disk to merge into the parent VHD. The VM must remain powered off until the merge is complete. If the VM is booted back up again before the merge is completed, then the merge process stops until the machine is powered off again. The bigger the AVHD, the longer the merge takes. Although SCVMM and Hyper-V manager have an indication in the GUI that a snapshot exists, when a snapshot is deleted it is deleted from the GUI right away even though the snapshot hangs around until the merge is complete. So if your VM in this state there may be no visible indication of it.

Usually, this occurs because the VM had a deleted snapshot for an unknown time period, which grew until it filled the underlying disk. If you cannot free up disk space,  the merge can not take place even if the VM was turned off.

Many we maintain a (250 GB or so) LUN on each Hyper-V cluster that is used for staging and emergencies. You can use System Center Virtual Machine Manager (SCVMM) to migrate the (power off) VM over the network to this "holding" or "temporary" LUN and allow the merge to complete and then move it back to its home. Problem solved.

To prevent this, do not disable free disk space monitoring on LUNs hosting VMs.  Since the LUNs routinely run low on space then SCOM will be routinely raising Chicken Little alerts that many ignore. 

Here's a better way: alert on a pending merge.

To do this, examine the difference between virtual machines in three different states:

  • one without a snapshot
  • one with a snapshot
  • one with a deleted snapshot

Start by downloading the PowerShell Management Library for Hyper-V. You can use configuration/global_settings/disk_merge_pending = true.

  • Use a VBScript  to create a temporary PowerShell script in the current directory. Because the VBScript is distributed by SCOM it will be on local disk,, allowing you to use RemoteSigned certificates and keep your servers safe.
  • Execute the PowerShell script by using the Windows Script Host Exec Method, which allows you to capture the output using the StdOut object of the Exec method
  • Return the state of the Hyper-V host (whether or not there was a pending merge) and the name of the VM with the pending merge to the monitor via the property bag API
  • Set the machine to a Warning state and raise an alert including the output of the script
  • Create a rule to run the script and a monitor to alert on the results. Jeremy modified the script and abandoned the property bag and just wrote the information to the local Application event log and the monitor would raise the alert based on that information. He configured the rule to run every 24 hours and for the alert to expire after 23 hours, so SCOM alerts him every day about the pending merge until resolves.

Of course you could customise this in your own implementation.


foreach ($file in Get-ChildItem "$Env:PROGRAMDATA\Microsoft\Windows\Hyper-V\Virtual Machines\*.xml" {
    $file.CopyTo((Split-Path -parent $MyInvocation.MyCommand.Definition) + "\" + $file.Name) | out-null
    [xml]$ConfigFile = Get-Content ((Split-Path -parent $MyInvocation.MyCommand.Definition) + "\" + $file.Name)
    $VMName = $ConfigFile.Configuration.properties.name."#text"
    $MergePending = $ConfigFile.configuration.global_settings.disk_merge_pending."#text"
    if ($MergePending -eq "True") {Write-Host "$VMName has a pending merge."}
    del ((Split-Path -parent $MyInvocation.MyCommand.Definition) + "\" + $file.Name)
    Remove-Variable VMName
    Remove-Variable ConfigFile
    Remove-Variable MergePending

Copy the configuration files to a temporary location before parsing them just in case

VBScript with embedded PowerShell

Option Explicit

Dim objFSO : Set objFSO = CreateObject("Scripting.FileSystemObject")
Dim objShell : Set objShell = CreateObject("WScript.Shell")
Dim strParentPath : strParentPath = objFSO.GetParentFolderName(WScript.ScriptFullName)
Dim objPSFile : Set objPSFile = objFSO.CreateTextFile(strParentPath & "\PSTemp.ps1", True)

'Get the path to the PowerShell executable from the registry
Dim strPowerShell : strPowerShell = objShell.RegRead("HKLM\SOFTWARE\Microsoft\PowerShell\1\ShellIds\Microsoft.PowerShell\Path")
Dim strCommand : strCommand = Chr(34) & strPowerShell & Chr(34) & " -NoProfile -NoLogo -File " & Chr(34) & strParentPath & "\PSTemp.ps1" & Chr(34)
Dim objExec
Dim strOutput : strOutput = ""
Dim i

objPSFile.WriteLine "foreach ($file in Get-ChildItem " & Chr(34) & "$Env:PROGRAMDATA\Microsoft\Windows\Hyper-V\Virtual Machines\*.xml" & Chr(34) & ") {"
objPSFile.WriteLine "    $file.CopyTo((Split-Path -parent $MyInvocation.MyCommand.Definition) + " & Chr(34) & "\" & Chr(34) & " + $file.Name) | out-null"
objPSFile.WriteLine "    [xml]$ConfigFile = Get-Content ((Split-Path -parent $MyInvocation.MyCommand.Definition) + " & Chr(34) & "\" & Chr(34) & " + $file.Name)"
objPSFile.WriteLine "    $VMName = $ConfigFile.Configuration.properties.name." & Chr(34) & "#text" & Chr(34)
objPSFile.WriteLine "    $MergePending = $ConfigFile.configuration.global_settings.disk_merge_pending." & Chr(34) & "#text" & Chr(34)
objPSFile.WriteLine "    if ($MergePending -eq " & Chr(34) & "True" & Chr(34) & ") {Write-Host " & Chr(34) & "$VMName has a pending merge." & Chr(34) & "}"
objPSFile.WriteLine "    del ((Split-Path -parent $MyInvocation.MyCommand.Definition) + " & Chr(34) & "\" & Chr(34) & " + $file.Name)"
objPSFile.WriteLine "    Remove-Variable VMName"
objPSFile.WriteLine "    Remove-Variable ConfigFile"
objPSFile.WriteLine "    Remove-Variable MergePending"
objPSFile.WriteLine "}"

Set objExec = objShell.Exec(strCommand)

'sanity check
Do While objExec.Status = 0
    WScript.Sleep 1000
    i = i + 1
    If i > 55 Then
        i = 0
    End If

Do While True
    If objExec.StdOut.AtEndOfStream Then
        Exit Do
        strOutput = strOutput & objExec.StdOut.Read(1)
    End If

If InStr(strOutput, "pending") Then objShell.LogEvent 2, strOutput

objFSO.DeleteFile strParentPath & "\PSTemp.ps1", True

Set objFSO = Nothing
Set objExec = Nothing
Set objPSFile = Nothing
Set objShell = Nothing





SCOM Monitor





Leave a Comment
  • Please add 7 and 2 and type the answer here:
  • Post
Wiki - Revision Comment List(Revision Comment)
Sort by: Published Date | Most Recent | Most Useful
  • Ed Price - MSFT edited Revision 10. Comment: Fixed header for TOC

  • Ed Price - MSFT edited Revision 9. Comment: Title, tags

  • Fernando Lugão Veltem edited Revision 4. Comment: added toc

  • Ed Price - MSFT edited Revision 3. Comment: Minor edits  

Page 1 of 1 (4 items)
Wikis - Comment List
Sort by: Published Date | Most Recent | Most Useful
Posting comments is temporarily disabled until 10:00am PST on Saturday, December 14th. Thank you for your patience.
  • Ed Price - MSFT edited Revision 3. Comment: Minor edits  

  • Does the management library have a fix for this situation:

    Our hyper-v host crashed but our VMs were stored remotely on a SAN so they are still intact. We cant use import to recover them because they werent exported and when we create a new VM and attach the old vhd file the snapshots arent there. Even when the snapshot link is pointed to the location of the old snaps, they dont show up in h-v manager. How can they be reattached to the old vhd in a new VM?

  • Fernando Lugão Veltem edited Revision 4. Comment: added toc

  • Ed Price - MSFT edited Revision 9. Comment: Title, tags

  • Ed Price - MSFT edited Revision 10. Comment: Fixed header for TOC

Page 1 of 1 (5 items)