Hyper-V VM Stuck checkpoint after restore from backup

This article discusses a situation where a ‘<VM Name> - Backup - (dd/mm/yyyy - hh:mm:ss)‘ checkpoint appears as a checkpoint next to a VM after you restore it from backup. This entry is a Hyper-V VM Stuck checkpoint and cannot be managed on the GUI.

Hyper-v Stuck Backup Checkpoint 'Phantom'

The only actions available on the ‘phantom’ checkpoint are to export the checkpoint.

Removing the checkpoint manually does not work and will fail with error 0x8000FFFF

Remove-VMSnapshot -VMName 'Auditor' -name 'Auditor - Backup - (25/09/2019 - 19:12:13)'

Further, when you attempt to Live Migrate, Migrate or Storage Migrate the Virtual Machine, you may encounter the error:

Hyper-V "Migration operation on '' failed" error message

Move Wizard
There was an error during move operation.
Migration operation on '<VM Name>' failed.

If you do, the Windows VMMS event log will record Event ID 20864:

Virtual machine failed to generate VHD tree: 'Catastrophic failure'('0x8000FFFF').

 

More Info

When you restore a Hyper-V Virtual Machine, it may try to retain an ancestral legacy to the backup set available in Windows Server Backup. Hyper-V will set the backup-set as the root checkpoint, from which the restore VM is descended.

You cannot perform any operations on this ‘phantom’ checkpoint, however the Hyper-V Manager will not allow the checkpoint to be deleted. If you wish to delete the checkpoint, you must manually edit the Virtual Machine Configuration file to remove the phantom.

 

The Fix

Please note that this guide only covers how to edit the VM Config file for the old XML format (up to Generation 5). It does not cover the Windows Server 2016 binary format, although the process will be logically similar.

  1. Merge or delete all desired checkpoints into a single VHD/VHDX file. Once completed there should be no checkpoints stored for the VM
  2. [Recommended] Locate the Virtual Machine .xml configuration file and create a proper backup (with permissions)
  3. Open the Virtual Machine .xml configuration file in notepad
  4. Locate the <snapshots> section for example:
<snapshots>
  <data_root type="string">c:\hyper-v</data_root>
  <list>
    <node0>
      <guid type="string">A74F62E9-4F8E-4F10-8BBC-95A5C75A465B</guid>
    </node0>
    <size type="integer">1</size>
  </list>
</snapshots>
  1. Note the value of the GUID.
    Perform a search of the file system for any files named using this GUID
    dir /a /s A74F62E9-4F8E-4F10-8BBC-95A5C75A465B*.*If any are found, move them to a temporary folder (but do not delete them until the process is complete)
  2. Delete the entire <snapshots> section, including the opening <snapshots> and closing </snapshots>
  3. The VM Configuration .xml file will be locked. To unlock the file, you must stop the Virtual Machine Management Service on the hypervisor.
    Please Note: Stopping the service will abort all replication, backup and migration jobs and may have other undesirable consequences. Proceed at your own risk.To stop the service on the hypervisor enter the command:
    net stop vmms
    Note: You must wait until the service has stopped. It may take 5-10 minutes (or longer) to complete depending on the number of VM’s, service job queue and current VM load.
  4. Save the modified .xml file
  5. Restart the Virtual Machine Management Service by issuing the command:
    net start vmms
  6. Review the corrupted VM in Hyper-V Manager. The phantom checkpoint will have gone.
  7. Test that the VM can boot and migrate before cleaning-up

 

Conclusion

After this process the VM should start, Live Migrate, Migrate and Storage Migrate correctly without any issues. If it does not, you are likely experiencing additional configuration file corruption.

Note: If you do not perform Step 5, the phantom backup checkpoint may reappear after you next checkpoint the VM.

For more on Hyper-V Error troubleshooting visit my Hyper-V Error Message Cheat-Sheet.