“Some migration network settings cannot be modified or removed because these migration network settings are in use by a cluster” in Hyper-V manager, Hyper-V Settings after a node is non-cleanly evicted from a failed cluster

System Requirements:

  • Windows Server 2008, 2008 R2, 2012, 2012 R2, 2016
  • Hyper-V

The Problem:

After a total cluster failure occurs, or a node is removed from a cluster before it has been cleanly removed. When attempting to locally manage the host as a stand-alone Hyper-V server, you are unable to edit the live migration settings in Hyper-V Settings via Hyper-V Manager with the error

“Some migration network settings cannot be modified or removed because these migration network settings are in use by a cluster”

The IP/Subnet addresses shown on the form will be greyed out and you will be unable to edit the live migration network settings.

Hyper-V Settings: Unable to edit Live Migration Networks

More Info

While you should ensure that you have performed a cluster clean-up on the host

Server 2012 +:
Clear-ClusterNode -Force -CleanupA

Server 2008/R2:
cluster node <hostname> /forcecleanup

This will not solve the live migration settings issue highlighted above.

The Fix

  1. Close Hyper-V Manager on the management workstation
  2. On the hypervisor experiencing these symptoms (not necessarily the management console). Navigate to HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Virtualization\Migration\NetworkSettings
  3. Underneath this key there will be a key for each entry shown on the Live Migration Settings screen. The keys will be named Network# e.g. Network0, Network1, Network2 and so on.
  4. Go through each of these Network# keys and locate the ‘Tags’ REG_MULTI_SZ. This will have a value of “Microsoft:ClusterManaged”
    Registry Screenshot: Error
  5. Change the data value to “Microsoft:UserManaged”
    Registry Settings: Fixed
  6. Complete the process for each Network# sub-key
  7. Restart Hyper-V Manager

You will now be able to add, edit and delete the live migration settings.

Hyper-V Discrete Device Assignment (DDA) with a TV Tuner (Hauppauge HVR-4400)

System Requirements:

  • Windows Server 2016
  • Hauppauge HVR-4400 PCIe Tuner

The Problem:

I am a DVBLink user. DVBLink does not play nicely with Windows Service and consequently it wants to run on a client OS. This means that I have lots of server hardware running server Operating Systems and one device with 4 TV Tuners in it running Windows 10.

With the release of Windows Server 2016 came the promise of VMWare like PCIe Pass-through, allowing physical devices on the PCI bus to be attached to VMs. The plan is to attach the PCIe TV Tuner and attempt to get DVBLink working in a VM so that the physical unit can be decommissioned (saving the power bill).

More Info

As part of the process, I was considering building a new server at the start of 2017 to perform a consolidation against. The Windows 10 DVBLink machine would be one consolidated devices onto more powerful modern hardware. I would also need new TV Tuners as only 2 of the 4 in the DVBLink TV Server is PCIe, the rest are PCI. Again, there are opportunities to consolidate that into fewer PCIe devices too.

The driver for the new server was Hyper-V PCIe Pass-through, or “Discrete Device Assignment” (DDA) as Microsoft are calling it. It is however quite difficult to find out whether BIOS firmware supports the proper implementations of I/O-MMU VT-d to permit it, making the purchase a risk. Equally, there is no guarantee that DDA will work with a TV Tuner.

Consequently, I decided to borrow a dual CPU Dell PowerEdge R630 to perform the experiment as there were several reports on-line that the R6xx and R7xx have the proper VT-d and SR-IOV feature set for this type of activity. Well done Dell (why don’t you advertise this?!).

After updating firmware, adding the TV Tuner and installing Windows Hyper-V Server 2016 on the machine, the first step was to – as an experiment – attempt to install the TV Tuner drivers on Windows Server 2016 (which errored). After that it was time to run the DDA Survey Script from Microsoft.

Download: DDA Survey Script (GitHub)

 

This was promising. The script found two devices that it stated were capable of being used with DDA

PERC H730 Mini
Express Endpoint -- more secure.
And its interrupts are message-based, assignment can work.
PCIROOT(0)#PCI(0100)#PCI(0000)

and

Hauppauge WinTV HVR-4400 (Model 121xxx, Hybrid DVB-T/S2, IR)
Express Endpoint -- more secure.
And it has no interrupts at all -- assignment can work.
PCIROOT(0)#PCI(0200)#PCI(0000)

The next step was to dismount the device from the Hypervisor and make it available to Hyper-V

# Find the HVR-4400
$pnpdevs = Get-PnpDevice -PresentOnly | Where-Object {$_.Class -eq "Media"} | Where-Object {$_.Service -eq "HCW85BDA"}# ... or if you know the hardware ID
$pnpdevs = Get-PnpDevice -PresentOnly | Where-Object {$_.InstanceId -eq "PCI\VEN_14F1&DEV_888
0&SUBSYS_C1080070&REV_04\4&39CDA168&0&0010"}foreach ($pnpdev in $pnpdevs) {
Disable-PnpDevice -InstanceId $pnpdev.InstanceId -Confirm:$false
Write-Host 'Device ' $pnpdev.InstanceId ' Disabled. NOTE: If this hangs, reboot and try again'
$instanceId = $pnpdev.InstanceId
$locationpath = ($pnpdev | get-pnpdeviceproperty DEVPKEY_Device_LocationPaths).data[0]
Write-Host 'Dismounting Device At: ' $locationpath ' (' $instanceId ')'
Dismount-VmHostAssignableDevice -locationpath $locationpath
Write-Host $locationpath
}

Initially, it hung PowerShell (and the system) so I had to hard reset the server. In this instance it was in fact necessary to reboot after issuing

Disable-PnpDevice

After trying again and rebooting the Dismount-VmHostAssignableDevice failed with

dismount-vmhostassignabledevice : The operation failed.
The manufacturer of this device has not supplied any directives for securing this device while exposing it to a
virtual machine. The device should only be exposed to trusted virtual machines.
This device is not supported when passed through to a virtual machine.
The operation failed.
The manufacturer of this device has not supplied any directives for securing this device while exposing it to a
virtual machine. The device should only be exposed to trusted virtual machines.
This device is not supported and has not been tested when passed through to a virtual machine. It may or may not
function. The system as a whole may become unstable. The device should only be exposed to trusted virtual machines.
At line:1 char:1

It would not proceed past this point. The trick was to change the line to

Dismount-VmHostAssignableDevice -locationpath $locationpath -Force

The next step was to ensure that the VM’s Automatic Stop Action was set to anything other than “Save”

Set-VM -Name “10-TEST” -AutomaticStopAction Shutdown

… and at this point it was simply a case of creating a VM and assigning the device

Add-VMAssignableDevice -LocationPath “$locationpath” -VMName “10-Test”

At which point the device immediately popped up in Device Manager under Windows 10 in the Generation 2 VM

DDA PCIe Passthrough in Device Manager

…. before the VM blue screened a few seconds later.

Blue Screen of Death

I tried to use several versions of the HVR-4400 driver that I could find and it made no difference. The VM would crash whenever it attempted to talk to the card. The Hypervisor itself did not seem to be impacted by the Blue Screen event and itself did not crash.

I also tried fully removing the device from the Hypervisor using DEVCON and clearing out the driver using pnputil. Superficially, this action made it worse as the VM wouldn’t boot at all now if it had a driver on-file for the TV Tuner. Before it would at least boot.

So this project was a failure and I will not be investing in new server hardware just yet. I’ll wait to see if Microsoft improve the feature set as allegedly this type of insanity (and yes, it is insane) is possible in VMWare. I do not want to change away from Hyper-V at the current time though, so I will have to stick with a client machine as a service.

This does not mean of course that this cannot work in Hyper-V. The HVR-4400 is a card from 2011/2012. So it is not exactly new hardware. PCIe TV Tuners designed to modern electrical standards and for use on PCIe 3.0 bus architectures may provide better interoperability out of the box. I just don’t have any other cards to test with and am in a bit of a chicken and egg situation over wanting to invest in new cards and servers unless I know they will work nicely together.

If you are interested in this too and would like me to have a go testing your hardware, please get in touch via HPC:Factor.

Error 0x80070002 when attempting to backup a Hyper-V Virtual Machine using Windows Server Backup

System Requirements:

  • Windows Server 2008 R2
  • Windows Server 2012
  • Windows Server 2012 R2

The Problem:

You backup, right? Of course you do! Only the cool people backup – and you are one of the cool people aren’t you?

…If only life was that simple.

So imagine for a moment that you are attempting to use VVS and WIndows Server Backup to backup a server. In particular a fully loaded Hypervisor running Windows Server 2012 R2 Datacentre in this case.

The backup process goes OK for the most part, but fails to complete on a number (but by no means all) VMs. The process fails with the following errors on Windows Server 2008 VMs, but not necessarily newer ones.

From Windows Server Backup:

Windows Server Backup “Failed” -or- “Completed with warnings” -or-“Backup failed to complete”

The component <VM Name>(Online) was skipped during the snapshot and will not be available for recovery. Error: The writer experienced a non-transient error. If the backup process is retried, the error is likely to reoccur

In the Hyper-V-VMMS\Admin log in Event Log:

‘<VM Name>’ cannot create the storage required for the checkpoint using disk E:\Virtual Machines\<VM Path>\Virtual Hard Disks\<VHD Filename>.vhdx: The system cannot find the file specified. (0x80070002). (Virtual machine ID <VM GUID>)

and…

Checkpoint operation for ‘<VM Name>’ failed. (Virtual machine ID <VM GUID>)

and…

Could not create backup checkpoint for virtual machine ‘<VM Name>’: The system cannot find the file specified. (0x80070002). (Virtual machine ID <VM GUID>)

and of course most helpfully…

The operation failed.

More Info

If you actually look at the backup file, you will see what looks to be a complete file set for the backup, however given that this error represents an error in VSS, you would not be advised to trust it.

As usual with Hyper-V error logs, the error message have little if anything to do with the actual issue and someone in the Microsoft Development team just needs to be shot for it… but I digress.

The odd think was that the issue was occurring on all of the Windows Server 2008 (R1) VMs, the Windows Server 2008 R2 and higher VMs were backing up correctly.

The Fix

So before I get into the issue I encountered, lets run past the generic fixes

  1. Check that you have enough disk space on the volume to perform the VSS. If your volume is sub-15%, try using the Hyper-V Manager to change the snapshot directory to another volume – plug in an external NTFS formatted hard drive if you have to.
  2. Check the permissions of the VHD stated in the error.
    icacls “C:\Users\Public\Documents\Hyper-V\Virtual Hard Disks\<VHD File>” /grant “NT VIRTUAL MACHINE\Virtual Machines”:F /TSource: Technet
    Source: System Center Central
  3. Ensure that Hyper-V is patched fully.
    Windows Server 20102 R2 users see: https://support.microsoft.com/en-us/kb/2920151
  4. Run chkdsk on the physcial volume on the Hypervisor and on the virtual volume in the VM
  5. Ensure that the Integration Service Components are at the latest version and that they VSS Writer module for it is enabled in the VM properties in Hyper-V Manager

Now the less well documented approaches

  1. Check that you can manually checkpoint/snapshot the VM while it is running.
    In Hyper-V Manager or in PowerShell, force a checkpoint on the VM and then delete it and wait for it to merge back. If this works, you are not having a physical VSS issue. If it fails, you need to troubleshoot this and not the WSB error.
  2. Live Migrate the VM off of the current server and onto a different Hypervisor, attempt the backup here, then bring it back to the original server and try again. This process will reset the permissions on the VM file set. If you cannot live or offline migrate the VM, the you need to troubleshoot this and not the WSB error.

My fix

In my case, the issue was to do with having the VM VHDX files split across a couple of different storage LUN/volumes. I usually move VM page files onto a dedicated partition on a dedicated spindle (usually an SSD) and leave OS and data volumes on larger arrays. This helps to keep the VMs running smoothly and keeps unnecessary paging operations off of parity checked storage volumes.

So imagine that the VM has the following file storage structure

Physical Hypervisor SSD (this is where the Page File’s live)
– D:\my-virtual-server-d-drive.vhdx

Physical Hypervisor Storage Array
– E:\Virtual Machines\<VM Name>\Planned Virtual Machines
– E:\Virtual Machines\<VM Name>\Snapshots
– E:\Virtual Machines\<VM Name>\Virtual Hard Drives\my-virtual-server-c-drive.vhdx
– E:\Virtual Machines\<VM Name>\Virtual Hard Drives\my-virtual-server-e-drive.vhdx
– E:\Virtual Machines\<VM Name>\Virtual Machines

It is actually this structure which breaks the WSB backup job. Contrary to the VSS event log error, the problem drive is NOT my-virtual-server-c-drive.vhdx it is actually my-virtual-server-d-drive.vhdx. The event log will actually log that the error was caused on the first drive attached to the system bus (I think).

If you weren’t too clever when you followed my advice above and live migrated all of the storage to the same location on a different Hypervisor, you probably found this out for yourself – the backup should have worked.

When you split the job back into separate LUNs, it fails again. The fix is oddly simple and continues to allow you to have split LUN storage if you wish. Change the file system structure to:

Physical Hypervisor SSD (this is where the Page File’s live)
– D:\<VM Name>\my-virtual-server-d-drive.vhdx

Physical Hypervisor Storage Array
– E:\Virtual Machines\<VM Name>\Planned Virtual Machines
– E:\Virtual Machines\<VM Name>\Snapshots
– E:\Virtual Machines\<VM Name>\Virtual Hard Drives\my-virtual-server-c-drive.vhdx
– E:\Virtual Machines\<VM Name>\Virtual Hard Drives\my-virtual-server-e-drive.vhdx
– E:\Virtual Machines\<VM Name>\Virtual Machines

Note the introduction of a folder on the D drive with the same name as the <VM Name> folder on the E drive. Do NOT shutdown the VM and move the storage there yourself, use the Hyper-V Manager or PowerShell processes to perform a “move” on the ‘storage only’ and just move the one drive. This will ensure that permissions are correct.

The next time that you run the backup, it will VSS correctly.

As for why it can do this on its own with Windows Server 2008 R2 or higher VMs, but not Windows Server 2008 or lower VMs… I have no idea although I suspect it to have something to do with the capabilities of the integration services components.

Edit: A post publish search on the issue reveals that I’m not alone in working this out
View: Technet

Error 0x80070005 when attempting to Perform a Shared Nothing migration between Hyper-V hosts or move a Hyper-V VM between CSV’s in the same or separate Clusters

System Requirements:

  • Windows Server 2012 R2
  • Windows Server 2016

The Problem:

Hyper-V 2012 R2 has a lot of new features that are worthy of note and one of the most appealing features for Virtualisation Administrators is shared nothing migration between hosts via SMB. If you are in an environment that doesn’t have shared storage it’s useful enough in itself because for VM purposes it may have just validated your decision not to get shared storage in the first place. Yet less well documented is the features value for setups where when you do have shared storage as you can use shared nothing migration as a mechanism to live migrate VM’s between clusters that are backed onto shared storage – or more specifically between “Cluster Shared Volumes” (CSV).

The picture on the back of the box of the smiling, happy systems administrator performing a shared nothing administrator makes it look so easy right? This is however an all too common occurrence:

0x80070005 Error

'General access denied error'('0x80070005')

 

There was an error during move operation.

Virtual machine migration operation failed at migration source.

Failed to create folder.

 

There was an error during move operation.

Virtual machine migration operation failed at migration source.

Failed to create folder.

Virtual machine migration operation for ‘<VM Name>’ failed at migration source ‘<Source Hypervisor name>’. (Virtual machine ID <VM-SID>)

Migration did not succeed. Failed to create folder ‘<RPC path>…\Virtual Hard Disks’: ‘General access denied error'(0x80070005’).

If you look at the specified destination path (e.g. c:\ClusterStorage\Volume1\test) after receiving this error, you will find that it has created the test folder and it will have created a ‘Planned Virtual Machines’ folder beneath it which will in turn contain a folder named with the VM’s VM-SID (the Virtual Machines unique security ID) and a .xml file named with the same VM-SID.

The migration will however not progress any further.

If you attempt to perform the same operation in PowerShell you will receive the PowerShell version of the same error:

VERBOSE: Move-VM will move the virtual machine "<VM Name>" to host "<Destination Server>"
Move-VM : Virtual machine migration operation for '<VM Name>' failed at migration source '<Source Server>'. (Virtual machine ID<VM-SID>)
Migration did not succeed. Failed to create folder
'\\<Destination Server>\<Source Server>.762091686$\{e166ba26-8a4a-4029-ac34-c2466451e439}\<VM Name>\Virtual Hard Disks': 'General access denied error'('0x80070005').
You do not have permission to perform the operation. Contact your administrator if you believe you should have permission to perform this operation.
At line:1 char:69
+ $vm = Get-VM -Name 'test' -ComputerName "<Source Server>" | Move-VM -Des ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : PermissionDenied: (Microsoft.Hyper...VMMigrationTask:VMMigrationTask) [Move-VM], Virtual izationOperationFailedException + FullyQualifiedErrorId : AccessDenied,Microsoft.HyperV.PowerShell.Commands.MoveVMCommand

Please Note: This document does not specifically address 0x80070005 for Hyper-V Replication Troubleshooting, which is a slightly different (yet related) issue.

More Info

Understanding the topology involved in my setup also reveals my reason for needing to get this working – this is important as setup and reasons yours may differ slightly. What I was attempting to do was migrate between two multi-node Windows Hyper-V Server 2012 R2 clusters while being able to initiate the migration from a third device, a Windows 8.1 management console.

Much of the discussion surrounding 0x80070005 suggests that you simply need to deal with the fact that you need to log onto the source workstation and initiate a push of the VM from the source server to the destination server using CredSSP. This is fine if you have a general purpose commodity server that happens to have Hyper-V on it. In the real world if you have a Hyper-V Cluster, you should not be running it in GUI mode, you should be using Server Core – and if you are using Windows Hyper-V Server to begin with, you don’t even have the option of a GUI.

So we can eliminate the use of the GUI tools or the simplicity of “just RDP into the server” immediately from this discussion. People answering as such are running in very simple Hyper-V setups and in environments with simple, very liberal security policies.

You can of course use PowerShell to perform a CredSSP migration on a Server Core installation and as a mater of good practice the ability to transfer VM’s using CredSSP should be confirmed as working before you start out with Kerberos. To do that, log onto the Source Server and execute the following command in a PowerShell session:

Get-VM -Name ‘<VM Name To Move>’ | Move-VM -DestinationHost “<Destination Server>” -DestinationStoragePath “C:\ClusterStorage\Volume1\<VM Name to Move>” -Verbose

If that doesn’t work, I recommend that you troubleshoot this issue before you look to go any further on the 0x80070005 issue.

Additionally, before make sure that you have done performed the basic troubleshooting steps and also ensure that you are simplifying the problem as much as possible before starting. The following provides an overview of such steps in no particular order:

  • Log-in as a Domain Admin to perform this test (if possible). After you have that working migrate down to delegated users and troubleshoot any issues that they are experiencing
  • Only try to ‘shared nothing’ migrate a VM that is turned off (create a new VM, attached a default sized dynamically expanding disk, don’t add any networks and leave it off as this means that you will only have 4MB of data to test move). Once you can migrate a VM that is off, attempt to migrate a running VM with a Live Migration.
  • Only test migrate between the Source Cluster storage (CSV) owner node and the Destination Cluster storage owner node
  • If possible, make the owner of the source and destination cluster core resources the same node that owns the CSV
  • Remember that you must use Hyper-V Manager after you have de-clustered the VM from within Failover Cluster Manager before you can perform a shared nothing migration – the fact that your VM has anything to do with a cluster is an aside for Hyper-V. Treat this process as a Hypervisor to Hypervisor move that happens to be on a CSV and forget about the cluster.
  • On the ‘Choose a new location for virtual machine’ page of the migration wizard, remember that you must enter a file system path (e.g. C:\ClusterStorage\volume 1\test) and not a UNC path (e.g. \\server\c$\ClusterStorage\volume 1\test). The migration is going to take place using RPC and not SMB. Thus do not use a UNC path.
    'Choose a new location for virtual machine' wizard page
  • Ensure that you can migrate the VM using CredSSP as discussed at the beginning of this section
  • Ensure that your Domain Controllers are running Windows Server 2008 or higher (or at least your logon server), Windows Server 2003 Domain Controllers are known to have issues here (possibly due to lack of AES support). Your domain / forest functional levels can reportedly be Windows Server 2003 if required. I have only tested with Windows Server 2008 domain functional and Windows Server 2008 forest functional levels
  • If you are attempting to move between servers in a domain trust, you must ensure that the domain trust supports AES
  • Keep your initial testing paths simple and avoid overly complicated NTFS structures. For example, target the destination to be a local sub folder of c:\ and not a junction (such as ClusterStorage\Volume #) or a non-drive letter NTFS Mount Point (i.e. a iSCSI share or drive mount point exposed as a sub-folder to a higher file system). See the links below for more on this.View: Snapshot – General access denied error (0x80070005)
    View: Migrating a Virtual Machine problemNote: The iCACLS command listed in the second link does not use the principal of least permission. The command to enact the principal of least permission would be as follows:

    icacls F:\hvtest /grant “NT VIRTUAL MACHINE\Virtual Machines”:(OI)(CI)(R,RD,RA,REA,WD,AD) /T

    Finally, keep in mind that for delegation purposes, permissions must be valid for the user account that you are using to perform the move as well as the SYSTEM account.

  • Initially, forget about testing the migration into the cluster CSV itself. Instead, create a new folder on the root of the C Drive of the destination server and migrate into this. There are a few suggestions online that you need to put a couple of folder depths between the root of the drive and the VM itself so try something like:
  • C:\VM Store\Test\
  • If you are following my advice, you will be testing with a 4MB VM called ‘test’ so there won’t be any issue with storage space and the use of the C Drive for testing
  • User PowerShell for testing, otherwise you will go insane from having to repeatedly re-enter information in the Move VM wizard. The general gist of the command is:
    Get-VM -Name ‘<VM Name To Move>’ -ComputerName “<Source Server>” | Move-VM -DestinationHost “<Destination Server>” -DestinationStoragePath “C:\ClusterStorage\Volume1\<VM Name to Move>” -Verbose

    With the 0x80070005 error, you should find that it will get to 2% and then error after a few seconds.

  • Ensure that you have enabled Kerberos authenticated Live Migrations in the properties for the Hypervisor in Hyper-V Manager
    Hypervisor PropertiesNote: You can perform this action in PowerShell using

    Enable-VMMigration -ComputerName <Server Hostname>
    Set-VMHost -ComputerName <Server Hostname> -VirtualMachineMigrationAuthenticationType Kerberos
  • Ensure that your Hypervisor’s and the Windows 8.1 management VM are up to date (at the same patch level) and are joined to the same domain
  • Ensure that all parties in the process have properly registered DNS records in AD DNS
  • Check your Windows Firewall rules – for testing purposes just turn them off if you can (remember to turn them back on afterwards!)
  • Check your ASA/Hardware Firewall rules for the same
  • Keep an eye on the Hyper-V event logs for any additional information. The log of consequence is found in event Viewer under:Applications and Services Logs > Microsoft > Windows > Hyper-V-VMMS > AdminIf you are experiencing the same problem that I was, you will see three events on the Source Server’s log (20414, 20770 and 21024). The 20770 error is the one being reflected by PowerShell or the Hyper-V Management console. Shortly there-after, the Destination Server will log a 13003 event informing you that the virtual machine from the Source Server (with the same VM-SID) was deleted, indicating that the Destination Server performed a clean-up of the initial migration process.

Permissions

There is a lot of discussion about permissions and 0x80070005 errors. Let us look at the salient points

VERBOSE: Move-VM will move the virtual machine "<VM Name>" to host "<Destination Server>"
Move-VM : Virtual machine migration operation for '<VM Name>' failed at migration source '<Source Server>'. (Virtual machine ID <VM-SID>)
Migration did not succeed. Failed to create folder
'\\<Destination Server>\<Source Server>.762091686$\{e166ba26-8a4a-4029-ac34-c2466451e439}\<VM Name>\Virtual Hard Disks': 'General access denied error'('0x80070005').
You do not have permission to perform the operation. Contact your administrator if you believe you should have permission to perform this operation.
At line:1 char:69
+ $vm = Get-VM -Name 'test' -ComputerName "<Source Server>" | Move-VM -Des ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : PermissionDenied: (Microsoft.Hyper...VMMigrationTask:VMMigrationTask) [Move-VM], VirtualizationOperationFailedException + FullyQualifiedErrorId : AccessDenied,Microsoft.HyperV.PowerShell.Commands.MoveVMCommand
  1. The Migration failed at the Source Server
  2. The Source Server failed the migration because it could not ‘create a folder
  3. We know that the folder in question is the Source Server being unable to create a ‘<VM Name>\Virtual Hard Disks‘ folder
  4. We know that the Source Server was able to create a ‘<VM Name>\Planned Virtual Machines’ folder because we can see it in the file system if we use the GUI Wizard to perform the migration.
    Note: The PowerShell version cleans up after itself!
  5. You have told the Hypervisor to use Kerberos to perform the migration

What does this tell us? It tells us that YOU, the administrator are being told that you cannot create the folder. You are using Kerberos to perform the migration, not CredSSP, so the entire process is being run end-to-end using YOUR credentials. The Management Workstation is logging onto the Source Server as YOU. The Management Workstation is telling the Source Server to initiate the move and in turn the Source Server is delegating your authentication session to the Destination Server and telling it to receive instructions from the Source Server using your credentials. At this point it has nothing to do with ‘NT Virtual Machine’ or VM-SID permissions, this comes after the migration of the core parts of the VM and during initialisation of the VM on the Destination Server. We are not there yet.

So the first thing to check is that your account is authorised to perform the move. If you are a Domain Admin, you should be OK, however you should ensure that the Domain Admin’s security group is a member of the Local Administrators Group on the all participating machines – source server, destination server and management workstation.

If you do not want the user account to have full local admin rights you can add them to the “Hyper-V Administrators” group on each server. To add an account to a local group on Server Core or Windows Hyper-V Server:

net localgroup "Hyper-V Administrators" /add domain\user
net localgroup "Administrators" /add domain\user

Constrained Delegation

When viewing the Delegation tab on the computer account in Active Directory Users & Computers (ADUC) ensure that:

  1. You are using “Trust this computer for delegation to specified services only” (it doesn’t appear to work if you use the “any service” option)
  2. You have selected “Use Kerberos only”
  3. You tick the ‘Expanded’ checkbox to view the full list of entries
  4. That (once Expanded) there are two entries for each type (types being CIFS and Microsoft Virtual System Migration Service), one entry will have the NetBIOS Name and the other will have the FQDN i.e. there are 4 entries for each delegated host, two with NetBIOS Names and two with FQDN entries.
  5. When you create the Kerberos Constrained Delegation, you need to ensure that the “Service Name” field column is blank. If there is something listed in the Service Name column, your delegation is not going to work properly.
  6. You need to have the same number of “CIFS” entries for each host as you do for “Microsoft Virtual System Migration Service”
  7. It is not necessary to add the Management Workstation to the Constrained Delegation

When you issue the Move-VM command in PowerShell, try substituting the -ComputerName and -DestinationHost values for four combinations of the NetBIOS Name and FQDN.

Get-VM -Name ‘<VM Name To Move>’ -ComputerName “<Source Server>” | Move-VM -DestinationHost “<Destination Server>” -DestinationStoragePath “C:\ClusterStorage\Volume1\<VM Name to Move>” -Verbose

For example, if your have Server1 and Server1 and your domain is domain.local the combinations to test are:

Source Destination
Server1 Server2
Server1.domain.local Server2
Server1 Server2.domain.local
Server1.domain.local Server2.domain.local

If you find that one of these works while the others do not, you have an error in the constrained delegation setup for DNS or NetBIOS aliasing. Carefully recreate the delegation.

After you have setup the delegation, go into a LDAP browser, ADSI Edit or the Attribute Editor in ADUC. For each delegated server, find the servicePrincipalName property and look at the value list. You should have two of each of the following entries (one with the NetBIOS Name and the other with the FQDN).

  • Hyper-V Replica Service/
  • Microsoft Virtual System Migration Service/
  • RestrictedKrbHost/

If you do not see these, you have a Delegation Error and/or an issue in creating SPN records. Either delete and try to recreate them by recreating the delegation or carefully add them by hand.

DNS

Bindings. I know that you checked them, but check them again. Trust me. On Server Core where you have very little contact with the actual server console this is very easy to overlook.

Constrained delegation may work with both NetBIOS and DNS, however Kerberos does not care for NetBIOS. If your DNS doesn’t work, you aren’t going to get a successful ticket session creation that you will need in order to pass credentials forward as part of the Constrained Delegation setup.

Check the following using short hand and FQDN lookups i.e. nslookup server1.domain.local and just nslookup server1. Are they both going where you expect? Crucially, which server NIC is the DNS query going out of and once the reply comes back, which NIC is being used to attempt to contact the host?

  1. The management console can query all domain controllers in DNS
  2. The management console can query all Hypervisors in DNS
  3. The hypervisors can all query the management console in DNS
  4. The hypervisors can all query all domain controllers in DNS
  5. The hypervisors can all query each other in DNS

This also requires you to check your default gateway settings.

This is important in the following scenario. Most of you will not encounter this because of the scale of your operations, however the fact is that at Enterprise level I did encounter this problem, hence why I able to write about it.

  1. Lets assume that you follow best practice and have separate public, management, cluster, iSCSI and heartbeat networks.
  2. Your management network is data centre local, on a private network with minimal routing and is designated to management of servers, IPC traffic, un-routed VM’s etc in a secure fashion
  3. Local DNS is available on the management network but does not expose Internet Resolution
  4. Your public VM address ranges come from the public network and are not exposed via NAT/PAT i.e. routing and firewall’s
  5. Your domain controllers exist on a public routed network subnet that is separate from the public VM address ranges used for VM’s
  6. You followed best practice and set your management networks binding order to be the first adapter in the binding order on the hypervisors
  7. You will now receive 0x80070005 when you attempt to replicate, live migrate of off-line migrate a VM between cluster nodes using Kerberos Constrained Delegation

The problem is the adapter binding order caused by the use of local DNS on a network that offers no connectivity to the domain controllers. When the KDC attempts to generate a Kerberos ticket for the constrained delegation, the lookups for the domain controllers will be performed using the DNS servers on the management network and will mistakenly attempt to connect to the domain controllers via the management network. This is simply going to time out – causing the wait during migration. Once it times out, Windows DNS doesn’t defer to the next set of DNS servers or attempt to get to the DC’s on a different NIC. It simply gives up.

The resulting very helpful error code that Hyper-V offers back is Access Denied while seemingly attempting to create files in the file system – the Hypervisor will log that it was unable to create the ‘Virtual Hard Drives’ folder on the destination Hypervisor. What it should actually say here is that it could not properly initialise the end to end Kerberos Constrained Delegation ticket session due to a timeout. It of course doesn’t do that.

In this situation the fixes are one of:

  1. Add an interface on the domain controllers on the management LAN
  2. Add a network interface which can connect to the domain controllers in a higher adapter binding order position in the Hypervisor binding order
  3. Remove the DNS servers from the management networks TCP/IP properties, thus forcing Windows Server to use the first available DNS server configuration on a lower ordinal adapter
  4. Allow routing from the management LAN to the domain controllers. Alias, stub or secondary zone the domain controllers in the management networks DNS and hope you remember to keep them up to date when you make changes to Domain Controller DNS records

Assuming that your constrained delegations are correct, it will start working as soon as the DNS updates have propagated.

The Fix

Ultimately the problem that I had was in the setup of the Constrained Delegation and in another case as discussed above, the DNS binding order. For the Constrained Delegation issueI only had NetBIOS values for the ‘Microsoft Virtual System Migration Service’ while I only had FQDN values for CIFS entries which in turn meant that the associated SPN records were missing.

I was originally using a script by Robin CM for this purpose, it appears that it is this script which isn’t quite ticking all of the boxes.

View: Robin CM’s IT Blog – PowerShell: Kerberos Constrained Delegation for Hyper-V Live Migration

 

In my environment, the following represents a corrected version of the script.

The script assumes that you have placed all of your Hypervisor’s in a dedicated OU. The script will obtain a list of all servers in the OU and automatically create the constrained delegation complete with both pairs of the NetBIOS Name and FQDN records.

In addition, the script also now ensures that the system is not adding a constrained delegation back to itself into the AD database.

You must be a domain admin or have permissions to write to msDS-AllowedToDelegateTo objects in AD in order to run this script.

$OU = [ADSI]"LDAP://OU=Hypervisor's,OU=Servers,DC=ad,DC=domain,DC=co,DC=uk"
$DNSSuffix = "ad.domain.co.uk"
$Computers = @{} # Hash tableforeach ($child in $OU.PSBase.Children){
# add each computer in the OU to the hash table
if ($child.ObjectCategory -like '*computer*'){
$Computers.Add($child.Name.Value, $child.distinguishedName.Value)
}
}# Process each AD computer object in the OU in turn
foreach ($ADObjectName in $Computers.Keys){
Write-Host $ADObjectName
Write-Host "Enable VM Live Migration"
Enable-VMMigration -ComputerName $ADObjectName
Write-Host "Set VM migration authentication to Kerberos"
Set-VMHost -ComputerName $ADObjectName -VirtualMachineMigrationAuthenticationType Kerberos
Write-Host "Processing KCD for AD object"
# Add delegation to the current AD computer object for each computer in the OU
foreach ($ComputerName in $Computers.Keys){
#Write-Host $ComputerName.toUpper() $ADObjectName.toUpper()
if ($ComputerName.toUpper() -ne $ADObjectName.toUpper()) {
Write-Host (" Processing "+$ComputerName+", added ") -NoNewline
$ServiceString = "cifs/"+$ComputerName+"."+$DNSSuffix
Set-ADObject -Identity $Computers.$ADObjectName -Add @{"msDS-AllowedToDelegateTo" = $ServiceString}
$ServiceString = "cifs/"+$ComputerName
Set-ADObject -Identity $Computers.$ADObjectName -Add @{"msDS-AllowedToDelegateTo" = $ServiceString}
Write-Host ("cifs") -NoNewline
$ServiceString = "Microsoft Virtual System Migration Service/"+$ComputerName
Set-ADObject -Identity $Computers.$ADObjectName -Add @{"msDS-AllowedToDelegateTo" = $ServiceString}
$ServiceString = "Microsoft Virtual System Migration Service/"+$ComputerName+"."+$DNSSuffix
Set-ADObject -Identity $Computers.$ADObjectName -Add @{"msDS-AllowedToDelegateTo" = $ServiceString}
Write-Host (", Microsoft Virtual System Migration Service")
}
}
}

Once you have run it, give the system a few minutes so that AD can distribute the update to all DC’s and for the Kerberos session on the respective nodes to refresh.

Update for Windows Server 2016

So I decided to reinstall a node to Hyper-V Server 2016 and have a play with it in amongst HyperV Server 2012 R2.

The experience did not go swimmingly well. Here is a quick overview of some issues and I encountered/created myself to keep in mind when troubleshooting this

  1. The Hyper-V server Win32 installer will perform an in-place upgrade as a clean install. Remember that this means that you will need to delete the AD computer account object and DNS records and then re-join the system to the domain in the correct OU.
  2. Once you have done this, you will need to re-create the Kerberos Constrained Delegation records for all Hyper-V nodes
  3. I was experiencing a problem where I could use Kerberos to Live Migrate or offline migrate to the Hyper-V 2016 host, however I could not migrate back unless I logged onto the 2016 node and use CredSSP to move it back again. Looking at the Windows Server 2008 R2 domain controller security logs, Kerberos authentication was failing. In the end the fix was to add a Delegation for CIFS and the ‘Microsoft Virtual Systems Migration’ delegation classes of the computer account object — TO ITSELF. Yes, if you have Computer Accounts HVNode01, HVNode02, HVNode03, the delegation tab for HVNode01 must include CIFS and MVSM entries in DNS and NetBIOS nomenclature for not only HVNode02 and HVNode03 but ALSO HVNode01 (itself). Once I did this, I could magically migrate the VMs back again.
  4. If you are using Jumbo Frames, remember to perform a test using the following command. If it doesn’t work, fix this before doing anything else
    ping <ipAddress> -l 8500 -f
  5. I made a silly mistake in late night PowerShell command entry when setting up the networking on the 2016 box, I entered
    add-vmnetworkadapter -managementos -Name Management

    when I actually meant to enter

    add-vmnetworkadapter -managementos -Name Management -SwitchName VS_Managmement

    This hooked up a new Virtual network adapter on the Hypervisor called ‘Management’ to each and every Virtual Switch on the Hypervisor. So I wound up with 3 NIC’s called Management all on different networks. They went off and got their own IP addresses from DHCP, registered themselves in DNS and created chaos in the adapter binding order. Naturally the one on the unrouted Management network wound up at the top of the binding order and things got a little upset!

  6. The very first randomly selected non-production critical VM that I attempted to migrate was the nodes local console VM. This VM was not designed to move from the node and didn’t have CPU compatibility mode enabled. This caused additional failure issues.
  7. The second randomly selected non-production critical VM that I attempted to migrate gave no hex error code or message what so ever either through the UI or the event log, just throwing Event ID 24024 and stating that the migration failed and the error message could not be found. To cut a long winded story short, in the end I (correctly) assumed it was the VM itself at fault and decided to Export / Import it in order to lazily cycle the file system permissions. It turns out that when I attempted to re-import the VM (as a restore) the import wizard notified me that it was expecting to find a snapshot file but that the snapshot itself was unavailable (this VM had no snapshot on the UI and no snapshot file in the export snapshots folder). The wizard asked me if it could clear the snapshot remnant and imported the VM. Once it was imported again, it could now live migrate and offline migrate properly. It had nothing to do with the 2016 node.Note: Remember to check on the source Hypervisor for remnants of the original Exported VM which may be left in place on the file system.

With the above issues resolved, everything is working correctly between the Hyper-V Server 2012 R2 nodes and the test Hyper-V Server 2016 node.