Error 0x80070002 when attempting to backup a Hyper-V Virtual Machine using Windows Server Backup

System Requirements:

  • Windows Server 2008 R2
  • Windows Server 2012
  • Windows Server 2012 R2

The Problem:

You backup, right? Of course you do! Only the cool people backup – and you are one of the cool people aren’t you?

…If only life was that simple.

So imagine for a moment that you are attempting to use VVS and WIndows Server Backup to backup a server. In particular a fully loaded Hypervisor running Windows Server 2012 R2 Datacentre in this case.

The backup process goes OK for the most part, but fails to complete on a number (but by no means all) VMs. The process fails with the following errors on Windows Server 2008 VMs, but not necessarily newer ones.

From Windows Server Backup:

Windows Server Backup “Failed” -or- “Completed with warnings” -or-“Backup failed to complete”

The component <VM Name>(Online) was skipped during the snapshot and will not be available for recovery. Error: The writer experienced a non-transient error. If the backup process is retried, the error is likely to reoccur

In the Hyper-V-VMMS\Admin log in Event Log:

‘<VM Name>’ cannot create the storage required for the checkpoint using disk E:\Virtual Machines\<VM Path>\Virtual Hard Disks\<VHD Filename>.vhdx: The system cannot find the file specified. (0x80070002). (Virtual machine ID <VM GUID>)

and…

Checkpoint operation for ‘<VM Name>’ failed. (Virtual machine ID <VM GUID>)

and…

Could not create backup checkpoint for virtual machine ‘<VM Name>’: The system cannot find the file specified. (0x80070002). (Virtual machine ID <VM GUID>)

and of course most helpfully…

The operation failed.

More Info

If you actually look at the backup file, you will see what looks to be a complete file set for the backup, however given that this error represents an error in VSS, you would not be advised to trust it.

As usual with Hyper-V error logs, the error message have little if anything to do with the actual issue and someone in the Microsoft Development team just needs to be shot for it… but I digress.

The odd think was that the issue was occurring on all of the Windows Server 2008 (R1) VMs, the Windows Server 2008 R2 and higher VMs were backing up correctly.

The Fix

So before I get into the issue I encountered, lets run past the generic fixes

  1. Check that you have enough disk space on the volume to perform the VSS. If your volume is sub-15%, try using the Hyper-V Manager to change the snapshot directory to another volume – plug in an external NTFS formatted hard drive if you have to.
  2. Check the permissions of the VHD stated in the error.
    icacls “C:\Users\Public\Documents\Hyper-V\Virtual Hard Disks\<VHD File>” /grant “NT VIRTUAL MACHINE\Virtual Machines”:F /TSource: Technet
    Source: System Center Central
  3. Ensure that Hyper-V is patched fully.
    Windows Server 20102 R2 users see: https://support.microsoft.com/en-us/kb/2920151
  4. Run chkdsk on the physcial volume on the Hypervisor and on the virtual volume in the VM
  5. Ensure that the Integration Service Components are at the latest version and that they VSS Writer module for it is enabled in the VM properties in Hyper-V Manager

Now the less well documented approaches

  1. Check that you can manually checkpoint/snapshot the VM while it is running.
    In Hyper-V Manager or in PowerShell, force a checkpoint on the VM and then delete it and wait for it to merge back. If this works, you are not having a physical VSS issue. If it fails, you need to troubleshoot this and not the WSB error.
  2. Live Migrate the VM off of the current server and onto a different Hypervisor, attempt the backup here, then bring it back to the original server and try again. This process will reset the permissions on the VM file set. If you cannot live or offline migrate the VM, the you need to troubleshoot this and not the WSB error.

My fix

In my case, the issue was to do with having the VM VHDX files split across a couple of different storage LUN/volumes. I usually move VM page files onto a dedicated partition on a dedicated spindle (usually an SSD) and leave OS and data volumes on larger arrays. This helps to keep the VMs running smoothly and keeps unnecessary paging operations off of parity checked storage volumes.

So imagine that the VM has the following file storage structure

Physical Hypervisor SSD (this is where the Page File’s live)
– D:\my-virtual-server-d-drive.vhdx

Physical Hypervisor Storage Array
– E:\Virtual Machines\<VM Name>\Planned Virtual Machines
– E:\Virtual Machines\<VM Name>\Snapshots
– E:\Virtual Machines\<VM Name>\Virtual Hard Drives\my-virtual-server-c-drive.vhdx
– E:\Virtual Machines\<VM Name>\Virtual Hard Drives\my-virtual-server-e-drive.vhdx
– E:\Virtual Machines\<VM Name>\Virtual Machines

It is actually this structure which breaks the WSB backup job. Contrary to the VSS event log error, the problem drive is NOT my-virtual-server-c-drive.vhdx it is actually my-virtual-server-d-drive.vhdx. The event log will actually log that the error was caused on the first drive attached to the system bus (I think).

If you weren’t too clever when you followed my advice above and live migrated all of the storage to the same location on a different Hypervisor, you probably found this out for yourself – the backup should have worked.

When you split the job back into separate LUNs, it fails again. The fix is oddly simple and continues to allow you to have split LUN storage if you wish. Change the file system structure to:

Physical Hypervisor SSD (this is where the Page File’s live)
– D:\<VM Name>\my-virtual-server-d-drive.vhdx

Physical Hypervisor Storage Array
– E:\Virtual Machines\<VM Name>\Planned Virtual Machines
– E:\Virtual Machines\<VM Name>\Snapshots
– E:\Virtual Machines\<VM Name>\Virtual Hard Drives\my-virtual-server-c-drive.vhdx
– E:\Virtual Machines\<VM Name>\Virtual Hard Drives\my-virtual-server-e-drive.vhdx
– E:\Virtual Machines\<VM Name>\Virtual Machines

Note the introduction of a folder on the D drive with the same name as the <VM Name> folder on the E drive. Do NOT shutdown the VM and move the storage there yourself, use the Hyper-V Manager or PowerShell processes to perform a “move” on the ‘storage only’ and just move the one drive. This will ensure that permissions are correct.

The next time that you run the backup, it will VSS correctly.

As for why it can do this on its own with Windows Server 2008 R2 or higher VMs, but not Windows Server 2008 or lower VMs… I have no idea although I suspect it to have something to do with the capabilities of the integration services components.

Edit: A post publish search on the issue reveals that I’m not alone in working this out
View: Technet

Error 0x80070005 when attempting to Perform a Shared Nothing migration between Hyper-V hosts or move a Hyper-V VM between CSV’s in the same or separate Clusters

System Requirements:

  • Windows Server 2012 R2
  • Windows Server 2016

The Problem:

Hyper-V 2012 R2 has a lot of new features that are worthy of note and one of the most appealing features for Virtualisation Administrators is shared nothing migration between hosts via SMB. If you are in an environment that doesn’t have shared storage it’s useful enough in itself because for VM purposes it may have just validated your decision not to get shared storage in the first place. Yet less well documented is the features value for setups where when you do have shared storage as you can use shared nothing migration as a mechanism to live migrate VM’s between clusters that are backed onto shared storage – or more specifically between “Cluster Shared Volumes” (CSV).

The picture on the back of the box of the smiling, happy systems administrator performing a shared nothing administrator makes it look so easy right? This is however an all too common occurrence:

0x80070005 Error

'General access denied error'('0x80070005')

 

There was an error during move operation.

Virtual machine migration operation failed at migration source.

Failed to create folder.

 

There was an error during move operation.

Virtual machine migration operation failed at migration source.

Failed to create folder.

Virtual machine migration operation for ‘<VM Name>’ failed at migration source ‘<Source Hypervisor name>’. (Virtual machine ID <VM-SID>)

Migration did not succeed. Failed to create folder ‘<RPC path>…\Virtual Hard Disks’: ‘General access denied error'(0x80070005’).

If you look at the specified destination path (e.g. c:\ClusterStorage\Volume1\test) after receiving this error, you will find that it has created the test folder and it will have created a ‘Planned Virtual Machines’ folder beneath it which will in turn contain a folder named with the VM’s VM-SID (the Virtual Machines unique security ID) and a .xml file named with the same VM-SID.

The migration will however not progress any further.

If you attempt to perform the same operation in PowerShell you will receive the PowerShell version of the same error:

VERBOSE: Move-VM will move the virtual machine "<VM Name>" to host "<Destination Server>"
Move-VM : Virtual machine migration operation for '<VM Name>' failed at migration source '<Source Server>'. (Virtual machine ID<VM-SID>)
Migration did not succeed. Failed to create folder
'\\<Destination Server>\<Source Server>.762091686$\{e166ba26-8a4a-4029-ac34-c2466451e439}\<VM Name>\Virtual Hard Disks': 'General access denied error'('0x80070005').
You do not have permission to perform the operation. Contact your administrator if you believe you should have permission to perform this operation.
At line:1 char:69
+ $vm = Get-VM -Name 'test' -ComputerName "<Source Server>" | Move-VM -Des ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : PermissionDenied: (Microsoft.Hyper...VMMigrationTask:VMMigrationTask) [Move-VM], Virtual izationOperationFailedException + FullyQualifiedErrorId : AccessDenied,Microsoft.HyperV.PowerShell.Commands.MoveVMCommand

Please Note: This document does not specifically address 0x80070005 for Hyper-V Replication Troubleshooting, which is a slightly different (yet related) issue.

More Info

Understanding the topology involved in my setup also reveals my reason for needing to get this working – this is important as setup and reasons yours may differ slightly. What I was attempting to do was migrate between two multi-node Windows Hyper-V Server 2012 R2 clusters while being able to initiate the migration from a third device, a Windows 8.1 management console.

Much of the discussion surrounding 0x80070005 suggests that you simply need to deal with the fact that you need to log onto the source workstation and initiate a push of the VM from the source server to the destination server using CredSSP. This is fine if you have a general purpose commodity server that happens to have Hyper-V on it. In the real world if you have a Hyper-V Cluster, you should not be running it in GUI mode, you should be using Server Core – and if you are using Windows Hyper-V Server to begin with, you don’t even have the option of a GUI.

So we can eliminate the use of the GUI tools or the simplicity of “just RDP into the server” immediately from this discussion. People answering as such are running in very simple Hyper-V setups and in environments with simple, very liberal security policies.

You can of course use PowerShell to perform a CredSSP migration on a Server Core installation and as a mater of good practice the ability to transfer VM’s using CredSSP should be confirmed as working before you start out with Kerberos. To do that, log onto the Source Server and execute the following command in a PowerShell session:

Get-VM -Name ‘<VM Name To Move>’ | Move-VM -DestinationHost “<Destination Server>” -DestinationStoragePath “C:\ClusterStorage\Volume1\<VM Name to Move>” -Verbose

If that doesn’t work, I recommend that you troubleshoot this issue before you look to go any further on the 0x80070005 issue.

Additionally, before make sure that you have done performed the basic troubleshooting steps and also ensure that you are simplifying the problem as much as possible before starting. The following provides an overview of such steps in no particular order:

  • Log-in as a Domain Admin to perform this test (if possible). After you have that working migrate down to delegated users and troubleshoot any issues that they are experiencing
  • Only try to ‘shared nothing’ migrate a VM that is turned off (create a new VM, attached a default sized dynamically expanding disk, don’t add any networks and leave it off as this means that you will only have 4MB of data to test move). Once you can migrate a VM that is off, attempt to migrate a running VM with a Live Migration.
  • Only test migrate between the Source Cluster storage (CSV) owner node and the Destination Cluster storage owner node
  • If possible, make the owner of the source and destination cluster core resources the same node that owns the CSV
  • Remember that you must use Hyper-V Manager after you have de-clustered the VM from within Failover Cluster Manager before you can perform a shared nothing migration – the fact that your VM has anything to do with a cluster is an aside for Hyper-V. Treat this process as a Hypervisor to Hypervisor move that happens to be on a CSV and forget about the cluster.
  • On the ‘Choose a new location for virtual machine’ page of the migration wizard, remember that you must enter a file system path (e.g. C:\ClusterStorage\volume 1\test) and not a UNC path (e.g. \\server\c$\ClusterStorage\volume 1\test). The migration is going to take place using RPC and not SMB. Thus do not use a UNC path.
    'Choose a new location for virtual machine' wizard page
  • Ensure that you can migrate the VM using CredSSP as discussed at the beginning of this section
  • Ensure that your Domain Controllers are running Windows Server 2008 or higher (or at least your logon server), Windows Server 2003 Domain Controllers are known to have issues here (possibly due to lack of AES support). Your domain / forest functional levels can reportedly be Windows Server 2003 if required. I have only tested with Windows Server 2008 domain functional and Windows Server 2008 forest functional levels
  • If you are attempting to move between servers in a domain trust, you must ensure that the domain trust supports AES
  • Keep your initial testing paths simple and avoid overly complicated NTFS structures. For example, target the destination to be a local sub folder of c:\ and not a junction (such as ClusterStorage\Volume #) or a non-drive letter NTFS Mount Point (i.e. a iSCSI share or drive mount point exposed as a sub-folder to a higher file system). See the links below for more on this.View: Snapshot – General access denied error (0x80070005)
    View: Migrating a Virtual Machine problemNote: The iCACLS command listed in the second link does not use the principal of least permission. The command to enact the principal of least permission would be as follows:

    icacls F:\hvtest /grant “NT VIRTUAL MACHINE\Virtual Machines”:(OI)(CI)(R,RD,RA,REA,WD,AD) /T

    Finally, keep in mind that for delegation purposes, permissions must be valid for the user account that you are using to perform the move as well as the SYSTEM account.

  • Initially, forget about testing the migration into the cluster CSV itself. Instead, create a new folder on the root of the C Drive of the destination server and migrate into this. There are a few suggestions online that you need to put a couple of folder depths between the root of the drive and the VM itself so try something like:
  • C:\VM Store\Test\
  • If you are following my advice, you will be testing with a 4MB VM called ‘test’ so there won’t be any issue with storage space and the use of the C Drive for testing
  • User PowerShell for testing, otherwise you will go insane from having to repeatedly re-enter information in the Move VM wizard. The general gist of the command is:
    Get-VM -Name ‘<VM Name To Move>’ -ComputerName “<Source Server>” | Move-VM -DestinationHost “<Destination Server>” -DestinationStoragePath “C:\ClusterStorage\Volume1\<VM Name to Move>” -Verbose

    With the 0x80070005 error, you should find that it will get to 2% and then error after a few seconds.

  • Ensure that you have enabled Kerberos authenticated Live Migrations in the properties for the Hypervisor in Hyper-V Manager
    Hypervisor PropertiesNote: You can perform this action in PowerShell using

    Enable-VMMigration -ComputerName <Server Hostname>
    Set-VMHost -ComputerName <Server Hostname> -VirtualMachineMigrationAuthenticationType Kerberos
  • Ensure that your Hypervisor’s and the Windows 8.1 management VM are up to date (at the same patch level) and are joined to the same domain
  • Ensure that all parties in the process have properly registered DNS records in AD DNS
  • Check your Windows Firewall rules – for testing purposes just turn them off if you can (remember to turn them back on afterwards!)
  • Check your ASA/Hardware Firewall rules for the same
  • Keep an eye on the Hyper-V event logs for any additional information. The log of consequence is found in event Viewer under:Applications and Services Logs > Microsoft > Windows > Hyper-V-VMMS > AdminIf you are experiencing the same problem that I was, you will see three events on the Source Server’s log (20414, 20770 and 21024). The 20770 error is the one being reflected by PowerShell or the Hyper-V Management console. Shortly there-after, the Destination Server will log a 13003 event informing you that the virtual machine from the Source Server (with the same VM-SID) was deleted, indicating that the Destination Server performed a clean-up of the initial migration process.

Permissions

There is a lot of discussion about permissions and 0x80070005 errors. Let us look at the salient points

VERBOSE: Move-VM will move the virtual machine "<VM Name>" to host "<Destination Server>"
Move-VM : Virtual machine migration operation for '<VM Name>' failed at migration source '<Source Server>'. (Virtual machine ID <VM-SID>)
Migration did not succeed. Failed to create folder
'\\<Destination Server>\<Source Server>.762091686$\{e166ba26-8a4a-4029-ac34-c2466451e439}\<VM Name>\Virtual Hard Disks': 'General access denied error'('0x80070005').
You do not have permission to perform the operation. Contact your administrator if you believe you should have permission to perform this operation.
At line:1 char:69
+ $vm = Get-VM -Name 'test' -ComputerName "<Source Server>" | Move-VM -Des ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : PermissionDenied: (Microsoft.Hyper...VMMigrationTask:VMMigrationTask) [Move-VM], VirtualizationOperationFailedException + FullyQualifiedErrorId : AccessDenied,Microsoft.HyperV.PowerShell.Commands.MoveVMCommand
  1. The Migration failed at the Source Server
  2. The Source Server failed the migration because it could not ‘create a folder
  3. We know that the folder in question is the Source Server being unable to create a ‘<VM Name>\Virtual Hard Disks‘ folder
  4. We know that the Source Server was able to create a ‘<VM Name>\Planned Virtual Machines’ folder because we can see it in the file system if we use the GUI Wizard to perform the migration.
    Note: The PowerShell version cleans up after itself!
  5. You have told the Hypervisor to use Kerberos to perform the migration

What does this tell us? It tells us that YOU, the administrator are being told that you cannot create the folder. You are using Kerberos to perform the migration, not CredSSP, so the entire process is being run end-to-end using YOUR credentials. The Management Workstation is logging onto the Source Server as YOU. The Management Workstation is telling the Source Server to initiate the move and in turn the Source Server is delegating your authentication session to the Destination Server and telling it to receive instructions from the Source Server using your credentials. At this point it has nothing to do with ‘NT Virtual Machine’ or VM-SID permissions, this comes after the migration of the core parts of the VM and during initialisation of the VM on the Destination Server. We are not there yet.

So the first thing to check is that your account is authorised to perform the move. If you are a Domain Admin, you should be OK, however you should ensure that the Domain Admin’s security group is a member of the Local Administrators Group on the all participating machines – source server, destination server and management workstation.

If you do not want the user account to have full local admin rights you can add them to the “Hyper-V Administrators” group on each server. To add an account to a local group on Server Core or Windows Hyper-V Server:

net localgroup "Hyper-V Administrators" /add domain\user
net localgroup "Administrators" /add domain\user

Constrained Delegation

When viewing the Delegation tab on the computer account in Active Directory Users & Computers (ADUC) ensure that:

  1. You are using “Trust this computer for delegation to specified services only” (it doesn’t appear to work if you use the “any service” option)
  2. You have selected “Use Kerberos only”
  3. You tick the ‘Expanded’ checkbox to view the full list of entries
  4. That (once Expanded) there are two entries for each type (types being CIFS and Microsoft Virtual System Migration Service), one entry will have the NetBIOS Name and the other will have the FQDN i.e. there are 4 entries for each delegated host, two with NetBIOS Names and two with FQDN entries.
  5. When you create the Kerberos Constrained Delegation, you need to ensure that the “Service Name” field column is blank. If there is something listed in the Service Name column, your delegation is not going to work properly.
  6. You need to have the same number of “CIFS” entries for each host as you do for “Microsoft Virtual System Migration Service”
  7. It is not necessary to add the Management Workstation to the Constrained Delegation

When you issue the Move-VM command in PowerShell, try substituting the -ComputerName and -DestinationHost values for four combinations of the NetBIOS Name and FQDN.

Get-VM -Name ‘<VM Name To Move>’ -ComputerName “<Source Server>” | Move-VM -DestinationHost “<Destination Server>” -DestinationStoragePath “C:\ClusterStorage\Volume1\<VM Name to Move>” -Verbose

For example, if your have Server1 and Server1 and your domain is domain.local the combinations to test are:

Source Destination
Server1 Server2
Server1.domain.local Server2
Server1 Server2.domain.local
Server1.domain.local Server2.domain.local

If you find that one of these works while the others do not, you have an error in the constrained delegation setup for DNS or NetBIOS aliasing. Carefully recreate the delegation.

After you have setup the delegation, go into a LDAP browser, ADSI Edit or the Attribute Editor in ADUC. For each delegated server, find the servicePrincipalName property and look at the value list. You should have two of each of the following entries (one with the NetBIOS Name and the other with the FQDN).

  • Hyper-V Replica Service/
  • Microsoft Virtual System Migration Service/
  • RestrictedKrbHost/

If you do not see these, you have a Delegation Error and/or an issue in creating SPN records. Either delete and try to recreate them by recreating the delegation or carefully add them by hand.

DNS

Bindings. I know that you checked them, but check them again. Trust me. On Server Core where you have very little contact with the actual server console this is very easy to overlook.

Constrained delegation may work with both NetBIOS and DNS, however Kerberos does not care for NetBIOS. If your DNS doesn’t work, you aren’t going to get a successful ticket session creation that you will need in order to pass credentials forward as part of the Constrained Delegation setup.

Check the following using short hand and FQDN lookups i.e. nslookup server1.domain.local and just nslookup server1. Are they both going where you expect? Crucially, which server NIC is the DNS query going out of and once the reply comes back, which NIC is being used to attempt to contact the host?

  1. The management console can query all domain controllers in DNS
  2. The management console can query all Hypervisors in DNS
  3. The hypervisors can all query the management console in DNS
  4. The hypervisors can all query all domain controllers in DNS
  5. The hypervisors can all query each other in DNS

This also requires you to check your default gateway settings.

This is important in the following scenario. Most of you will not encounter this because of the scale of your operations, however the fact is that at Enterprise level I did encounter this problem, hence why I able to write about it.

  1. Lets assume that you follow best practice and have separate public, management, cluster, iSCSI and heartbeat networks.
  2. Your management network is data centre local, on a private network with minimal routing and is designated to management of servers, IPC traffic, un-routed VM’s etc in a secure fashion
  3. Local DNS is available on the management network but does not expose Internet Resolution
  4. Your public VM address ranges come from the public network and are not exposed via NAT/PAT i.e. routing and firewall’s
  5. Your domain controllers exist on a public routed network subnet that is separate from the public VM address ranges used for VM’s
  6. You followed best practice and set your management networks binding order to be the first adapter in the binding order on the hypervisors
  7. You will now receive 0x80070005 when you attempt to replicate, live migrate of off-line migrate a VM between cluster nodes using Kerberos Constrained Delegation

The problem is the adapter binding order caused by the use of local DNS on a network that offers no connectivity to the domain controllers. When the KDC attempts to generate a Kerberos ticket for the constrained delegation, the lookups for the domain controllers will be performed using the DNS servers on the management network and will mistakenly attempt to connect to the domain controllers via the management network. This is simply going to time out – causing the wait during migration. Once it times out, Windows DNS doesn’t defer to the next set of DNS servers or attempt to get to the DC’s on a different NIC. It simply gives up.

The resulting very helpful error code that Hyper-V offers back is Access Denied while seemingly attempting to create files in the file system – the Hypervisor will log that it was unable to create the ‘Virtual Hard Drives’ folder on the destination Hypervisor. What it should actually say here is that it could not properly initialise the end to end Kerberos Constrained Delegation ticket session due to a timeout. It of course doesn’t do that.

In this situation the fixes are one of:

  1. Add an interface on the domain controllers on the management LAN
  2. Add a network interface which can connect to the domain controllers in a higher adapter binding order position in the Hypervisor binding order
  3. Remove the DNS servers from the management networks TCP/IP properties, thus forcing Windows Server to use the first available DNS server configuration on a lower ordinal adapter
  4. Allow routing from the management LAN to the domain controllers. Alias, stub or secondary zone the domain controllers in the management networks DNS and hope you remember to keep them up to date when you make changes to Domain Controller DNS records

Assuming that your constrained delegations are correct, it will start working as soon as the DNS updates have propagated.

The Fix

Ultimately the problem that I had was in the setup of the Constrained Delegation and in another case as discussed above, the DNS binding order. For the Constrained Delegation issueI only had NetBIOS values for the ‘Microsoft Virtual System Migration Service’ while I only had FQDN values for CIFS entries which in turn meant that the associated SPN records were missing.

I was originally using a script by Robin CM for this purpose, it appears that it is this script which isn’t quite ticking all of the boxes.

View: Robin CM’s IT Blog – PowerShell: Kerberos Constrained Delegation for Hyper-V Live Migration

 

In my environment, the following represents a corrected version of the script.

The script assumes that you have placed all of your Hypervisor’s in a dedicated OU. The script will obtain a list of all servers in the OU and automatically create the constrained delegation complete with both pairs of the NetBIOS Name and FQDN records.

In addition, the script also now ensures that the system is not adding a constrained delegation back to itself into the AD database.

You must be a domain admin or have permissions to write to msDS-AllowedToDelegateTo objects in AD in order to run this script.

$OU = [ADSI]"LDAP://OU=Hypervisor's,OU=Servers,DC=ad,DC=domain,DC=co,DC=uk"
$DNSSuffix = "ad.domain.co.uk"
$Computers = @{} # Hash tableforeach ($child in $OU.PSBase.Children){
# add each computer in the OU to the hash table
if ($child.ObjectCategory -like '*computer*'){
$Computers.Add($child.Name.Value, $child.distinguishedName.Value)
}
}# Process each AD computer object in the OU in turn
foreach ($ADObjectName in $Computers.Keys){
Write-Host $ADObjectName
Write-Host "Enable VM Live Migration"
Enable-VMMigration -ComputerName $ADObjectName
Write-Host "Set VM migration authentication to Kerberos"
Set-VMHost -ComputerName $ADObjectName -VirtualMachineMigrationAuthenticationType Kerberos
Write-Host "Processing KCD for AD object"
# Add delegation to the current AD computer object for each computer in the OU
foreach ($ComputerName in $Computers.Keys){
#Write-Host $ComputerName.toUpper() $ADObjectName.toUpper()
if ($ComputerName.toUpper() -ne $ADObjectName.toUpper()) {
Write-Host (" Processing "+$ComputerName+", added ") -NoNewline
$ServiceString = "cifs/"+$ComputerName+"."+$DNSSuffix
Set-ADObject -Identity $Computers.$ADObjectName -Add @{"msDS-AllowedToDelegateTo" = $ServiceString}
$ServiceString = "cifs/"+$ComputerName
Set-ADObject -Identity $Computers.$ADObjectName -Add @{"msDS-AllowedToDelegateTo" = $ServiceString}
Write-Host ("cifs") -NoNewline
$ServiceString = "Microsoft Virtual System Migration Service/"+$ComputerName
Set-ADObject -Identity $Computers.$ADObjectName -Add @{"msDS-AllowedToDelegateTo" = $ServiceString}
$ServiceString = "Microsoft Virtual System Migration Service/"+$ComputerName+"."+$DNSSuffix
Set-ADObject -Identity $Computers.$ADObjectName -Add @{"msDS-AllowedToDelegateTo" = $ServiceString}
Write-Host (", Microsoft Virtual System Migration Service")
}
}
}

Once you have run it, give the system a few minutes so that AD can distribute the update to all DC’s and for the Kerberos session on the respective nodes to refresh.

Update for Windows Server 2016

So I decided to reinstall a node to Hyper-V Server 2016 and have a play with it in amongst HyperV Server 2012 R2.

The experience did not go swimmingly well. Here is a quick overview of some issues and I encountered/created myself to keep in mind when troubleshooting this

  1. The Hyper-V server Win32 installer will perform an in-place upgrade as a clean install. Remember that this means that you will need to delete the AD computer account object and DNS records and then re-join the system to the domain in the correct OU.
  2. Once you have done this, you will need to re-create the Kerberos Constrained Delegation records for all Hyper-V nodes
  3. I was experiencing a problem where I could use Kerberos to Live Migrate or offline migrate to the Hyper-V 2016 host, however I could not migrate back unless I logged onto the 2016 node and use CredSSP to move it back again. Looking at the Windows Server 2008 R2 domain controller security logs, Kerberos authentication was failing. In the end the fix was to add a Delegation for CIFS and the ‘Microsoft Virtual Systems Migration’ delegation classes of the computer account object — TO ITSELF. Yes, if you have Computer Accounts HVNode01, HVNode02, HVNode03, the delegation tab for HVNode01 must include CIFS and MVSM entries in DNS and NetBIOS nomenclature for not only HVNode02 and HVNode03 but ALSO HVNode01 (itself). Once I did this, I could magically migrate the VMs back again.
  4. If you are using Jumbo Frames, remember to perform a test using the following command. If it doesn’t work, fix this before doing anything else
    ping <ipAddress> -l 8500 -f
  5. I made a silly mistake in late night PowerShell command entry when setting up the networking on the 2016 box, I entered
    add-vmnetworkadapter -managementos -Name Management

    when I actually meant to enter

    add-vmnetworkadapter -managementos -Name Management -SwitchName VS_Managmement

    This hooked up a new Virtual network adapter on the Hypervisor called ‘Management’ to each and every Virtual Switch on the Hypervisor. So I wound up with 3 NIC’s called Management all on different networks. They went off and got their own IP addresses from DHCP, registered themselves in DNS and created chaos in the adapter binding order. Naturally the one on the unrouted Management network wound up at the top of the binding order and things got a little upset!

  6. The very first randomly selected non-production critical VM that I attempted to migrate was the nodes local console VM. This VM was not designed to move from the node and didn’t have CPU compatibility mode enabled. This caused additional failure issues.
  7. The second randomly selected non-production critical VM that I attempted to migrate gave no hex error code or message what so ever either through the UI or the event log, just throwing Event ID 24024 and stating that the migration failed and the error message could not be found. To cut a long winded story short, in the end I (correctly) assumed it was the VM itself at fault and decided to Export / Import it in order to lazily cycle the file system permissions. It turns out that when I attempted to re-import the VM (as a restore) the import wizard notified me that it was expecting to find a snapshot file but that the snapshot itself was unavailable (this VM had no snapshot on the UI and no snapshot file in the export snapshots folder). The wizard asked me if it could clear the snapshot remnant and imported the VM. Once it was imported again, it could now live migrate and offline migrate properly. It had nothing to do with the 2016 node.Note: Remember to check on the source Hypervisor for remnants of the original Exported VM which may be left in place on the file system.

With the above issues resolved, everything is working correctly between the Hyper-V Server 2012 R2 nodes and the test Hyper-V Server 2016 node.

Error 0x80070490 or 0x00000490 when attempting to connect to a Printer queue on a Windows Print Server

System Requirements:

  • Windows Vista, 7, 8, 8.1,10
  • Windows Server 2008, 2008 R2, 2012, 2012 R2

The Problem:

I was having some problems automating the connection to a printer queue from a set of managed Windows 7 systems to foreign Windows Server managed SafeCom printer queue. The device in question was a generic follow-me Printing queue for Xerox Workcentre 7655 devices (not that it is especially relevant).

On opening the SMB share to the print server and connecting to the printer queue, the system would go off for 60 seconds before coming back with error 0x00000490 and no description.

Exploration of this error in event viewer under Event Viewer > Applications and Services > Microsoft > Windows > PrintService > Admin reveals:

Installing printer driver Xerox GPD PS V3.2.303.16.0 failed, error code 0x490, HRESULT 0x80070490. See the event user data for context information.

The only additional information available in the user data of any substance was either

Parse Inf
ProcessDriverDependencies failed

or

PerformInfInstallActions
ParseInf failed

More Info

I also tried the following recommendations from general troubleshooting/elsewhere:

  1. Attempting to manually install the driver didn’t help
  2. Using pnputil -d to delete the driver oemXX.inf didn’t help (i.e. clearing the driver out of C:\Windows\System32\DriverStore\FileRepository)
  3. Using pnputil -a to manually add the desired driver didn’t help
  4. Using the Print Management MMC snap-in to flush the driver out (including renaming any reference dll’s under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Print\Environments\Windows x64\Print Processors\winprint to xxx.old and restarting the print server to allow the deletion of stuck print drivers) didn’t help
  5. Obtaining older and new drivers from Xerox didn’t help
  6. The Microsoft Printer Troubleshooter tool
    View: Fixing Printer Problems
  7. Enabling the Operational log under Event Viewer > Applications and Services > Microsoft > Windows > PrintService didn’t add anything significant to the troubleshooting process just:
    ParseInfAndCommitFileQueue
    PerformInfInstallActions failedProcessDriverDependencies
    FindLatestCoreDrivers failed
  8. Checking setupapi.app.log and setupapi.dev.log for errors under C:\Windows\inf did not show any errors, everything was reporting ‘success’

The Fix

In downloading and installing the older version of the print driver, event viewer was showing an identical error for the driver install to that shown when displaying the error for the install of the most up to date driver version, however comparing the source driver files to the destination files that appeared after repository injection in C:\Windows\System32\DriverStore\FileRepository revealed some slight differences in file dates, the print server was sending slightly modified versions of the driver compared to the vanilla Xerox source of the same version number (2015 file dates for a 2013 Xerox driver package).

Some sleuthing through monitoring tools ultimately presented the cause of the issue and ultimately its fix. Windows was downloading the driver package from the target foreign print server (with its modified files) and injecting it into the repository correctly. Immediately afterwards however it was going off to our internal, public driver repository (a SMB share on a build server) and finding additional copies of a compatible x64 Xerox driver, finding that they were newer and then attempting to use the newer driver.

Without the driver customisation’s (presumably part of the SafeCom suite configuration for follow-me printing) the print server was immediately rejecting the connection.

So the lesson from this experience was that even if you are explicitly telling Windows to use a specific driver version, if it can find a newer version in a driver search path, it will attempt to pick it up and use it instead. Remove any media with drivers (UFD, CD, DVD, Floppy) and check/modify you driver search paths for conflicting drivers as listed in:

HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\DevicePath

0x80090020 when attempting to load a .PFX Private Key Certificate into a CAPICOM_MEMORY_STORE using Store.Load() or Certificate.Load() using CAPICOM 2.1.0.2

System Requirements:

  • Windows Server 2008, R2
  • Windows Vista
  • Windows 7
  • Windows 8, 8.1
  • 7.0, 7.5, 8.0
  • ASP 3.0 (Classic)
  • CAPICOM 2.1.0.2

The Problem:

Ah encryption, that most noble of things. One thing that is sure to drive every developer close to the brink on the odd occasion. The one time where clear, concise API documentation should be considered mandatory – and the one place where good API documentation it seems is an obligation itself not to provide. Be it Microsoft, Java, BouncyCastle, PHP it would seem they are all blighted with the same issue.

Attempting to use legacy API on an unsupported platform should seem like an exercise in masochism, however, you know how much I like to avoid using .net whenever I can.

If you attempt to do this

Dim cert
set cert = Server.CreateObject("CAPICOM.Certificate")
call cert.load("c:\myPrivateKey.pfx", "test", CAPICOM_KEY_STORAGE_EXPORTABLE)

or this

const CAPICOM_MEMORY_STORE = 0
const CAPICOM_LOCAL_MACHINE_STORE = 1
const CAPICOM_STORE_OPEN_READ_WRITE = 1
const CAPICOM_KEY_STORAGE_EXPORTABLE = 1Dim store
set store = Server.CreateObject("CAPICOM.Store")
call store.Open(CAPICOM_MEMORY_STORE, "MemoryStore1", CAPICOM_STORE_OPEN_READ_WRITE)
call store.load("c:\myPrivateKey.pfx", "test", CAPICOM_KEY_STORAGE_EXPORTABLE)

you will get back

error '80090020'
/file.asp, line ###

If you send in a .cer file instead of a .pfx, it works without error but doesn’t allow you to access the Private Key.

More Info

Taking the two code samples in order

Dim cert
set cert = Server.CreateObject("CAPICOM.Certificate")
call cert.load("c:\myPrivateKey.pfx", "test", CAPICOM_KEY_STORAGE_EXPORTABLE)

Should you be getting a 0x80070056 error, your password is wrong. If the file doesn’t have a password, only send parameter 1 (which is about to cause you a problem). To resolve the 0x80090020 error while using a CAPICOM_MEMORY_STORE, you need to stop CAPICOM from attempting to insert the certificate as a resource for a user. If the IIS worker process that you are using doesn’t connect to a user account and has no permissions, the default parameter CAPICOM_CURRENT_USER_KEY or 0 will throw 0x80090020.

To change the scope, ensure that you use the fourth parameter and set the value to CAPICOM_LOCAL_MACHINE_KEY.

const CAPICOM_CURRENT_USER_KEY = 0
const CAPICOM_LOCAL_MACHINE_KEY = 1Dim cert
set cert = Server.CreateObject("CAPICOM.Certificate")
call cert.load("c:\myPrivateKey.pfx", "test", CAPICOM_KEY_STORAGE_EXPORTABLE, CAPICOM_LOCAL_MACHINE_KEY)

To resolve the second issue, modify the original code to make use of the now fixed certificate.load() call and import it vie the long route.

const CAPICOM_MEMORY_STORE = 0
const CAPICOM_LOCAL_MACHINE_STORE = 1
const CAPICOM_STORE_OPEN_READ_WRITE = 1
const CAPICOM_KEY_STORAGE_EXPORTABLE = 1Dim cert
Dim store
set store = Server.CreateObject("CAPICOM.Store")
call store.Open(CAPICOM_MEMORY_STORE, "MemoryStore1", CAPICOM_STORE_OPEN_READ_WRITE)set cert = Server.CreateObject("CAPICOM.Certificate")
call cert.load("c:\myPrivateKey.pfx", "test", CAPICOM_KEY_STORAGE_EXPORTABLE, CAPICOM_LOCAL_MACHINE_KEY)

call store.add(cert)

If you receive 0x80070005, you are either getting an Access Denied error to the MEMORY_STORE or you are attempting to import a certificate into the instantiated store which already exists. Similarly, if you receive 0x80070056, your password is wrong.