“RPC server unavailable. Unable to establish communication between and ” when connecting to Hyper-V 2008, 2008 R2, 2012, 2012 R2 from Hyper-V Manager version 1709

System Requirements:

  • Windows 10 1709
  • Windows Server 2016
  • Hyper-V Management Console
  • RSAT 2016/1709 for Windows 10 version 1709

The Problem:

After upgrading to Windows 10 version 1709 and installing the updated Windows Server 2016 (version 2016 or version 1709) RSAT tools for Windows 10 1709. On attempting to connect to a down-level Windows Server 2012 R2, 2012, 2008 R2, 2008 Hyper-V Server via the Hyper-V Manager MMC snap-in. You receive the error even though no configuration changes have been made on the Hyper-V hosts:

"RPC server unavailable. Unable to establish communication between <management host> and <Hyper-V host>"

At this point you are unable to manage down-level version of Hyper-V from Windows 10. This issue does not impact the management of remote Windows Server 2016 or Windows Server 1709 Hyper-V instances.

View: Remote Server Administration Tools for Windows 10 (RSAT)

The Fix

This appears to be related to a change in the default firewall behaviour on Windows 10 1709 installs. to fix the problem. On the client system, where you have installed RSAT to remote manage the hypervisor (i.e. not on the hypervisor itself):

  1. Open ‘Administrative Tools’ in the Windows Control Panel
  2. Open ‘Windows Defender Firewall with Advanced Security’
  3. Select ‘Inbound Rules’ from the left hand side
  4. Scroll down until you get to ‘Windows Management Instrumentation (ASync-In)’
  5. Enable the rule for domain/private/public networks as required
    Note: By default the Windows firewall MMC will only display WMI rules for domain and private networks. If you are not running against a domain and Windows has not been explicitly told that you are on a private network, Windows will assume that you are on a public network. Check in network settings in the settings app to ensure that you are not running on a public network, or if you are edit the firewall rule to include public networks. In general, it is a bad idea to open WMI up to traffic on public networks.
  6. Restart Hyper-V Manager

You should now find that you can connect to down-level versions of Hyper-V from Windows 10 1709.

“Some migration network settings cannot be modified or removed because these migration network settings are in use by a cluster” in Hyper-V manager, Hyper-V Settings after a node is non-cleanly evicted from a failed cluster

System Requirements:

  • Windows Server 2008, 2008 R2, 2012, 2012 R2, 2016
  • Hyper-V

The Problem:

After a total cluster failure occurs, or a node is removed from a cluster before it has been cleanly removed. When attempting to locally manage the host as a stand-alone Hyper-V server, you are unable to edit the live migration settings in Hyper-V Settings via Hyper-V Manager with the error

“Some migration network settings cannot be modified or removed because these migration network settings are in use by a cluster”

The IP/Subnet addresses shown on the form will be greyed out and you will be unable to edit the live migration network settings.

Hyper-V Settings: Unable to edit Live Migration Networks

More Info

While you should ensure that you have performed a cluster clean-up on the host

Server 2012 +:
Clear-ClusterNode -Force -CleanupA

Server 2008/R2:
cluster node <hostname> /forcecleanup

This will not solve the live migration settings issue highlighted above.

The Fix

  1. Close Hyper-V Manager on the management workstation
  2. On the hypervisor experiencing these symptoms (not necessarily the management console). Navigate to HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Virtualization\Migration\NetworkSettings
  3. Underneath this key there will be a key for each entry shown on the Live Migration Settings screen. The keys will be named Network# e.g. Network0, Network1, Network2 and so on.
  4. Go through each of these Network# keys and locate the ‘Tags’ REG_MULTI_SZ. This will have a value of “Microsoft:ClusterManaged”
    Registry Screenshot: Error
  5. Change the data value to “Microsoft:UserManaged”
    Registry Settings: Fixed
  6. Complete the process for each Network# sub-key
  7. Restart Hyper-V Manager

You will now be able to add, edit and delete the live migration settings.

Hyper-V Discrete Device Assignment (DDA) with a TV Tuner (Hauppauge HVR-4400)

System Requirements:

  • Windows Server 2016
  • Hauppauge HVR-4400 PCIe Tuner

The Problem:

I am a DVBLink user. DVBLink does not play nicely with Windows Service and consequently it wants to run on a client OS. This means that I have lots of server hardware running server Operating Systems and one device with 4 TV Tuners in it running Windows 10.

With the release of Windows Server 2016 came the promise of VMWare like PCIe Pass-through, allowing physical devices on the PCI bus to be attached to VMs. The plan is to attach the PCIe TV Tuner and attempt to get DVBLink working in a VM so that the physical unit can be decommissioned (saving the power bill).

More Info

As part of the process, I was considering building a new server at the start of 2017 to perform a consolidation against. The Windows 10 DVBLink machine would be one consolidated devices onto more powerful modern hardware. I would also need new TV Tuners as only 2 of the 4 in the DVBLink TV Server is PCIe, the rest are PCI. Again, there are opportunities to consolidate that into fewer PCIe devices too.

The driver for the new server was Hyper-V PCIe Pass-through, or “Discrete Device Assignment” (DDA) as Microsoft are calling it. It is however quite difficult to find out whether BIOS firmware supports the proper implementations of I/O-MMU VT-d to permit it, making the purchase a risk. Equally, there is no guarantee that DDA will work with a TV Tuner.

Consequently, I decided to borrow a dual CPU Dell PowerEdge R630 to perform the experiment as there were several reports on-line that the R6xx and R7xx have the proper VT-d and SR-IOV feature set for this type of activity. Well done Dell (why don’t you advertise this?!).

After updating firmware, adding the TV Tuner and installing Windows Hyper-V Server 2016 on the machine, the first step was to – as an experiment – attempt to install the TV Tuner drivers on Windows Server 2016 (which errored). After that it was time to run the DDA Survey Script from Microsoft.

Download: DDA Survey Script (GitHub)

 

This was promising. The script found two devices that it stated were capable of being used with DDA

PERC H730 Mini
Express Endpoint -- more secure.
And its interrupts are message-based, assignment can work.
PCIROOT(0)#PCI(0100)#PCI(0000)

and

Hauppauge WinTV HVR-4400 (Model 121xxx, Hybrid DVB-T/S2, IR)
Express Endpoint -- more secure.
And it has no interrupts at all -- assignment can work.
PCIROOT(0)#PCI(0200)#PCI(0000)

The next step was to dismount the device from the Hypervisor and make it available to Hyper-V

# Find the HVR-4400
$pnpdevs = Get-PnpDevice -PresentOnly | Where-Object {$_.Class -eq "Media"} | Where-Object {$_.Service -eq "HCW85BDA"}# ... or if you know the hardware ID
$pnpdevs = Get-PnpDevice -PresentOnly | Where-Object {$_.InstanceId -eq "PCI\VEN_14F1&DEV_888
0&SUBSYS_C1080070&REV_04\4&39CDA168&0&0010"}foreach ($pnpdev in $pnpdevs) {
Disable-PnpDevice -InstanceId $pnpdev.InstanceId -Confirm:$false
Write-Host 'Device ' $pnpdev.InstanceId ' Disabled. NOTE: If this hangs, reboot and try again'
$instanceId = $pnpdev.InstanceId
$locationpath = ($pnpdev | get-pnpdeviceproperty DEVPKEY_Device_LocationPaths).data[0]
Write-Host 'Dismounting Device At: ' $locationpath ' (' $instanceId ')'
Dismount-VmHostAssignableDevice -locationpath $locationpath
Write-Host $locationpath
}

Initially, it hung PowerShell (and the system) so I had to hard reset the server. In this instance it was in fact necessary to reboot after issuing

Disable-PnpDevice

After trying again and rebooting the Dismount-VmHostAssignableDevice failed with

dismount-vmhostassignabledevice : The operation failed.
The manufacturer of this device has not supplied any directives for securing this device while exposing it to a
virtual machine. The device should only be exposed to trusted virtual machines.
This device is not supported when passed through to a virtual machine.
The operation failed.
The manufacturer of this device has not supplied any directives for securing this device while exposing it to a
virtual machine. The device should only be exposed to trusted virtual machines.
This device is not supported and has not been tested when passed through to a virtual machine. It may or may not
function. The system as a whole may become unstable. The device should only be exposed to trusted virtual machines.
At line:1 char:1

It would not proceed past this point. The trick was to change the line to

Dismount-VmHostAssignableDevice -locationpath $locationpath -Force

The next step was to ensure that the VM’s Automatic Stop Action was set to anything other than “Save”

Set-VM -Name “10-TEST” -AutomaticStopAction Shutdown

… and at this point it was simply a case of creating a VM and assigning the device

Add-VMAssignableDevice -LocationPath “$locationpath” -VMName “10-Test”

At which point the device immediately popped up in Device Manager under Windows 10 in the Generation 2 VM

DDA PCIe Passthrough in Device Manager

…. before the VM blue screened a few seconds later.

Blue Screen of Death

I tried to use several versions of the HVR-4400 driver that I could find and it made no difference. The VM would crash whenever it attempted to talk to the card. The Hypervisor itself did not seem to be impacted by the Blue Screen event and itself did not crash.

I also tried fully removing the device from the Hypervisor using DEVCON and clearing out the driver using pnputil. Superficially, this action made it worse as the VM wouldn’t boot at all now if it had a driver on-file for the TV Tuner. Before it would at least boot.

So this project was a failure and I will not be investing in new server hardware just yet. I’ll wait to see if Microsoft improve the feature set as allegedly this type of insanity (and yes, it is insane) is possible in VMWare. I do not want to change away from Hyper-V at the current time though, so I will have to stick with a client machine as a service.

This does not mean of course that this cannot work in Hyper-V. The HVR-4400 is a card from 2011/2012. So it is not exactly new hardware. PCIe TV Tuners designed to modern electrical standards and for use on PCIe 3.0 bus architectures may provide better interoperability out of the box. I just don’t have any other cards to test with and am in a bit of a chicken and egg situation over wanting to invest in new cards and servers unless I know they will work nicely together.

If you are interested in this too and would like me to have a go testing your hardware, please get in touch via HPC:Factor.

Error 0x80070002 when attempting to backup a Hyper-V Virtual Machine using Windows Server Backup

System Requirements:

  • Windows Server 2008 R2
  • Windows Server 2012
  • Windows Server 2012 R2

The Problem:

You backup, right? Of course you do! Only the cool people backup – and you are one of the cool people aren’t you?

…If only life was that simple.

So imagine for a moment that you are attempting to use VVS and WIndows Server Backup to backup a server. In particular a fully loaded Hypervisor running Windows Server 2012 R2 Datacentre in this case.

The backup process goes OK for the most part, but fails to complete on a number (but by no means all) VMs. The process fails with the following errors on Windows Server 2008 VMs, but not necessarily newer ones.

From Windows Server Backup:

Windows Server Backup “Failed” -or- “Completed with warnings” -or-“Backup failed to complete”

The component <VM Name>(Online) was skipped during the snapshot and will not be available for recovery. Error: The writer experienced a non-transient error. If the backup process is retried, the error is likely to reoccur

In the Hyper-V-VMMS\Admin log in Event Log:

‘<VM Name>’ cannot create the storage required for the checkpoint using disk E:\Virtual Machines\<VM Path>\Virtual Hard Disks\<VHD Filename>.vhdx: The system cannot find the file specified. (0x80070002). (Virtual machine ID <VM GUID>)

and…

Checkpoint operation for ‘<VM Name>’ failed. (Virtual machine ID <VM GUID>)

and…

Could not create backup checkpoint for virtual machine ‘<VM Name>’: The system cannot find the file specified. (0x80070002). (Virtual machine ID <VM GUID>)

and of course most helpfully…

The operation failed.

More Info

If you actually look at the backup file, you will see what looks to be a complete file set for the backup, however given that this error represents an error in VSS, you would not be advised to trust it.

As usual with Hyper-V error logs, the error message have little if anything to do with the actual issue and someone in the Microsoft Development team just needs to be shot for it… but I digress.

The odd think was that the issue was occurring on all of the Windows Server 2008 (R1) VMs, the Windows Server 2008 R2 and higher VMs were backing up correctly.

The Fix

So before I get into the issue I encountered, lets run past the generic fixes

  1. Check that you have enough disk space on the volume to perform the VSS. If your volume is sub-15%, try using the Hyper-V Manager to change the snapshot directory to another volume – plug in an external NTFS formatted hard drive if you have to.
  2. Check the permissions of the VHD stated in the error.
    icacls “C:\Users\Public\Documents\Hyper-V\Virtual Hard Disks\<VHD File>” /grant “NT VIRTUAL MACHINE\Virtual Machines”:F /TSource: Technet
    Source: System Center Central
  3. Ensure that Hyper-V is patched fully.
    Windows Server 20102 R2 users see: https://support.microsoft.com/en-us/kb/2920151
  4. Run chkdsk on the physcial volume on the Hypervisor and on the virtual volume in the VM
  5. Ensure that the Integration Service Components are at the latest version and that they VSS Writer module for it is enabled in the VM properties in Hyper-V Manager

Now the less well documented approaches

  1. Check that you can manually checkpoint/snapshot the VM while it is running.
    In Hyper-V Manager or in PowerShell, force a checkpoint on the VM and then delete it and wait for it to merge back. If this works, you are not having a physical VSS issue. If it fails, you need to troubleshoot this and not the WSB error.
  2. Live Migrate the VM off of the current server and onto a different Hypervisor, attempt the backup here, then bring it back to the original server and try again. This process will reset the permissions on the VM file set. If you cannot live or offline migrate the VM, the you need to troubleshoot this and not the WSB error.

My fix

In my case, the issue was to do with having the VM VHDX files split across a couple of different storage LUN/volumes. I usually move VM page files onto a dedicated partition on a dedicated spindle (usually an SSD) and leave OS and data volumes on larger arrays. This helps to keep the VMs running smoothly and keeps unnecessary paging operations off of parity checked storage volumes.

So imagine that the VM has the following file storage structure

Physical Hypervisor SSD (this is where the Page File’s live)
– D:\my-virtual-server-d-drive.vhdx

Physical Hypervisor Storage Array
– E:\Virtual Machines\<VM Name>\Planned Virtual Machines
– E:\Virtual Machines\<VM Name>\Snapshots
– E:\Virtual Machines\<VM Name>\Virtual Hard Drives\my-virtual-server-c-drive.vhdx
– E:\Virtual Machines\<VM Name>\Virtual Hard Drives\my-virtual-server-e-drive.vhdx
– E:\Virtual Machines\<VM Name>\Virtual Machines

It is actually this structure which breaks the WSB backup job. Contrary to the VSS event log error, the problem drive is NOT my-virtual-server-c-drive.vhdx it is actually my-virtual-server-d-drive.vhdx. The event log will actually log that the error was caused on the first drive attached to the system bus (I think).

If you weren’t too clever when you followed my advice above and live migrated all of the storage to the same location on a different Hypervisor, you probably found this out for yourself – the backup should have worked.

When you split the job back into separate LUNs, it fails again. The fix is oddly simple and continues to allow you to have split LUN storage if you wish. Change the file system structure to:

Physical Hypervisor SSD (this is where the Page File’s live)
– D:\<VM Name>\my-virtual-server-d-drive.vhdx

Physical Hypervisor Storage Array
– E:\Virtual Machines\<VM Name>\Planned Virtual Machines
– E:\Virtual Machines\<VM Name>\Snapshots
– E:\Virtual Machines\<VM Name>\Virtual Hard Drives\my-virtual-server-c-drive.vhdx
– E:\Virtual Machines\<VM Name>\Virtual Hard Drives\my-virtual-server-e-drive.vhdx
– E:\Virtual Machines\<VM Name>\Virtual Machines

Note the introduction of a folder on the D drive with the same name as the <VM Name> folder on the E drive. Do NOT shutdown the VM and move the storage there yourself, use the Hyper-V Manager or PowerShell processes to perform a “move” on the ‘storage only’ and just move the one drive. This will ensure that permissions are correct.

The next time that you run the backup, it will VSS correctly.

As for why it can do this on its own with Windows Server 2008 R2 or higher VMs, but not Windows Server 2008 or lower VMs… I have no idea although I suspect it to have something to do with the capabilities of the integration services components.

Edit: A post publish search on the issue reveals that I’m not alone in working this out
View: Technet