This article discusses a a bug situation where you cannot backup a Hyper-V Server 2019 machine / Server Core 2019 (?) machine to a VHDX file located on a SMB share. You also cannot format a partition on a remote mounted VHDX file using either NTFS or ReFS.
I have confirmed that this issue does not impact Windows Server 2019 GUI installs.
The Issue
I often backup systems into dynamically expanding VHDX files stored on SMB shares. It is a simple way to use Windows Server Backup, while maintaining versioning – something that cannot be achieved if you backup directly to SMB.
When attempting to backup a Hyper-V Server 2019 system state into a SMB based VHDX. The backup refused to run citing:
ERROR - The specified backup location could not be found or is not a supported backup storage location.
The following were logged in the Event Viewer Application log:
Event ID 12348
Volume Shadow Copy Service warning: VSS was denied access to the root of volume \\?\Volume{<GUID>}\. Denying administrators from accessing volume roots can cause many unexpected failures, and will prevent VSS from functioning properly. Check security on the volume, and try the operation again.
Operation:
Removing auto-release shadow copies
Loading provider
Context:
Execution Context: System Provider
Event ID 12348
Volume Shadow Copy Service warning: VSS was denied access to the root of volume \\?\Volume{<GUID>}\. Denying administrators from accessing volume roots can cause many unexpected failures, and will prevent VSS from functioning properly. Check security on the volume, and try the operation again.
Operation:
Query diff area for this volume
Context:
Volume Name: \\?\Volume{<GUID>}\
Event ID 8193
Volume Shadow Copy Service error: Unexpected error calling routine Error calling CreateFile on volume '\\?\Volume{<GUID>}\'. hr = 0x8000ffff, Catastrophic failure.
Operation:
Query diff area for this volume
Context:
Volume Name: \\?\Volume{<GUID>}\
I was however able to manually browse to the directory structure using CMD as well as PowerShell. I tried both ReFS and NTFS target VHDX’s with the same result. PowerShell listed the drive as present and online using Get-Volume while error checking the VHDX and underlying storage LUN all passed successfully. Using a completely new VHDX file at the target also continued to exhibit the error.
The issue also appeared on every single Hyper-V Server 2019 install that I manage, without exception.
My Environment
The test environment has a combination of Hyper-V Server 2019 and Windows Server 2019 LTSB with GUI machines. The latter backup to VHDX’s on the same SMB share without issue.
The target SMB share where the VHDX’s reside is on a Synology NAS running the latest DSM 6.
Symptoms
Inspecting the Windows Server Backup view of the disk confirmed the reason for the backup failing. Running wbadmin get disks returned:
Disk name: Microsoft Virtual Disk
Disk number: 5
Disk identifier: {c766bd61-d712-44fc-a783-34be73dcfacf}
Total space: 3072.00 GB
Used space : 3071.99 GB
Volumes: There are no volumes on the disk or Windows is unable to retrieve volume
information.
Following up with diskpart list disk correctly showed:
DISKPART> list disk Disk ### Status Size Free Dyn Gpt -------- ------------- ------- ------- --- --- Disk 0 Online 238 GB 0 B * Disk 1 Online 3726 GB 0 B * Disk 4 Online 1860 GB 0 B * Disk 5 Online 3072 GB 1024 KB *
Similarly, running list partition for the disk in question correctly showed the drive layout:
DISKPART> list part Partition ### Type Size Offset ------------- ---------------- ------- ------- Partition 1 Reserved 15 MB 17 KB Partition 2 Primary 3071 GB 16 MB
However running list volume indicated an issue
DISKPART> list vol Volume ### Ltr Label Fs Type Size Status Info ---------- --- ----------- ----- ---------- ------- --------- -------- Volume 0 C NTFS Partition 237 GB Healthy Boot Volume 1 Recovery NTFS Partition 499 MB Healthy Hidden Volume 2 FAT32 Partition 100 MB Healthy System Volume 3 SSD_64K ReFS Partition 3726 GB Healthy C:\4TB-MIR\ Volume 4 SSD_Mirror_ ReFS Partition 1859 GB Healthy C:\2TB-MIR\
The B:\ volume was not present. This despite the volume being fully accessible and usable in CMD/PowerShell and PowerShell’s Get-Volume command listing the B:\ mount successfully.
DriveLetter FriendlyName FileSystemType DriveType HealthStatus OperationalStatus SizeRemaining Size ----------- ------------ -------------- --------- ------------ ----------------- ------------- ---- B Backup NTFS Fixed Healthy OK 3 TB 3 TB
More Info
It would appear that there is a problem in legacy applications reading the Windows API responsible for volume status and management. PowerShell, with its newer code base is able to see the mount, however diskpart and wbadmin are not able to see it. Consequently the backup fails.
The Fix
I encountered the issue on 18th July, with the previous backup on the test server having been made on 12th February. To find the cause I laboriously uninstalled Windows Updates in the following order (rebooting between each) and then re-tested.
NB: Use wusa /uninstall /kb:xxxxxxx to do this on Server Core after running wmic qfe list to view a list of updates.
5003258 4494174 4601558 4589208 4580422 5003646 5003171 5001342
After uninstalling KB5001342 and rebooting, the drive letter reappeared under diskpart list volume and Windows Server Backup / wbadmin ran successfully.
NB: You will not be able to see/uninstall 5001342 (or equivalent cumulative update) until you have uninstalled all superseding updates and rebooted. Uninstall the updates in descending date order until you can see the previous Cumulative Update (e.g. KB5001342) using wmic qfe list.
5001342 is the 13th April 2021 “2021-04 Cumulative Update for Windows Server 2019 for x64-based Systems (KB5001342)”. This update and its successor Cumulative Updates appear to have a defect that is conclusively the cause of the problem. Having rolled Windows back to a March 2021 state on all impacted Hyper-V Server 2019 systems, backups resumed without exception on all hosts.
The fix is thus to revert the system back to a state prior to April 2021.
Side Effects
After devolving some Hyper-V installs from a July 2021 patch level to an April 2021 patch level. I was unable to live migrate from July 2021 to April 2021 systems. Migrations cited "The virtual machine is using processor-specific features not supported on physical computer '<hostname>'". This is not true as all physical nodes are on identical hardware in this case. Please keep this in mind if you are trying to maintain up-times.
Secondly, Remote Desktop connections to some of the Hyper-V Server 2019 systems would not work after uninstalling 5001342.
Re-applying Updates
Once the test machine was working again, I began reapplying the updates.
1> 2020-07 Cumulative Update for .NET Framework 3.5, 4.7.2 and 4.8 for Windows Server 2019 for x64 (KB4566516) 2> 2020-08 Cumulative Update for .NET Framework 3.5, 4.7.2 and 4.8 for Windows Server 2019 for x64 (KB4570505) 3> Security Update for Windows Server 2019 for x64-based Systems (KB4535680) 4> 2021-02 Cumulative Update for .NET Framework 3.5, 4.7.2 and 4.8 for Windows Server 2019 for x64 (KB4601887) 5> 2021-07 Cumulative Update for Windows Server 2019 for x64-based Systems (KB5004244) 6> 2021-07 Cumulative Update for .NET Framework 3.5, 4.7.2 and 4.8 for Windows Server 2019 for x64 (KB5004228) 7> 2021-01 Update for Windows Server 2019 for x64-based Systems (KB4589208)
First, I installed all four of the .net updates, rebooted and confirmed that the drive letter remained visible in diskpart list volume.
Next I installed “2021-01 Update for Windows Server 2019 for x64-based Systems (KB4589208)”, rebooted and confirmed the operation and repeated for “Security Update for Windows Server 2019 for x64-based Systems (KB4535680)”.
Finally it was the turn of “2021-07 Cumulative Update for Windows Server 2019 for x64-based Systems (KB5004244)”. After restarting, Remote Desktop connections came back to life, however diskpart list volume was once again unable to see the mount for the VHDX.
Re-removing 5004244 broke Remote Desktop, but restored VHDX functionality. Proving that the bug is from the hereditary of the mainline “Cumulative Update for Windows Server 2019 for x64-based Systems” monthly patch.
The next step was to manually install “2021-03 Cumulative Update for Windows Server 2019 for x64-based Systems (KB5001638)” from the Microsoft Update Catalogue. Remote Desktop worked after installing this update as did the drive map.
The final step was to manually install “2021-04 Cumulative Update Preview for Windows Server 2019 for x64-based Systems (KB5001384)” from the Microsoft Update Catalogue. The mount point for the VHDX was not visible in diskpart and the backup would not run after applying this update. Uninstalling KB5001384 restored the system to a fully working state empirically proving that the issue entered the production branch of Windows Server after the release branch for KB5001638 was committed.
I have posted about this issue to Microsoft via the following Microsoft boards if you are experiencing a similar issue and wish to engage in the discussion.
Edit 19/07/2021: Bug now reported to https://aka.ms/AAd8q27
Update 16/10/2021: Installing “2021-08 Cumulative Update for Windows Server 2019 for x64-based Systems (KB5005030)” now appears to contain a resolution to this issue.
I can also report that backup process continues to work after the installation of “2021-10 Cumulative Update for Windows Server 2019 for x64-based Systems (KB5006672)“.