“The system image restore failed. Error details: Incorrect function (0x80070001)” when attempting to restore a Windows Server Backup / Windows Backup Image over the network from Windows Backup or bootable recovery media

System Requirements:

  • Windows Vista, 7, 8, 8.1, 10
  • Windows Server 2008, 2008 R2, 2012, 2012 R2, 2016

The Problem:

If you attempt to restore a Windows Server Backup or Windows Backup recovery image (system state) from a bootable recovery media (either DVD, USB or a recovery partition) you receive the following error message

The system image restore failed.

Error details: Incorrect function (0x80070001)

More Info

Just to keep this simple. The limited amount of information available on-line on this error suggests that either the image is corrupt or you are having a network reliability issue.

These could be true. In my case, it wa the path length that I was using to recover the image from. I was recovering from

\\192.168.15.106\BackupFiles\Servers\2012 R2\WindowsImageBackup\<hostname>\…

It turns out that the entire path must be 110 or fewer characters otherwise it will fail with the 0x80070001 error.

A simple fix is to temporarily create the path \\192.168.15.106\BackupFiles\WindowsImageBackup on your share and then copy the <hostname> folder into it. This worked in my case.

Netgear ReadyNAS Duo v2 as a Windows Server Backup Target across SMB while allowing differencing in the backup type

System Requirements:

  • Netgear ReadyNAS Duo v2 or any SMB capable NAS
  • Windows Server 2008, 2008 R2, 2012, 2012 R2

The Problem:

One of the most frustrating “features” of Windows Server since the release of Windows Server 2008 has been the backup set. Windows Server Backup added support for backing up to SMB, however only if you perform a full, rather than incremental or differential backup of the host server.

The main problem with this is the time it takes to perform the backup. Depending on the size of the disk array involved, a normal backup job can take tens of hours, even days. If you want to run the backup job daily and the job is taking more than a day to complete while saturating the network, then it is not a very effective backup solution.

Yet the real power in using the network in the first place is the fact that it permits the distribution of the backup to a remote location without the need to go and physically disconnect a drive and carry it. The drives can also be a lot further away than with USB, eSATA or firewire. In another building or in another country.

Further more, the array is more expandable than a typical USB disk. The maximum supported size of this little ReadyNAS duo v2 is 2x4TB in RAID 0, resulting in 8GB of storage. I could also run it in RAID 1 if I needed higher levels of data security. This is much better than a typical USB disk. With a 4 or 8 bay NAS, you can even grow the array by adding new drives and expand the VHDX file according to you needs (up to the limit of the native NAS file system or the NTFS volume limit). Devices with more bays also allow for additional RAID types and associated data security such as RAID 5, 6 or 10.

In many situations, you can use iSCSI for this purpose. Most high end and Enterprise NAS storage and SAN solutions are designed to provide thick or thin provisioned iSCSI targets which you can easily mount via the iSCSI initiator in Windows. An iSCSI mounted drive in Windows is – at least as far as Window is concerned – presented as a local disk and therefore you can perform a differencing backup under the control of Windows Server Backup (WSB).

So what can you do if you have a consumer grade NAS appliance or an old model device that does not expose iSCSI services? A device such as the Netgear ReadyNAS Duo v2? While the v2 version has an unofficial iSCSI Target plugin, this does not work on the v2 model and so having a very low power, ARM based NAS lying around with 6TB of disks in it, it seems a shame to relegate it to the dustbin.

More Info

Storage virtualisation is the answer.

Simply put, Windows Server Backup (WSB) cannot itself perform differential backups to a SMB share, however a SMB share (even a SMB 2.0 share) can host a virtualised storage disk… and Windows can mount a virtualised disk across SMB. Once mounted, WSB is agnostic to the underlying disk location or the fact that it is stored on a SMB share as Windows presents the disk as being locally attached and abstracts the ‘what’ and ‘were’ entirely to the virtualisation layer.

The Test

If you are going to attempt this, I strongly recommend that you enable Jumbo Frames on the device as you may be able to squeeze a 10-20 Mbps of additional write speed out of the device.

View: Netgear ReadyNAS Duo V2 and Jumbo Frames

  1. Ensure that your machine can connect to the ReadyNAS over SMB i.e. \\<ipAddress\<shareName>
  2. Create a share on the NAS for the backup. Create a dedicated one so that you minimise SMB file system update requests to the share. As will be mentioned below, this causes between a 10-20Mbps loss of performance even if nothing is actually happening in Windows Explorer.
  3. Disable as many services as you can on the NAS. The less work the CPU is doing and the more free RAM, the better this will be.
  4. Open PowerShell on Windows 8, 8.1, 10, 2012, 2012 R2 or 2016 and enter:
    New-VHD –Path “\\<ipAddress\<shareName>\Backup.vhdx” –SizeBytes 4096GB

    Substitute the 4096GB (4TB) with the size that you require. This will create a dynamically expanding Hyper-V Virtual Hard Drive on the ReadyNAS

  5. In PowerShell issue the following command to mount the VHDX file
    Mount-VHD -Path "\\<ipAddress\<shareName>\Backup.vhdx"

    Now use DiskPart or Disk Manager to initialise, partition and format the disk. Remember to format it using 64K sectors as this will be important to preserve performance for the large files involved.

    Alternately you can execute the entire initialisation and mounting process in PowerShell using:

    Mount-VHD -Path "\\<ipAddress\<shareName>\Backup.vhdx" -Passthru |Initialize-Disk -Passthru |New-Partition -DriveLetter B -UseMaximumSize
    Format-Volume -DriveLetter B -FileSystem NTFS -NewFileSystemLabel "Backup Disk" -AllocationUnitSize 65536 -Confirm:$false -Force

    This will create a 64K NTFS partition called “Backup Disk” and mount it on B:\ using the VHDX file found on the ReadyNAS

  6. Now, if you attempt to use Windows Server Backup you will be able to create a differencing disk backup set.

Does it work?

Windows Server Backup (WSB) certainly accepts the disk without any complaints and is dutifully able to create the first Normal copy after which it is able to easily perform the delta-backup as would be familiar for an incremental or differential backup type. So yes, it does work. It tricks Windows into accepting the SMB target.

Performance is however a sticking point.

To apply some context: According to a “Legit Reviews” review of the WD Red 5400rpm 3TB drives in the NAS, each drive should easily have been able to manage a write speed of 80MB/s or 640Mbps at a minimum – with something around 147MB/s or 1176Mbps being expected for sequential writes.

Source: WD Red 3TB NAS Hard Drive Review (Page 3)

 

Creating a Hyper-V VHDX file and writing linear zero’s to it across the network results in a write speed variance of between 445Mbps and 495Mbps on the wire (55.6MB/s – 61.25MB/s). The highest that I saw it peak at was 537Mbps in burst.

Performing a backup onto the drive took 23 hours and 2 minutes with the MTU set to 1500 bytes with the average bit rate being approximately 420Mbps – 430Mbps for the backup. Particularly painful for the first normal backup. This is however comparable to the performance of a USB 2.0 drive.

So we can safely conclude that the bottleneck is not the drives. The bottleneck is the ReadyNAS Duo v2. Other, newer devices with more CPU horsepower, more RAM, larger NIC buffers, native Jumbo Frame offload support and more NICs (as well as more drives) should be able to offer better performance.

As an interesting side observation, having a Windows Explorer session sitting open to a SMB share and doing nothing slowed the zeroing process by 10-20Mbps on its own. This highlights the impact of having the NAS CPU performing other actions and its impact of write performance.

Reality Check

There are however some problems here. Windows Server Backup is not aware that this is a virtual drive, it expects the drive to perform and present like a physical hard drive and it will treat it as such consequently

  1. It is going to have poor support for and tolerance of power management (suspend and standby).
  2. It is going to have little to no tolerance for an unreliable network connection i.e. never try to do this over wireless or an unstable Internet connection.
  3. It is going to be extremely susceptible to power outages. You really should use UPS on the NAS, switch(s) and the source machine to prevent data corruption during a power outage. Note that the important part here is that the source machine stops the backup and dismounts the VHDX in the time it spends on the UPS. After that it can all turn off quite happily.
  4. Windows is not going to automatically mount the VHDX. WSB will not do this for you. You will have to either ensure that it mounts as boot or schedule it to mount before the backup.
  5. Windows is going to need to ensure that it cleanly dismounts the VHDX during shutdown and power management operations. WSB is not going to do this for you either.
  6. Use write caching on the NAS and on the host operating system at your risk i.e. definitely have a UPS if you want this performance benefit.
  7. If you need to perform a bare metal recovery of the server, the extra steps of getting the VHDX mounted in the boot recovery environment may prove frustrating.
  8. While VHDX in Server 2012 R2 can technically be ued as a shared medium, you should probably avoid even contemplating trying to share one VHDX between multiple WSB hosts. Create one VHDX for each server.
  9. The current maximum size of a VHDX is 64TB. If this is an issue 1) why are you using a consumer grade NAS? 2) you need a SAN 3) you shouldn’t be using WSB

It should be noted that all of the above postential disadvantages also apply to some degree to the use of iSCSI. The advantage of this approach is that you get the data virtualisation advantage where as with iSCSI your NAS would have to expose this i.e. you can literally just pickup the VHDX and move it to a new HDD, Array, NAS or SAN and WSB isn’t going to care or even notice.

So what can you do? My suggestion is this: do not use the WSB UI to schedule the backup. Use task scheduler and the WSB command line tool WBAdmin.exe to perform the backup in a PowerShell script. Something like the following:

Mount-VHD -Path "\\192.168.0.100\Backup\Backup.vhdx"

Start-Sleep -s 60       # Wait 60 seconds for the disk to come online

C:\Windows\System32\wbadmin.exe start backup -backupTarget:B: -allCritical -include:C: -systemState -vssFull -quiet

Start-Sleep -s 120      # Wait 120 seconds for the disk to go offline

DisMount-VHD -Path "\\192.168.0.100\Backup\Backup.vhdx"

When task scheduler fires the script it will mount the VHDX, wait 60 seconds to allow the file system to mount, perform the backup, wait 120 seconds for the backup sub-system to shutdown and then cleanly dismount the VHDX.

Optimisation and issues

The 256MB RAM, ARM based ReadyNAS Duo v2 was never intended for these kinds of workloads and that does show. Most of the issues encountered with it are simply as a result of the low power, low resource hardware specification.

I have already covered the need to:

  • Use Jumbo Frames on the NIC
  • Do not use wireless connections to mount the VHDX
  • Use 64K sectors on the NTFS volume
  • Optionally use the write cache setting on the ReadyNAS
  • Optionally enable write caching and prevent buffer flushing on the volume as exposed via host operating system

To this I will add the following:

Do not use VHD files, only use VHDX. VHDX are far,far safer to use over SMB compared to VHD as they have error correction and handling built-in. consequently, there is a reasonable chance that the file will actually survive a disconnect of the network cable or power of the source or destination as a result. This does however restrict you to using Windows 8/Server 2012 or higher at the expense of Windows Server 2008/2008 R2

Only use 1Gbps or 10Gbps networks. Do not use 802.11 wireless and do not use 10/100 Fast Ethernet.

Use server grade NICs in your devices if you can

Use MPIO and multiple switches if you can spare/afford the hardware

Keep your VHDX defragged just like any other NTFS formatted hard drive

If you have managed switches, consider preventing broadcast and multicast traffic from reaching the NAS. This will reduce CPU load a little although it will prevent NetBIOS discovery and may impact other services.

Do not use the NAS for anything else, especially small SMB file storage. Client access will degrade the write performance and consume CPU time. In particular do not leave the NAS SMB mount point mounted as a network drive as this also holds a SMB session open with the Linux Samba service.

Do not use dynamically expanding VHDX files. Using a dynamically expanding VHDX file was in reality fine (if you accept the limitations of the device). It took nearly 4TB of data without incident, however the use of dynamically expanding disks is itself inefficient. Dynamic disks have a performance penalty associated with them as the disk head is constantly being told to zero the trailing 12MB of the VHDX file to permit future growth of the VHDX. There are also associated writes to the metadata of the VHDX to update the file boundary markers. In trying to squeeze every last bit of performance out of the ReadyNAS Duo v2, I wanted to use a fixed size VHDX file to see if it was any more performance efficient.

One of the first issues encountered was on the length of time it takes to allocate and deallocate space from the Linux disk journal. Allocation is proportionately faster then deallocation, however on attempting to allocate 5.4TB of disk space to a singe VHDX file, it would take the system an extended period to process and the VHDX creation process on Windows would timeout, causing the VHDX to be corrupted. At this point the VHDX would be deleted by Windows. This storage deallocation could take upwards of 20 minutes to appear as released in the ReadyNAS web UI.

Looking at ‘top’ in the SSH session, it was clear that the CPU was the culprit, capping out at 100% throughout the entire operation before dropping down to <1% once the journal had been updated.

After some trial and error, I found that with the web UI closed, only necessary services running, SSH logged out and no active Windows Explorer sessions open, I could allocate 2TB at a time without it causing a timeout.

The following script can thus be used to create the VHDX at 2TB, expand it to 4TB and then expand it to the desired 5.3TB (the maximum size of the ReadyNAS volume I was using was 5.4TB).

New-VHD -Path "\\192.168.0.100\Backup\Backup.vhdx" –Fixed –SizeBytes 2TB
Resize-VHD –Path "\\192.168.0.100\Backup\Backup.vhdx" –SizeBytes 4TB
Resize-VHD –Path "\\192.168.0.100\Backup\Backup.vhdx" –SizeBytes 5.3TB

Remember, this script is creating a Fixed Size VHDX file. Consequently it is going to pre-zero each sector on the disk instead of performing a constant 12MB zeroing chase at the end of the file. This means that it will take an extremely long time to complete (especially at only ~470Mbps) i.e. over 24 hours! So I suggest that you copy and paste all three lines at once into the PowerShell buffer and walk away. Once it has finished chewing over all three lines, mount the disk, partition it and format it as outlined earlier in the article.

Note: There are a coupe of utilities out on the Internet that can create a fixed size VHDX from free space without performing the zeroing operation. You can save yourself a lot of time using such tools however you should NEVER use them in a production or in a shared environment due to reasons of data safety, privacy and security.

After the allocation of the space in the Journal and during the zeroing process the CPU use remains high, running constantly at 100% with about 20MB of RAM showing as free out of the 256MB total. This proves that the sub 500Mbps cap on the transfer speed is being caused by the CPU and not the disks. You must thus be realistic about the capability of the appliance or pay for more robust, more capable hardware.

You can technically also disable journalling on the volume using SSH, however you must ensure that you have a UPS wired into the NAS and the UPS can perform a controlled shutdown of the NAS if you try and use it. I elected not to do this.

tune4fs -O ^has_journal /dev/sda0
e4fsck –f /dev/sda0
sudo reboot

Final Results

If you have read this and the Jumbo Frames article on the ReadyNAS Duo v2, I am sure that you might be interested to hear what the cumulative impact of all of the performance tweaks and optimisation’s was.

With write caching enabled on both the NAS and the Windows Server and buffer flushing disabled on Windows Server, plus all of the other tweaks listed, backup throughput rose to a fairly consistent 560Mbps – 590Mbps with bursts up to 638Mbps. That equates to 70MB/s – 73.75MB/s and 79.75MB/s at burst. While nowhere near the capability of the drives themselves, it is at least now tantalisingly close to the benchmark value for the drives random write performance test and network write performance is nearly 200Mbps faster.

Performing the backup job (which without any optimisation’s on a Dynamic VHDX took 23 hours and 2 minutes) with all optimisation’s enabled – and actually a significantly larger workload due to the addition of VM state backups in the job – took some 16 and 47 minutes. A considerable improvement! That works out at around 200GB per hour.

Most importantly, when the job ran again the next evening, it took less than 30 minutes thanks to it only having to backup file differences.

So why is this? It is predominantly related to the fixed size VHDX file. The higher throughput is being achieved because the ReadyNAS CPU is sitting at around 5% – 30% idle during the ~600Mbps copy. The Linux file system sees the write process as constituting changes that are internal to the VHDX file and the file itself isn’t growing, therefore the file system driver on the NAS has significantly less work to do. It is instead NTFS on the backup server that is processing the MFT updates into the file allocation table of the VHDX completely transparently to the NAS. This means that the CPU work has been transferred to the backup server, resulting in a performance increase (and a slightly cooler, less power consuming NAS).

Error 0x80070002 when attempting to backup a Hyper-V Virtual Machine using Windows Server Backup

System Requirements:

  • Windows Server 2008 R2
  • Windows Server 2012
  • Windows Server 2012 R2

The Problem:

You backup, right? Of course you do! Only the cool people backup – and you are one of the cool people aren’t you?

…If only life was that simple.

So imagine for a moment that you are attempting to use VVS and WIndows Server Backup to backup a server. In particular a fully loaded Hypervisor running Windows Server 2012 R2 Datacentre in this case.

The backup process goes OK for the most part, but fails to complete on a number (but by no means all) VMs. The process fails with the following errors on Windows Server 2008 VMs, but not necessarily newer ones.

From Windows Server Backup:

Windows Server Backup “Failed” -or- “Completed with warnings” -or-“Backup failed to complete”

The component <VM Name>(Online) was skipped during the snapshot and will not be available for recovery. Error: The writer experienced a non-transient error. If the backup process is retried, the error is likely to reoccur

In the Hyper-V-VMMS\Admin log in Event Log:

‘<VM Name>’ cannot create the storage required for the checkpoint using disk E:\Virtual Machines\<VM Path>\Virtual Hard Disks\<VHD Filename>.vhdx: The system cannot find the file specified. (0x80070002). (Virtual machine ID <VM GUID>)

and…

Checkpoint operation for ‘<VM Name>’ failed. (Virtual machine ID <VM GUID>)

and…

Could not create backup checkpoint for virtual machine ‘<VM Name>’: The system cannot find the file specified. (0x80070002). (Virtual machine ID <VM GUID>)

and of course most helpfully…

The operation failed.

More Info

If you actually look at the backup file, you will see what looks to be a complete file set for the backup, however given that this error represents an error in VSS, you would not be advised to trust it.

As usual with Hyper-V error logs, the error message have little if anything to do with the actual issue and someone in the Microsoft Development team just needs to be shot for it… but I digress.

The odd think was that the issue was occurring on all of the Windows Server 2008 (R1) VMs, the Windows Server 2008 R2 and higher VMs were backing up correctly.

The Fix

So before I get into the issue I encountered, lets run past the generic fixes

  1. Check that you have enough disk space on the volume to perform the VSS. If your volume is sub-15%, try using the Hyper-V Manager to change the snapshot directory to another volume – plug in an external NTFS formatted hard drive if you have to.
  2. Check the permissions of the VHD stated in the error.
    icacls “C:\Users\Public\Documents\Hyper-V\Virtual Hard Disks\<VHD File>” /grant “NT VIRTUAL MACHINE\Virtual Machines”:F /TSource: Technet
    Source: System Center Central
  3. Ensure that Hyper-V is patched fully.
    Windows Server 20102 R2 users see: https://support.microsoft.com/en-us/kb/2920151
  4. Run chkdsk on the physcial volume on the Hypervisor and on the virtual volume in the VM
  5. Ensure that the Integration Service Components are at the latest version and that they VSS Writer module for it is enabled in the VM properties in Hyper-V Manager

Now the less well documented approaches

  1. Check that you can manually checkpoint/snapshot the VM while it is running.
    In Hyper-V Manager or in PowerShell, force a checkpoint on the VM and then delete it and wait for it to merge back. If this works, you are not having a physical VSS issue. If it fails, you need to troubleshoot this and not the WSB error.
  2. Live Migrate the VM off of the current server and onto a different Hypervisor, attempt the backup here, then bring it back to the original server and try again. This process will reset the permissions on the VM file set. If you cannot live or offline migrate the VM, the you need to troubleshoot this and not the WSB error.

My fix

In my case, the issue was to do with having the VM VHDX files split across a couple of different storage LUN/volumes. I usually move VM page files onto a dedicated partition on a dedicated spindle (usually an SSD) and leave OS and data volumes on larger arrays. This helps to keep the VMs running smoothly and keeps unnecessary paging operations off of parity checked storage volumes.

So imagine that the VM has the following file storage structure

Physical Hypervisor SSD (this is where the Page File’s live)
– D:\my-virtual-server-d-drive.vhdx

Physical Hypervisor Storage Array
– E:\Virtual Machines\<VM Name>\Planned Virtual Machines
– E:\Virtual Machines\<VM Name>\Snapshots
– E:\Virtual Machines\<VM Name>\Virtual Hard Drives\my-virtual-server-c-drive.vhdx
– E:\Virtual Machines\<VM Name>\Virtual Hard Drives\my-virtual-server-e-drive.vhdx
– E:\Virtual Machines\<VM Name>\Virtual Machines

It is actually this structure which breaks the WSB backup job. Contrary to the VSS event log error, the problem drive is NOT my-virtual-server-c-drive.vhdx it is actually my-virtual-server-d-drive.vhdx. The event log will actually log that the error was caused on the first drive attached to the system bus (I think).

If you weren’t too clever when you followed my advice above and live migrated all of the storage to the same location on a different Hypervisor, you probably found this out for yourself – the backup should have worked.

When you split the job back into separate LUNs, it fails again. The fix is oddly simple and continues to allow you to have split LUN storage if you wish. Change the file system structure to:

Physical Hypervisor SSD (this is where the Page File’s live)
– D:\<VM Name>\my-virtual-server-d-drive.vhdx

Physical Hypervisor Storage Array
– E:\Virtual Machines\<VM Name>\Planned Virtual Machines
– E:\Virtual Machines\<VM Name>\Snapshots
– E:\Virtual Machines\<VM Name>\Virtual Hard Drives\my-virtual-server-c-drive.vhdx
– E:\Virtual Machines\<VM Name>\Virtual Hard Drives\my-virtual-server-e-drive.vhdx
– E:\Virtual Machines\<VM Name>\Virtual Machines

Note the introduction of a folder on the D drive with the same name as the <VM Name> folder on the E drive. Do NOT shutdown the VM and move the storage there yourself, use the Hyper-V Manager or PowerShell processes to perform a “move” on the ‘storage only’ and just move the one drive. This will ensure that permissions are correct.

The next time that you run the backup, it will VSS correctly.

As for why it can do this on its own with Windows Server 2008 R2 or higher VMs, but not Windows Server 2008 or lower VMs… I have no idea although I suspect it to have something to do with the capabilities of the integration services components.

Edit: A post publish search on the issue reveals that I’m not alone in working this out
View: Technet