Category: Software

System Requirements:

• Netgear ReadyNAS Duo v2 or any SMB capable NAS
• Windows Server 2008, 2008 R2, 2012, 2012 R2

The Problem:

One of the most frustrating “features” of Windows Server since the release of Windows Server 2008 has been the backup set. Windows Server Backup added support for backing up to SMB, however only if you perform a full, rather than incremental or differential backup of the host server.

The main problem with this is the time it takes to perform the backup. Depending on the size of the disk array involved, a normal backup job can take tens of hours, even days. If you want to run the backup job daily and the job is taking more than a day to complete while saturating the network, then it is not a very effective backup solution.

Yet the real power in using the network in the first place is the fact that it permits the distribution of the backup to a remote location without the need to go and physically disconnect a drive and carry it. The drives can also be a lot further away than with USB, eSATA or firewire. In another building or in another country.

Further more, the array is more expandable than a typical USB disk. The maximum supported size of this little ReadyNAS duo v2 is 2x4TB in RAID 0, resulting in 8GB of storage. I could also run it in RAID 1 if I needed higher levels of data security. This is much better than a typical USB disk. With a 4 or 8 bay NAS, you can even grow the array by adding new drives and expand the VHDX file according to you needs (up to the limit of the native NAS file system or the NTFS volume limit). Devices with more bays also allow for additional RAID types and associated data security such as RAID 5, 6 or 10.

In many situations, you can use iSCSI for this purpose. Most high end and Enterprise NAS storage and SAN solutions are designed to provide thick or thin provisioned iSCSI targets which you can easily mount via the iSCSI initiator in Windows. An iSCSI mounted drive in Windows is – at least as far as Window is concerned – presented as a local disk and therefore you can perform a differencing backup under the control of Windows Server Backup (WSB).

So what can you do if you have a consumer grade NAS appliance or an old model device that does not expose iSCSI services? A device such as the Netgear ReadyNAS Duo v2? While the v2 version has an unofficial iSCSI Target plugin, this does not work on the v2 model and so having a very low power, ARM based NAS lying around with 6TB of disks in it, it seems a shame to relegate it to the dustbin.

More Info

Storage virtualisation is the answer.

Simply put, Windows Server Backup (WSB) cannot itself perform differential backups to a SMB share, however a SMB share (even a SMB 2.0 share) can host a virtualised storage disk… and Windows can mount a virtualised disk across SMB. Once mounted, WSB is agnostic to the underlying disk location or the fact that it is stored on a SMB share as Windows presents the disk as being locally attached and abstracts the ‘what’ and ‘were’ entirely to the virtualisation layer.

The Test

If you are going to attempt this, I strongly recommend that you enable Jumbo Frames on the device as you may be able to squeeze a 10-20 Mbps of additional write speed out of the device.

View: Netgear ReadyNAS Duo V2 and Jumbo Frames

1. Ensure that your machine can connect to the ReadyNAS over SMB i.e. \\<ipAddress\<shareName>
2. Create a share on the NAS for the backup. Create a dedicated one so that you minimise SMB file system update requests to the share. As will be mentioned below, this causes between a 10-20Mbps loss of performance even if nothing is actually happening in Windows Explorer.
3. Disable as many services as you can on the NAS. The less work the CPU is doing and the more free RAM, the better this will be.
4. Open PowerShell on Windows 8, 8.1, 10, 2012, 2012 R2 or 2016 and enter:
   New-VHD –Path “\\<ipAddress\<shareName>\Backup.vhdx” –SizeBytes 4096GB

   Substitute the 4096GB (4TB) with the size that you require. This will create a dynamically expanding Hyper-V Virtual Hard Drive on the ReadyNAS

5. In PowerShell issue the following command to mount the VHDX file
   Mount-VHD -Path "\\<ipAddress\<shareName>\Backup.vhdx"

   Now use DiskPart or Disk Manager to initialise, partition and format the disk. Remember to format it using 64K sectors as this will be important to preserve performance for the large files involved.

   Alternately you can execute the entire initialisation and mounting process in PowerShell using:

   Mount-VHD -Path "\\<ipAddress\<shareName>\Backup.vhdx" -Passthru |Initialize-Disk -Passthru |New-Partition -DriveLetter B -UseMaximumSize
Format-Volume -DriveLetter B -FileSystem NTFS -NewFileSystemLabel "Backup Disk" -AllocationUnitSize 65536 -Confirm:$false -Force

   This will create a 64K NTFS partition called “Backup Disk” and mount it on B:\ using the VHDX file found on the ReadyNAS

6. Now, if you attempt to use Windows Server Backup you will be able to create a differencing disk backup set.

Does it work?

Windows Server Backup (WSB) certainly accepts the disk without any complaints and is dutifully able to create the first Normal copy after which it is able to easily perform the delta-backup as would be familiar for an incremental or differential backup type. So yes, it does work. It tricks Windows into accepting the SMB target.

Performance is however a sticking point.

To apply some context: According to a “Legit Reviews” review of the WD Red 5400rpm 3TB drives in the NAS, each drive should easily have been able to manage a write speed of 80MB/s or 640Mbps at a minimum – with something around 147MB/s or 1176Mbps being expected for sequential writes.

Source: WD Red 3TB NAS Hard Drive Review (Page 3)

 

Creating a Hyper-V VHDX file and writing linear zero’s to it across the network results in a write speed variance of between 445Mbps and 495Mbps on the wire (55.6MB/s – 61.25MB/s). The highest that I saw it peak at was 537Mbps in burst.

Performing a backup onto the drive took 23 hours and 2 minutes with the MTU set to 1500 bytes with the average bit rate being approximately 420Mbps – 430Mbps for the backup. Particularly painful for the first normal backup. This is however comparable to the performance of a USB 2.0 drive.

So we can safely conclude that the bottleneck is not the drives. The bottleneck is the ReadyNAS Duo v2. Other, newer devices with more CPU horsepower, more RAM, larger NIC buffers, native Jumbo Frame offload support and more NICs (as well as more drives) should be able to offer better performance.

As an interesting side observation, having a Windows Explorer session sitting open to a SMB share and doing nothing slowed the zeroing process by 10-20Mbps on its own. This highlights the impact of having the NAS CPU performing other actions and its impact of write performance.

Reality Check

There are however some problems here. Windows Server Backup is not aware that this is a virtual drive, it expects the drive to perform and present like a physical hard drive and it will treat it as such consequently

1. It is going to have poor support for and tolerance of power management (suspend and standby).
2. It is going to have little to no tolerance for an unreliable network connection i.e. never try to do this over wireless or an unstable Internet connection.
3. It is going to be extremely susceptible to power outages. You really should use UPS on the NAS, switch(s) and the source machine to prevent data corruption during a power outage. Note that the important part here is that the source machine stops the backup and dismounts the VHDX in the time it spends on the UPS. After that it can all turn off quite happily.
4. Windows is not going to automatically mount the VHDX. WSB will not do this for you. You will have to either ensure that it mounts as boot or schedule it to mount before the backup.
5. Windows is going to need to ensure that it cleanly dismounts the VHDX during shutdown and power management operations. WSB is not going to do this for you either.
6. Use write caching on the NAS and on the host operating system at your risk i.e. definitely have a UPS if you want this performance benefit.
7. If you need to perform a bare metal recovery of the server, the extra steps of getting the VHDX mounted in the boot recovery environment may prove frustrating.
8. While VHDX in Server 2012 R2 can technically be ued as a shared medium, you should probably avoid even contemplating trying to share one VHDX between multiple WSB hosts. Create one VHDX for each server.
9. The current maximum size of a VHDX is 64TB. If this is an issue 1) why are you using a consumer grade NAS? 2) you need a SAN 3) you shouldn’t be using WSB

It should be noted that all of the above postential disadvantages also apply to some degree to the use of iSCSI. The advantage of this approach is that you get the data virtualisation advantage where as with iSCSI your NAS would have to expose this i.e. you can literally just pickup the VHDX and move it to a new HDD, Array, NAS or SAN and WSB isn’t going to care or even notice.

So what can you do? My suggestion is this: do not use the WSB UI to schedule the backup. Use task scheduler and the WSB command line tool WBAdmin.exe to perform the backup in a PowerShell script. Something like the following:

When task scheduler fires the script it will mount the VHDX, wait 60 seconds to allow the file system to mount, perform the backup, wait 120 seconds for the backup sub-system to shutdown and then cleanly dismount the VHDX.

Optimisation and issues

The 256MB RAM, ARM based ReadyNAS Duo v2 was never intended for these kinds of workloads and that does show. Most of the issues encountered with it are simply as a result of the low power, low resource hardware specification.

I have already covered the need to:

• Use Jumbo Frames on the NIC
• Do not use wireless connections to mount the VHDX
• Use 64K sectors on the NTFS volume
• Optionally use the write cache setting on the ReadyNAS
• Optionally enable write caching and prevent buffer flushing on the volume as exposed via host operating system

To this I will add the following:

Do not use VHD files, only use VHDX. VHDX are far,far safer to use over SMB compared to VHD as they have error correction and handling built-in. consequently, there is a reasonable chance that the file will actually survive a disconnect of the network cable or power of the source or destination as a result. This does however restrict you to using Windows 8/Server 2012 or higher at the expense of Windows Server 2008/2008 R2

Only use 1Gbps or 10Gbps networks. Do not use 802.11 wireless and do not use 10/100 Fast Ethernet.

Use server grade NICs in your devices if you can

Use MPIO and multiple switches if you can spare/afford the hardware

Keep your VHDX defragged just like any other NTFS formatted hard drive

If you have managed switches, consider preventing broadcast and multicast traffic from reaching the NAS. This will reduce CPU load a little although it will prevent NetBIOS discovery and may impact other services.

Do not use the NAS for anything else, especially small SMB file storage. Client access will degrade the write performance and consume CPU time. In particular do not leave the NAS SMB mount point mounted as a network drive as this also holds a SMB session open with the Linux Samba service.

Do not use dynamically expanding VHDX files. Using a dynamically expanding VHDX file was in reality fine (if you accept the limitations of the device). It took nearly 4TB of data without incident, however the use of dynamically expanding disks is itself inefficient. Dynamic disks have a performance penalty associated with them as the disk head is constantly being told to zero the trailing 12MB of the VHDX file to permit future growth of the VHDX. There are also associated writes to the metadata of the VHDX to update the file boundary markers. In trying to squeeze every last bit of performance out of the ReadyNAS Duo v2, I wanted to use a fixed size VHDX file to see if it was any more performance efficient.

One of the first issues encountered was on the length of time it takes to allocate and deallocate space from the Linux disk journal. Allocation is proportionately faster then deallocation, however on attempting to allocate 5.4TB of disk space to a singe VHDX file, it would take the system an extended period to process and the VHDX creation process on Windows would timeout, causing the VHDX to be corrupted. At this point the VHDX would be deleted by Windows. This storage deallocation could take upwards of 20 minutes to appear as released in the ReadyNAS web UI.

Looking at ‘top’ in the SSH session, it was clear that the CPU was the culprit, capping out at 100% throughout the entire operation before dropping down to <1% once the journal had been updated.

After some trial and error, I found that with the web UI closed, only necessary services running, SSH logged out and no active Windows Explorer sessions open, I could allocate 2TB at a time without it causing a timeout.

The following script can thus be used to create the VHDX at 2TB, expand it to 4TB and then expand it to the desired 5.3TB (the maximum size of the ReadyNAS volume I was using was 5.4TB).

Remember, this script is creating a Fixed Size VHDX file. Consequently it is going to pre-zero each sector on the disk instead of performing a constant 12MB zeroing chase at the end of the file. This means that it will take an extremely long time to complete (especially at only ~470Mbps) i.e. over 24 hours! So I suggest that you copy and paste all three lines at once into the PowerShell buffer and walk away. Once it has finished chewing over all three lines, mount the disk, partition it and format it as outlined earlier in the article.

Note: There are a coupe of utilities out on the Internet that can create a fixed size VHDX from free space without performing the zeroing operation. You can save yourself a lot of time using such tools however you should NEVER use them in a production or in a shared environment due to reasons of data safety, privacy and security.

After the allocation of the space in the Journal and during the zeroing process the CPU use remains high, running constantly at 100% with about 20MB of RAM showing as free out of the 256MB total. This proves that the sub 500Mbps cap on the transfer speed is being caused by the CPU and not the disks. You must thus be realistic about the capability of the appliance or pay for more robust, more capable hardware.

You can technically also disable journalling on the volume using SSH, however you must ensure that you have a UPS wired into the NAS and the UPS can perform a controlled shutdown of the NAS if you try and use it. I elected not to do this.

Final Results

If you have read this and the Jumbo Frames article on the ReadyNAS Duo v2, I am sure that you might be interested to hear what the cumulative impact of all of the performance tweaks and optimisation’s was.

With write caching enabled on both the NAS and the Windows Server and buffer flushing disabled on Windows Server, plus all of the other tweaks listed, backup throughput rose to a fairly consistent 560Mbps – 590Mbps with bursts up to 638Mbps. That equates to 70MB/s – 73.75MB/s and 79.75MB/s at burst. While nowhere near the capability of the drives themselves, it is at least now tantalisingly close to the benchmark value for the drives random write performance test and network write performance is nearly 200Mbps faster.

Performing the backup job (which without any optimisation’s on a Dynamic VHDX took 23 hours and 2 minutes) with all optimisation’s enabled – and actually a significantly larger workload due to the addition of VM state backups in the job – took some 16 and 47 minutes. A considerable improvement! That works out at around 200GB per hour.

Most importantly, when the job ran again the next evening, it took less than 30 minutes thanks to it only having to backup file differences.

So why is this? It is predominantly related to the fixed size VHDX file. The higher throughput is being achieved because the ReadyNAS CPU is sitting at around 5% – 30% idle during the ~600Mbps copy. The Linux file system sees the write process as constituting changes that are internal to the VHDX file and the file itself isn’t growing, therefore the file system driver on the NAS has significantly less work to do. It is instead NTFS on the backup server that is processing the MFT updates into the file allocation table of the VHDX completely transparently to the NAS. This means that the CPU work has been transferred to the backup server, resulting in a performance increase (and a slightly cooler, less power consuming NAS).