Performance impact of 512byte vs 4K sector sizes

When you are designing your storage subsystem. On modern hardware, you will often be asked to choose between formatting using 512 byte or 4K (4096 byte) sectors. This article discusses whether there is any statistically observable performance difference between the two in a 512 vs. 4K performance test.

NB: Do not get confused between the EXT4 INODE size and the LUN sector size. The INODE size places a mathematical cap on the number of files that a file system can store, and by consequence how large the volume can be. The sector size relates to how the file system interacts with the physical underlying hardware.

QNAP Sector Size selection
Sector Size selection on QNAP QTS 4.3.6

Method

  • A QNAP TS-1277XU-RP with 8x WD Red Pro 7200 RPM WD6003FFBX-68MU3N0 drives running firmware 83.00A83 were installed with 8 drives in bays 5 – 12
  • Storage shelf firmware was updated to QTS version 4.3.6.0923, providing the latest platform enhancements
  • A Storage Pool comprising all 8 disks in RAID 6 was configured, ensuring redundancy
  • A 4GB volume was added allowing QNAP app installation so that the systme could finish installing
  • The disk shelf was rebooted after it had completed its own setup tasks
  • RAID sync was allowed to fully complete over the next 12 hours
  • Two identical 4096 GB iSCSI targets were created with identical configurations apart from one having 512 byte and the other 4k sector sizes
  • SSD caching was disabled on the storage shelf
  • 2x10Gbps Ethernet, dedicated iSCSI connections were made available through two Dell PowerConnect SAN switches. Each NIC on its own VLAN. 9k jumbo frames were enabled accross the fabric
  • A Windows Server 2016 hypervisor was connected to the iSCSI target and mounted the storage volume. iSCSI MPIO was enabled in Round Robin mode. Representing a typical hypervisor configuration
  • The two storage LUNs were formatted with 64K NTFS partitions (recommended for dedicated VHDX volumes)
  • A Windows 10 VM was migrated onto each of the targets and the test performed using Anvil’s Storage Utilities 1.1.0.20140101. The VM had no live network connections. The Super Fetch and Windows Update services were disabled, preventing undesirable disk I/O. The VM was not rebooted between tests, had no other running tasks and had been idling for 6 hours prior to the test
  • No other tasks, load or data were present on the storage array

 

512 vs. 4K Performace Results

The results of the two tests are shown below.

Anvil Storage Utilities Screenshot with 512bytes results
Anvil Storage Utilities Screenshot with 512bytes results

"Anvil

IOPS
512 byte 4K 4K Diff 4K Diff % +/-
Read Seq 4MB 417.45 403.3 -14.15 -3.51
4K 3001.56 3164.56 163 5.15
4K QD4 6021.45 6006.7 -14.75 -0.25
4K QD16 24228.16 24062.61 -165.55 -0.69
32K 2742.39 2807.47 65.08 2.32
128K 2628.86 2620.8 -8.06 -0.31
Write Seq 4MB 233.2 230.79 -2.41 -1.04
4K 2090.79 2165.45 74.66 3.45
4K QD4 5976.18 5983.65 7.47 0.12
4K QD16 8254.84 7874.67 -380.17 -4.83

 

Analysis and Recommendations

The results show that there is little difference between the two. Repeating the tests multiple times showed that the figures for both the 512 byte and 4K LUNs are within the margin of error of each other. A bias towards 512 byte was consistently present, but was not statistically significant.

The drives in the test disk array are 512e drives. 512e is an industry transition technology between pure 512 byte and pure 4K drives. 512e drives use physical 4K sectors on the platter, but that the firmware uses 512 byte logic. A firmware emulation layer converts between the two. This creates a performance penalty during write operations due to the computation and delay of the re-mapping operation. Neither sector size will prevent this from occurring.

My recommendations are

  • If all of your drives are legacy 512 byte drives, only use 512
  • Should you intend to mount the LUN with an operating system that does not support 4K sectors. Only use 512
  • In situations where you have 512e drives, you can use either. Unless you intend to clone the LUN onto 4K drives in the future, stick with 512 for maximum compatibility
  • Never create an array that mixes 512 and 4K disks. Ensure that you create storage pools and volumes accordingly
  • Where all of your drives are 4K, only use 4K

 

QNAP TVS-1271U Top Row of Drives (drives 9-12) do not start on boot/reboot of the NAS appliance

System Requirements:

  • QNAP TVS-1271U
  • QTS 4.3

The Problem:

A brand new, out of the box 12 drive 2u NAS appliance, with the “firmware” up to date gets thrown into production. During its first maintenance cycle, the firmware is updated again and it is rebooted.

On completion of the reboot, the array is offline. Visual inspection of the appliance reveals that the top row of drives in the array, drives 9, 10, 11 and 12 are offline. They just are not powered. Drives 1-8 are online and all green.

Rebooting the array shows the diagnostic LED’s on all 12 drives flash red, but the 4 drives in question then go into a powered down state and the boot completes with an inconsistent volume layout.

  1. Hot swapping the drives did not make a difference
  2. Rebooting the array did not make a difference.
  3. Testing all of the top rows worth of drives externally showed no errors in any drives
  4. Performing a couple of hard power downs and cold booting did solve the problem and allowed the array to start normally.

The same thing happened during the next test and diagnostics window.

The Fix

Disgruntled customer note: I want to add at this point that trying to phone QNAP UK is an exercise in futility. I sat in their queue system for more than half an hour listening to the same loop of music without getting anywhere. By the time that I gave up, I’d managed to get the thing to start through cold boot cycles and done quick SMART tests on all of the drives. When it came to this occurring the second time, I used the online chat feature with their people in Taiwan, who were more responsive and most helpful, but would not entertain speaking on the phone come what may.

My bigger issue is that QNAP support seems to have little sympathy for the needs of a production IT department, where as you would have thought that QNAP would be the kings of working to such constraints.

 

Once I had got to second line, the fix in itself was quite simple in the end, but its implementation leaves something to be desired. The system needed a BIOS update.

You might assume that when you are updating your “firmware“, this sort of things is being accommodated for in those updates. Apparently not! It seems as this new device may have been sitting in a warehouse for some time so was out of date, but it was immediately firmware serviced as soon as it was first booted. You would have expected that this sort of well know about, intermittent issue was being dealt with through the update delivery mechanism.

As soon as the BIOS of the TVS-1271U was updated to version QW10IR12 and rebooted, the problem was fixed. Why QNAP have not put information on this on their website knowledge base I do not know. This would seem more than sensible. QNAP does itself more reputational damage by trying to hide the issue and hope that only a few people see it. The reality is that they are likely causing stress and grief to end users and unnecessary RMA’s to their suppliers.

After checking power rails, our initial response was that it was a dead backplane and we were assuming it would have to be an RMA. Fortunately, it wasn’t and fortunately I went to QNAP before I went back to the supplier, but you do not always, especially with companies increasingly insisting that you deal with suppliers for RMA processes. A little public disclosure about this known issue would have just saved a lot of headaches (and disclosure makes you look good QNAP!).

The execution of the BIOS update itself was unfortunately quite cumbersome, requiring the support tech to back-up and then re-programme all of the NIC MAC addresses built into the motherboard. Something that could have easily have been sorted in a shall script and then transparently bundled into the firmware delivery, saving QNAP Taiwan more than an hours’ worth of time on this and some 8 emails.

It did however fix the problem and the Taiwan support team were pretty accomodating about doing it at 7am UK time.

Here for the benefit of the rest of the world, is the process that the tech went through to flash the BIOS. I have substituted real MAC addresses with fake ones below.

Note: I strongly recommend that you do not try this yourself and that you contact QNAP via their web chat support if you have a need to perform this procedure. If you try it, it is entirely at your own risk.

login as: admin
admin@192.168.1.1's password:
[~] # md_checker Welcome to MD superblock checker (v1.4) - have a nice day~ Scanning system... HAL firmware detected!
Scanning Enclosure 0...
RAID metadata found!
UUID: 3e5d7c85:95d82d3e:42647860:c0aaec32
Level: raid6
Devices: 12
Name: md2
Chunk Size: 512K
md Version: 1.0
Creation Time: Apr 19 11:57:42 2017
Status: ONLINE (md2) [UUUUUUUUUUUU]
===============================================================================
Disk | Device | # | Status | Last Update Time | Events | Array State
===============================================================================
1 /dev/sdc3 0 Active Jun 16 07:15:08 2017 156 AAAAAAAAAAAA
2 /dev/sdd3 1 Active Jun 16 07:15:08 2017 156 AAAAAAAAAAAA
3 /dev/sde3 2 Active Jun 16 07:15:08 2017 155 AAAAAAAAAAAA
4 /dev/sdf3 3 Active Jun 16 07:15:08 2017 155 AAAAAAAAAAAA
5 /dev/sdk3 4 Active Jun 16 07:15:08 2017 155 AAAAAAAAAAAA
6 /dev/sdl3 5 Active Jun 16 07:15:08 2017 155 AAAAAAAAAAAA
7 /dev/sdm3 6 Active Jun 16 07:15:08 2017 155 AAAAAAAAAAAA
8 /dev/sdn3 7 Active Jun 16 07:15:08 2017 155 AAAAAAAAAAAA
9 /dev/sdg3 8 Active Jun 16 07:15:08 2017 155 AAAAAAAAAAAA
10 /dev/sdh3 9 Active Jun 16 07:15:08 2017 155 AAAAAAAAAAAA
11 /dev/sdi3 10 Active Jun 16 07:15:08 2017 155 AAAAAAAAAAAA
12 /dev/sdj3 11 Active Jun 16 07:15:08 2017 155 AAAAAAAAAAAA
=============================================================================== [~] # cd /share/CACHEDEV2_DATA/Public/
[/share/CACHEDEV2_DATA/Public] # ls
@Recycle/ messages
[/share/CACHEDEV2_DATA/Public] # wget http://download.qnap.com/Storage/tsd/bios/TVS-1271U_QW10IR12.zip
--2017-06-16 07:16:51-- http://download.qnap.com/Storage/tsd/bios/TVS-1271U_QW10IR12.zip
Resolving download.qnap.com (download.qnap.com)... 2.17.149.135, 2a02:26f0:ec:38c::1b52, 2a02:26f0:ec:398::1b52
Connecting to download.qnap.com (download.qnap.com)|2.17.149.135|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5931607 (5.7M) [application/zip]
Saving to: ‘TVS-1271U_QW10IR12.zip’ TVS-1271U_QW10IR12.zip 100%[=====================================================================>] 5.66M 5.50MB/s in 1.0s 2017-06-16 07:16:52 (5.50 MB/s) - ‘TVS-1271U_QW10IR12.zip’ saved [5931607/5931607] [/share/CACHEDEV2_DATA/Public] # chmod +x TVS-1271U_QW10IR12.zip
[/share/CACHEDEV2_DATA/Public] # unzip TVS-1271U_QW10IR12.zip
Archive: TVS-1271U_QW10IR12.zip
creating: BIOS_QW10IR12/
inflating: BIOS_QW10IR12/flashrom
inflating: BIOS_QW10IR12/QW10IR12.bin
[/share/CACHEDEV2_DATA/Public] # ls
@Recycle/ BIOS_QW10IR12/ TVS-1271U_QW10IR12.zip* messages
[/share/CACHEDEV2_DATA/Public] # dmidecode -t bios | grep version
[/share/CACHEDEV2_DATA/Public] # cd
[~] # cd /
[/] # dmidecode -t bios | grep version
[/] # cd -
/root
[~] # cd /share/CACHEDEV2_DATA/Public/
[/share/CACHEDEV2_DATA/Public] # head /etc/config/uLinux.conf
[System]
Model = TS-X71U
Internal Model = TS-X71
Server comment =
Version = 4.3.3
Build Number = 20170606
Number = 0224
Time Zone = Europe/London
Enable Daylight Saving Time = TRUE
Workgroup = QNAP
[/share/CACHEDEV2_DATA/Public] # cat /etc/model.conf | grep INTERNAL_NET_PORT_NUM
INTERNAL_NET_PORT_NUM = 4
[/share/CACHEDEV2_DATA/Public] # hal_app --se_sys_get_mac obj_index=0
35:35:35:35:36:31
[/share/CACHEDEV2_DATA/Public] # hal_app --se_sys_get_mac obj_index=1
35:35:35:35:36:32
[/share/CACHEDEV2_DATA/Public] # hal_app --se_sys_get_mac obj_index=2
35:35:35:35:36:33
[/share/CACHEDEV2_DATA/Public] # hal_app --se_sys_get_mac obj_index=3
35:35:35:35:36:34
[/share/CACHEDEV2_DATA/Public] # cd
[~] # cd /share/Public
[/share/Public] # ls
@Recycle/ BIOS_QW10IR12/ TVS-1271U_QW10IR12.zip* messages
[/share/Public] # cd BIOS_QW10IR12/
[/share/Public/BIOS_QW10IR12] # ls
QW10IR12.bin flashrom
[/share/Public/BIOS_QW10IR12] # ls
QW10IR12.bin flashrom
[/share/Public/BIOS_QW10IR12] # chmod +x *
[/share/Public/BIOS_QW10IR12] # ls
QW10IR12.bin* flashrom*
[/share/Public/BIOS_QW10IR12] # ./flashrom -c MX25L128050 --programmer internal -w QW10IR12.bin
flashrom v0.9.8-unknown on Linux 4.2.8 (x86_64)
flashrom is free software, get the source code at http://www.flashrom.org Error: Unknown chip 'MX25L128050' specified.
Run flashrom -L to view the hardware supported in this flashrom version.
[/share/Public/BIOS_QW10IR12] # ./flashrom -c MX25L12805D --programmer internal -w QW10IR12.bin
flashrom v0.9.8-unknown on Linux 4.2.8 (x86_64)
flashrom is free software, get the source code at http://www.flashrom.org Calibrating delay loop... OK.
Found chipset "Intel C226".
This chipset is marked as untested. If you are using an up-to-date version
of flashrom *and* were (not) able to successfully update your firmware with it,
then please email a report to flashrom@flashrom.org including a verbose (-V) log.
Thank you!
Enabling flash write... Warning: SPI Configuration Lockdown activated.
OK.
Found Macronix flash chip "MX25L12805D" (16384 kB, SPI) mapped at physical address 0xff000000.
Reading old flash chip contents... done.
Erasing and writing flash chip... Erase/write done.
Verifying flash... VERIFIED.
[/share/Public/BIOS_QW10IR12] # hal_app --se_sys_set_mac obj_index=0,value=35:35:35:35:36:81
eth port = 0, Set MAC address = 35:35:35:35:36:31,ret = 0
[/share/Public/BIOS_QW10IR12] # hal_app --se_sys_set_mac obj_index=1,value=35:35:35:35:36:82
eth port = 1, Set MAC address = 35:35:35:35:36:32,ret = 0
[/share/Public/BIOS_QW10IR12] # hal_app --se_sys_set_mac obj_index=2,value=35:35:35:35:36:83
eth port = 2, Set MAC address = 35:35:35:35:36:33,ret = 0
[/share/Public/BIOS_QW10IR12] # hal_app --se_sys_set_mac obj_index=3,value=35:35:35:35:36:84
eth port = 3, Set MAC address = 35:35:35:35:36:34,ret = 0
[/share/Public/BIOS_QW10IR12] # reboot

 

iSCSI MPIO Recommendations & Best Practice on Windows Server

System Requirements:

  • Windows Server 2008 Storage Server
  • Windows Server 2008 R2 Storage Server
  • Windows Server 2012, 2012 R2, 2016

The Problem:

I needed to outline some of the general thinking relating to exactly how a practitioner should logically and physically understand MPIO, however most of the discourse on the subject skips a fair amount of the obvious questions that people starting out with the technology may be asking (or trying to answer). I therefore present some thinking on the subject of understanding MPIO optimisation and best practice for iSCSI.

The information presented in this document is intended for those who are new to the concept of iSCSI and MPIO and is not intended to be product specific.

More Info

Multi-Path Input/Output or MPIO is a server technology that usually sits on the storage side of load balancing, failover and aggregation technologies. If you are getting into SAS, iSCSI or Enterprise RAID solutions where it is most commonly used (encountered), then you this may (or may not) help you with understanding what MPIO is any why it (possibly) isn’t what you think it is!

The document is written from the perspective of an iSCSI user where it can be conceptually a little harder for new users to understand the best way to approach MPIO.

Logically understanding what MPIO is all about

So you have 2x1Gbps ports in a MPIO team, that means you’ve got a 2Gbps link right? Wrong. That isn’t what is going on with MPIO.

MPIO (and in fact pretty much the majority of balancing and aggregation technologies) doesn’t double the speed, but it does roughly double the bandwidth available to the system. Confused? Think of it like this:

You own a car. The car has a top speed of 70mph and not one mph more. You get on a one way, single track road in a country where there are no speed limits. You are now happily driving along at 70mph. Some bright spark at your local council decides that you should be able to drive at 140mph, so they cut down the trees on one side of the road and add a second one-way carriage way, going in the same direction as the first.

Can your car now drive at 140mph because of the new lane? No. The public official is wrong. Your engine can only offer you 70mph. The extra lane doesn’t help you, but it does help the guy in the car next to you also driving at 70mph arrive at the other end of the road at the same time. It also means that when you encounter a tractor ambling along in your lane, you have somewhere else to go without slowing down.

This is fundamentally what MPIO is doing. So why isn’t it a 2Gbps link? Basically, because networking technology is a serial communications medium and by adding a second lane and calling it a faster way to get data to the end of it, you get into the different world of parallel communications. Under parallel communications you have to split (fragment) information into smaller pieces and push it down each one of the wires to the destination. This in turn infers the need to have more complicated buffer/caching designs to store information as part of a strategy that is designed to be able to cope with each section of the data arriving at a different time, it arriving all at the same time, in a different order than intended or of course, it not arriving at all. Something known as clock skew.

To fix this, you need to introduce overhead either to synchronise delivery to be reliable (thus slowing it down and reducing error tolerance) or adding overheard mechanisms designed to deconstruct, sequence, wait for or re-request missing or corrupt data sections and track timing – alls something that you really don’t want in an iSCSI or SAS environment where response time (latency) is king. Consequently, there is a diminishing return on how much of this parallel working you can derive a benefit from in any system, including an MPIO system. iSCSI MPIO, if correctly configured, will offer something at around about the boundary between worth-while and not bothering in the first place. Yet it is important to understand that it will not be a 100% increase in performance, neither will likely be a 50% increase, but more realistically something around the 30-40% mark.

Performance is only one of the intended design considerations for MPIO, and in that it is not the primary consideration. The primary consideration is for fault tolerance and reliability.

In a correctly designed iSCSI system, independent NICs connected to more than one switch and usually to more than one controller on the storage side and more than one server on the host side. If one of these fails, in a correctly implemented system, your production service probably won’t even notice. You can even be as bold to perform live switch re-wiring on iSCSI systems without impacting the client services involved – although it should be stressed that this is for bragging rights and in practice should not be attempted.

To summarise, MPIO allows you to get twice (+) as much data down to the end of the link, but you cannot get it there any faster. In general, if you can avoid using fragmented streams, you will reap the maximum benefit. The obvious approach here is that each “lane” should be using unrelated data: instead of carving up a single video file and pushing little bits of it down each lane one bit at a time (MPIO can do this), one lane is used for the video and the second lane is used for literally anything and everything else. This is a simplification of what MPIO generally does, however in practice is offers a good way to get your head around it.

Techniques

So how does MPIO carve up the traffic?

There are broadly speaking 4 different paradigms for carving up MPIO traffic

Failover/Redundant In this mode, one link is active, while the other is passive i.e. up, but not doing anything. If the first link fails, the second path takes over and all existing traffic streams continue to receive the same bandwidth (% of the total available pie) on the same terms as before. This would give us a completely separate road that can only be used in emergencies. It may not be as fast or robust, or it may be identically spec’d and just as capable. A failover design may or may not return traffic to the first channel once it becomes available once again.
Round Robin This mode alternates traffic between channel 1 and channel 2, then goes back to channel 1, channel 2 and so on. Both links are active, both receive traffic in a slight skew as the data is de-queued at the sender. This offers the 2 lane analogy used above with each 70mph car getting to the end at roughly the same time.
Least Queue Depth This puts the traffic into the channel that has the least amount on it (or to be more accurately about to go onto it). If one channel is busier than the other (e.g. the large video file) then it will put other traffic down the second channel, allowing the video to transfer without needing to slow down to allow new traffic to join, delaying its delivery. There are many different algorithms that exist on how this is achieved, including varieties that use hashing to offer clients consistent paths based upon Layer 2 or Layer 3 addresses.
Path Weighting Weighted paths and least blocking methods assess the state/capabilities of the channels. This is more useful if there are lots of hops between source and destination, multiple routes between a destination or different channels have different capabilities. For example, if you have iSCSI running through a routed network, then there could be multiple ways for it to get there. One route may go through 5 routers and another 18 routers. Generally, the 5 router path might be preferable, provided the lower hop route genuinely gets the data there faster. Equally the weighting could be based upon the speed of the path through to the recipient or finally, if channel 1 is 10Gbps and channel 2 is only 1Gbps, then you might prefer the 10Gbps path to be used with a higher preference. Usually, a lower weighting number means a higher preference. This would be the equivalent of a 70mph road with a backup road with a max speed of 50mph. You know that it will get you to the destination, but you can guarantee that if you have to use it, it will take longer.

So, more lanes equal more stuff then?

Sounds simple doesn’t it? Just keep throwing lanes into the road and then everyone gets to travel smoothly at 70mph.

In principle, it is a nice idea, but in practice it doesn’t actually work in most iSCSI implementations.

For starters, server grade network card (which you should be using for MPIO, and not client adapters) are expensive and server backplanes can only accept a finite amount of them. Server NIC’s also consume power and power consumes money! Keep that in mind if you do decide to throw extra ports at an iSCSI solution.

The reality is that if you have an MPIO solution that will allow you to experiment with more than 2 NIC adapters in a MPIO group, you will likely see the performance gain rapidly tail off. In turn it will actually wind up presenting you with steadily worsening performance, not the increase that you are expecting.

Attempting to MPIO iSCSI traffic across 4x 1Gbps NIC’s actually offers worse read and write speeds for a virtual machine than 2x 1Gbps under a Hyper-V environment (see tests E and F below). The system starts to waste so much time trying to break apart and put back together each lane’s worth of traffic that it just doesn’t help the hypervisor.

Where a 4 NIC configuration is beneficial is actually in providing you with a “RAID 6” MPIO solution. Here you can have 2 active and 2 passive adapters – remember in an idealised scenario they could be 2x10Gbe and 2x1Gbe with a hard-coded preference for the 10Gbe and a method of failing traffic back to the 10Gbe. Just be aware that you can only use the 10Gbe set OR the 1Gbe set at the same time, not one port from each. The exception to this rule is for hashing based channel assignment as these offer more paths to “permanently” assign data into, without the overhead of path swapping or de-fragmentation of traffic.

Some DSM’s (effectively a OEM specific MPIO driver under Windows, such as Dell Host Integration Tools [HIT] or NetApp Host Utilities) logically limit a MPIO to two active NIC’s if the storage controller is only exposing 2 usable NICs back to the HIT instance. Dell EqualLogic Host Integration Tools (the EqualLogic DSM) will grab the first two paths it finds and shutdown any others into a passive state, no matter how hard you try to start them up.

What should a MPIO network “look” like?

Ultimately this is down to what you want to get out of the MPIO solution and within the bounds of what your hardware vendor will support.

There are effectively three schools of thought here (I won’t comment on which is right because as you’ll see, it isn’t that simple)

MPIO is about Meshing

If you see MPIO is a mesh then 2 NIC’s in a server connecting to 2 NIC’s in a storage appliance equals a mesh where each NIC has a path to the other. This is more aligned with how you probably already think about Ethernet networks.

MPIO is about Pathing

If you see MPIO in this model it is simple about more than one line being drawn between two different end points, with no line crossing or adding any complexity, complication and confusion. This is more aligned with how you likely currently think about SAS, Fibre Channel and hard drive wiring.

MPIO is about Redundancy

This is the purest of the three views. It sees the complexity and overheads associated with MPIO as being a problem – there will always be some sort of increase in latency, a drop in some aspect performance by trying to squeeze more bandwidth out of MPIO. This view attempts to keep the design simple, run everything at an unimpeded wire speed but maintain the failover functionality afforded by MPIO.

The three schools of thought are outlined in the diagram below.

Why not Meshing?

When you start out with MPIO, you may be tempted towards implementing option 1. After all, your Server NICs (circles) are likely connected to a switch, as is your storage array (squares). The switch allows you to design to this topology and if you allow the MPIO system to have knowledge over all possible permeations of connectivity, the system will highly redundant, making it very robust.

Yes and no! Yes, it is very robust, but at this point in your implementation, how do you know which path traffic is taking? How do you know that it is optimised? What is stopping Server NIC1 and Server NIC2 from both talking to storage NIC1 at the same time? If they do that, then they have to share 1Gbps of bandwidth between them while Storage NIC2 is left idle. Suddenly all of your services will have intermediate bursts of speed and infuriating drops in performance. The more server NICs that you add, the faster the decrease in performance will be. With 4 Server NICs, there is nothing to stop the MPIO load balancer from intermittently pushing the data from all 4 Server NICs towards a single Storage NIC.

In a Round Robin setup, in a full Mesh design (as shown in #1) it will likely order the RR protocol in the order that you gave the system access to the paths. Given the following IP Addresses

Server: 192.168.0.1, 192.168.0.2
Storage: 192.168.0.11, 192.168.0.12

The RR table could like this

  1. 192.168.0.1 -> 192.168.0.11
  2. 192.168.0.2 -> 192.168.0.11
  3. 192.168.0.1 -> 192.168.0.12
  4. 192.168.0.2 -> 192.168.0.12

Or it could like like this

  1. 192.168.0.1 -> 192.168.0.11
  2. 192.168.0.1 -> 192.168.0.12
  3. 192.168.0.2 -> 192.168.0.11
  4. 192.168.0.2 -> 192.168.0.12

In both examples you either have two different sets of traffic being sent from the same Server NIC concurrently or received by the same Storage NIC concurrently. This is going to undermine performance, not improve it (this is outlined in Mbps terms in the tests shown later in this document).

In a failure situation, the performance issue is exacerbated

  • If #3 fails, then nothing changes in performance or bandwidth.
  • If #2 fails then the total bandwidth available to the system halves and all services contend using the first link.
  • If #1 fails then as with #2, all services suffer with contended bandwidth, however the system also has the overhead of MPIO to further reduce performance.

What benefit is there to MPIO operating in scenario #1? In this failed state, should one of the Storage NICs also fail, the system will continue to operate. In #2 if the working Storage NIC fails, the entire system will fail despite the fact that the Storage NIC on the second path is actually working. It is up to you and your design as to whether you think that the performance hit that you will experience is worth this extra safeguard? In a highly secure system, mission critical or safety system it may be worth the extra overhead.

There are however some middleware layers that can manage this for you. Dell Host Integration Tools (HIT), does, for example, attempt to undertake some management of these types of situations, optimising the mesh by putting the links that will cause overhead into a failover only state, while maintaining the optimal number of active mesh links. In my experience though, the HIT solution is not able to perfectly manage the optimal risk. It does not provide any consideration over redundant NIC controllers. For example, if you have 2 physical Dual Port NICs in your Server with the intention of one port from each NIC making up the active “pair”, Dell HIT is not able to detect or be programmed to ensure that the active paths are prioritised around ensuring that the correct controller is being used. In my experience, it will tend to bunch them together onto the same physical NIC controller, leaving the second controller idle.

Fixing this problem requires an additional layer of complex, expensive and usually proprietary middleware logic, further impacting performance and increasing cost. Therefore, industry best practice is to avoid thinking of iSCSI MPIO as being a Full or even a Partial Mesh, but instead think of it as offering independent channels akin to those shown in #2. It is for this reason that virtually all iSCSI MPIO vendors insist that each Server -> Storage NIC pair exist on its own logical IP subnet as this completely negates the possibility of interweaving the MPIO paths while also ensuring that any subnet-local issue (such as a broadcast or unicast storm) is only likely to take down one of the subnets, not both.

iSCSI as part of a Virtual Network Adapter, Converged Fabric LBFO Team

Since the release of Windows Server 2012, Microsoft have allowed to be hinted at the idea of using iSCSI through Converged Fabric* Load balancing Failover (LBFO) teams — as long as the iSCSI NICs are Virtual and they connect through a Hyper-V VM Switch which itself backs onto a Windows Server LBFO team. Even the venerable Aidan Finn has hinted at it. I have, however, never seen a discussion of it being attempted online, neither have I ever seen it benchmarked.

To be clear over what we are talking about when I say a Virtualised, Converged Fabric, LBFO Team:

  1. 4x 1Gbps Ethernet physical adapters
  2. Grouped into a Windows Server 2016 LBFO Team, appearing to Windows as a single logical network adapter called “ConvergedNIC”
  3. “ConvergedNIC” is connected to an External Virtual Switch called “ConvergedSwitch”
  4. A Virtual Machine Network Adapter is created on the Hypervisor’s Parent Partition (ManagementOS) and this is assigned to the correct VLAN, given an IP address and hooked up to the iSCSI Target
  5. 4 physical NICs, no MPIO, 1 logical NIC

So, does it work?

Yes! It does work and it appears to be stable and even usable; but with some sacrifice in performance (keep reading for some benchmark numbers as “test A” below). I have however had test VMs running under this design for nearly a year without any perceivable issues in either VM or hypervisor stability.

* If you are not familiar with the Concept of a Converged Fabric: A Converged Fabric is a data centre architecture model in which the concept of 1 NIC = 1 Network/Subnet/VLAN/Traffic Type is abandoned. Instead, NICs are usually pooled together into Teams with multiple traffic types, Networks, Subnets and VLANs being allowed to use any of the available bandwidth within the team. Quality of Service (QoS) algorithms are used to ensure that priority traffic types are defined (such as iSCSI in this example), ensuring that the iSCSI system is never starved for bandwidth by someone performing a large file transfer across the team. A Converged Fabric architecture is considered to be more efficient, lower cost and offer better failover reliability than traditional methods in which entire 1GbE or 10GbE NICs could be left idle, waiting for traffic that while high bandwidth, may be infrequent. A Converged Fabric architecture allows other users/systems to benefit from the available bandwidth when not needed by its primary application. It can also offer the primary application additional bandwidth in some situations.

If you have an 8 NIC Hypervisor setup with 2 physical iSCSI NICs, 2 physical production network NICs, 1 physical heartbeat NIC, 1 physical live migration NIC, 1 management network NIC and 1 out of bounds management NIC, then you are paying to power but to not derive much of any benefit from NICs 4-8 due to how infrequently they are used. If this sounds familiar to you, then you should consider migrating to a Converged Fabric design.

Quantifying Best Practice

So far, this article has discussed MPIO, meshing, pathing and redundancy as well as a quick detour into using converged fabric LBFO for iSCSI connections. So let’s look at some numbers that underpin these approaches.

Tests were undertaken using the following hardware configuration:

  • Dell EqualLogic PS4110x running firmware 9.1.1 R436216, with 2 active 1GbE NIC’s on a single controller
  • Dell PowerEdge P630 with 8x1GbE adapters (4x Broadcom NetXtreme and 4x Intel I350 adapters) with 9K Jumbo Frames correctly enabled
  • Windows Server 2016
  • Switching on Cat6a cabling via 2x Cisco Catalyst 2960-48’s
  • The 64K block, GPT formatted, 3TB target LUN was setup as a CSV and the nodes were in a Cluster with a second identical node idling as a second cluster member (CSV-FS has a natural performance hit compared to NTFS)

7 tests were performed as outlined in the following table

Physical Paths
Active NICs
Test Description Active Passive Intel Broadcom LBFO Team Dell HIT MPIO Mode
A
4 NIC in LBFO Team, No MPIO
4
0
0
4
Y
N
n/a
B
4 NICs, fully meshed, RR
8
0
2
2
N
N
Round Robin
C
2 NICs, no mesh (point to point)
2
2
2
0
N
N
Round Robin
D
1 NIC only (control test)
1
1
1
0
N
N
n/a
E
4 NICs, fully meshed, LQD
8
0
2
2
N
N
Least Queue Depth
F
4 NICs, partial mesh, RR
4
0
2
2
N
N
Round Robin
G
2 NICs, no mesh (point to point) with EqualLogic Host Integration Tools
1
1
2
0
N
Y
Least Queue Depth

If you are more visual, the following diagram summarises the above in a graphical format

The Results

The following table summarises the read/write performance of each test on Sequential 4MB reads as outlined through “Anvil’s Storage Utilities”, version 1.1.0, build 1st January 2014. all tests were performed on the same Windows 10 Enterprise VM without rebooting in between each test and without performing any other activities on the VM disk.

The results below are ordered by test, from the test offering the best performance to the test offering the worst performance, using the Read MB/s column as the sort index.

Sequential 4MB (Read)
Sequential 4MB (Write)
Test
Response (ms)
MB read
IOPS
MB/s
Control Deviance (%)
Response (ms)
MB written
IOPS
MB/s
Control Deviance (%)
C
30.4791
1052
32.81
131.24
32.17
21.7266
1024
46.03
184.11
70.25
F
39.801
804
25.13
100.50
1.21
468.9896
772
2.13
8.53
-92.11
D
40.2814
796
24.83
99.30
0
36.9883
1024
27.04
108.14
0
A
51.3782
624
19.46
77.85
-21.60
89.5977
1024
11.16
44.64
-58.72
G
60.7197
528
16.47
65.88
-33.66
23.8047
1024
42.01
168.03
55.38
E
273.9667
120
3.65
14.60
-85.30
1010.7556
360
0.99
3.96
-96.34
B
404.65
80
2.47
9.89
-90.04
964.766
376
1.04
4.15
-96.16

Response (ms) = Lower is better
MB read/written = Higher is better
IOPS = Higher is better

Control Deviance (%) = the positive or negative impact in MB/s performance compared to the single NIC, no MPIO control test (test D).

Test A | Converged Fabric LBFO

The Microsoft dream of virtualising everything does hold up – at not being completely terrible. Sitting in the middle of the table, using a fully converged fabric, virtualised setup across 4 NICs resulted in a 22% reduction in read speed compared to a single NIC and a 59% reduction in write speed.

There may be some improvements to made by creating multiple Virtual iSCSI interfaces connected to the virtual switch, however these were not tried. Based upon the current view of the technology, while it works and offers a data centre design simplification, that simplification factor is not worth the performance sacrifice.

Test B | Round Robin, Full Mesh

This test proves that viewing an iSCSI setup as a full mesh and throwing NICs at the proverbial problem is going to do nothing to help you. Your iSCSI should be configured in a 1:1 “path” setup between initiator and target. Any additional NICs should be put into “Round Robin with subset” i.e. made to be passive fail-over adapters. That is a 90% and 96% reduction in respective read/write performance!

Test C | Round Robin, 1:1 Paths

This test proves how you are supposed to use iSCSI. Two, non-crossing paths allows for a full bandwidth connection down each path between the initiator and the target. This configuration provided an increase in performance over a single adapter and was the only test that provided improvements to both read and write metrics.

Test D | Control

This was the baseline control test for this experiment. 1 NIC talking to 1 controller port. Nothing complicated here.

Test E | Least Queue Depth, Full Mesh

This test repeated Test B, but changed the MPIO model from RR to LQD to see if it made any difference. Read performance was slightly better than under RR, but was still 85% worse than the control test.

Test F | Partial Active Mesh

This test looked to see whether having a partial active mesh made any difference. There was a very small 1% increase in read performance from this, but a significant write penalty. In practice, you cannot push/pull 2Gbps to/from a 1Gbps source, so the design is not conducive towards improved speed under a synthetic load.

Test G | Least Queue Depth, 1:1 Paths

Test G was a genuine surprise. I was expecting to see Dell EqualLogic Host Integration Tools (HIT) version 4.9 offer an increase in performance, not a decrease. However, repeating the test yielded the same results. In my experience, this has never usually been the case, with VM’s feeling more responsive with HIT installed compared to not. Experience suggests to me that something else was at play here, perhaps the HIT version being poorly optimised for Windows Server 2016, or the Dell stack getting grumbly about it using a retail Intel I350-T4 adapter instead of a Dell one. Dell HIT forces the use of pathing no matter what you try and set all other adapters into passive mode. It used LQD as the MPIO algorithm. Evidently this resulted in an increase in writes but a reduction in read performance, be it not as high as without HIT being installed.

Although not shown in the results above, HIT did help improve performance in some of the Anvil Tests. The long queue depth tests resulted in higher IOPS figures for both read and write values by a small margin. None of the other tests yielded such an improvement.

Conclusion

As you can see from these results. There is only one way that you should be conceptually thinking about your iSCSI environment – 1:1, point to point paths. Anything over and above this should be set to being passive/failover/offline in order not to impact performance.

General Subnet Recommendations

Subnet recommendations go hand in hand with this, but you should note are generally made by the storage vendor — and you should follow their advice. I have encapsulated the general recommendations/requirements of a number of providers in the table below. The subnet count column is in essence a statement that for each NIC on the storage device, there should be a dedicated subnet (and ideally broadcast domain/VLAN) back to the iSCSI server.

Vendor Subnet Count Source
Dell (Non-EqualLogic)
2 View
Dell EMC
2 View
Dell EqualLogic
1 View
Microsoft
2 View
NetApp
? I couldn’t find any guidance from an official source. There is community evidence of both being used by end-users
NetGear
2 View
QNAP
2 View
Synology
2 View

As you can see, with the exception of Dell EqualLogic which provides a middleware solution known as the Host Integration Tools (HIT) to cope with this, most vendors are quite specific on the use of a “single path” logical topology for server/storage connectivity — aka one subnet per storage appliance NIC.

General Advice

I will end this piece with some general advice and tips for working with MPIO. It isn’t exhaustive, but they are some quick observations from experience of using the technology for many years. Some of them are obvious, some of them might help you avoid a head scratcher.

  1. If you are using an enterprise iSCSI solution, follow the vendor’s advice, forget anything you read on the Internet. Everyone is a know-it-all on the internet and there are plenty of “I’m a Linux user so I know best” screaming matches about how EqualLogic are wrong about the recommendations for EqualLogic’s own hardware. I’m pretty sure that EqualLogic… uh, tested their stuff before writing their user manual.
  2. If you are using an enterprise solution and the vendor offers a DSM (MPIO driver), use it. Dell HIT vs the generic Microsoft DSM for Windows Server is noticeably faster, but only works will Dell SAN hardware (naturally). Also ensure that you keep you DSMs up to date.
  3. Follow you vendor’s guidelines with respect to subnets. If in doubt, drop them an email. You’ll usually find them quite accommodating.
  4. Unless your vendor has expressly told you to, you do not MPIO back from the storage system – i.e. don’t team, MPIO, load balance etc on the storage side. Do it all on the server initiating the request.
  5. Stick to two port/1:1 path MPIO designs. If you need more create multiple pairs and have each on different networks going to different storage systems so that the driver knows where to send traffic explicitly while maintaining isolation.
  6. If you want to think about your MPIO as a meshing design, it has to be meshed for redundancy, not active links (unless your system needs to keep living, breathing human beings alive and do so at all costs).
  7. With iSCSI and SAN MPIO, try and avoid network hops (routers).
  8. All ports in a group must be the same type, speed and duplex.
  9. Disable port negotiation and manually set the speed on the client and switch, this will make failover/failback processes faster for your redundant paths.
  10. Use VLAN’s as much as possible (try and avoid overlaying broadcast domains across a shared Layer 2 topology).
  11. Use Jumbo Frames as much as possible unless the iSCSI subnet involves client traffic.
    Hint: Your iSCSI subnet should not involve client traffic!
  12. Ensure that your NIC drivers and firmware are kept up-to-date
  13. Disable all Windows NIC service bindings apart from vanilla IPv4 on your iSCSI networks. For example, Client for Microsoft Networks, QoS Packet Scheduler, File and Printer Sharing for Microsoft Networks etc. If you aren’t using it, disable IPv6 too on the iSCSI interfaces to prevent IPv6 node-chatter.
  14. In the driver config for your server grade NIC (because you are using server grade NIC’s, right?) max out the send and receive buffer sizes on the iSCSI port. If the server NIC has iSCSI features that are relevant (such as iSCSI offloading), enable them.
  15. When you are building a Windows Server, script the MPIO install, enable MPIO during the script and set the default policy as part of the build process —- then patch and REBOOT the system before you even start configuration. If I had a £1 for every time I’d had to rescue someone from not doing that and then not REBOOTING…
  16. If you are using a SOHO/SME general purpose commodity NAS, if (and only if) you have a UPS, disable Journaling and/or Sync Writes on your iSCSI partitions/devices. There is a benefit, but remember if you are hosting SMB shares on a commodity appliance you actually do want Journaling running on those volumes.
  17. Keep your NAS/SAN firmware up to date.
  18. Keep your storage system and iSCSI block sizes, cluster and sector sizes optimised for the workload. Generally this means bigger is better for virtualisation storage and video. 256/64K, 128/64K or 64/64K depending on what your solution can offer.
  19. Keep volumes under 80% of capacity as much as possible.
  20. Use UPS’s: Remember, iSCSI and SAS are hard drive/storage protocols. They are designed to get data onto permanent storage medium just like RAID controllers. RAID controllers have backup batteries because you do not want to lose what is in process in the RAID controller cache when the power goes out. Similarly, you need to think of your iSCSI and External SAS sub-systems much the same as you would a RAID sub-system.
  21. If you have a robust UPS solution, enable write caching and write behind/write back cache features on your storage systems and iSCSI mounted services to gain extra performance benefits. Be mindful that there is risk in this if your power and shutdown solution isn’t bullet proof.
  22. Test it! Build a test VM and yank a cable out a few times. You’ll be glad you sacrificed a Windows install or two to ensure it is right when you actually pull an iSCSI cable out of a running server… Believe me I know what a relief that is.

Netgear ReadyNAS Duo V2 and Jumbo Frames

System Requirements:

  • Netgear ReadyNAS Duo V2
  • Firmware 5.3.12
  • 9k compatible NIC’s and intermediate Layer 2/Layer 3 hardware

The Problem:

The ReadyNAS Duo V2 is a now legacy, ARM based dual 3.5″ drive SOHO NAS appliance with a single NIC and 256MB of RAM. The device was never intended to support performance or even low-end enterprise tasks.

The ReadyNAS Duo V2 is not designed for Jumbo Frames and there are no user interface entry points to enable it. It is not clear on-line whether anyone has had any actual success with enabling it. This document explores the issue.

More Info

In wanting to use the device simply as a on-line backup appliance, I wanted to try and squeeze as much performance out of it as I could. One of the obvious things to try is to enable Jumbo Frames, which allows more data to be transmitted in a single Ethernet Frame before the data has to be re-wrapped in a new header and trailer for Layer 2 transmission over an Ethernet Network. The logic being that the fewer CPU cycles being used to process the header and generate and process the CRC and footer, the faster the transfer into memory and thus the smoother the transmission of the data can occur into the disk sub-system.

In order the enable Jumbo Frames, the Maximum Transmission Unit (MTU) has to be adjusted on ALL devices in the transmission path – sender NIC, receiver NIC and any and all intermediary switch ports, bridge ports and router ports. If any one devices does not have Jumbo Frames configured to the same (or higher) value, it will fault. If you have one devices with a higher value and the other devices with a lower value, you will almost certainly see a performance reduction when transmitting from that device. Therefore: Set all of your devices to the same, common MTU values.

Depending on the manufacturer, device and driver these are usually:

Name Rounded Offset
Normal
1500
1514
3K
3000
3014
4K
4000
4088
5K
5000
5014
7K
7000
7014
Note: As a general rule of thumb, if you are using a PCI NIC, this is the largest MTU you can hope to achieve. PCI Express NIC’s can go up to 9k
9K
9000
9014
Note: Specialist enterprise grade hardware is required for MTUs larger than 9K and these sizes are not usually available on 1Gbps NIC hardware (10 or 40Gbps hardware or higher)
16K
16000
16128
24K
24000
32K
32000
64K
64000
Note: This is the maximum transfer size of a TCP Segment

Step 1

This was only tested on Firmware Version 5.3.12. Ensure that your firmware is up to date.

Step 2

You will need to enable Root SSH Access on the device via the official Netgear plugin. Install the plugin via the web UI and reboot the NAS before attempting to proceed.

Download: Enable Root SSH Access Plugin

Step 3

Set a static IP address on the device (or at the very least set and then unset it) to ensure that the config files have been written out correctly. It appears that a clean OS install does not make use of the interfaces file as expected until after a static IP address has been set.

Step 4

SSH into the device (using Bash, Putty for Windows or your preferred client). Usually this is root@<ipaddress>

Step 5

Perform a test to see whether Jumbo Frames currently works

ping <ipaddress> -l 8000

If you receive a successful “reply from…” then it is already working between your ReadyNAS and your PC. The expected result is however for this to fail, indicating that Jumbo Frames is not enabled

Step 6

Perform a volatile test by enabling Jumbo Frames for the session. If you lose contact with your single NIC ReadyNAS, simply reboot it to restore functionality. Under the SSH session issue the following command. If you need to set a lower Jumbo Frame value (for example 7k) change the 9000 value as appropriate.

ip link set dev eth0 mtu 9000

Step 7

Repeat the ping test from your PC

ping <ipaddress> -l 8000

This should now be successful

Step 8

If you wish to make the change permanent so that the setting persists after a restart of the ReadyNAS you must edit the interfaces file. Return to the SSH session

vi /etc/network/interfaces

In VI press i to commence insert mode

Find the entry for the eth0 interface and at the bottom of the section enter mtu 9000 (or the frame setting that you require) e.g.

iface eth0 inet static
address 192.168.0.100
netmask 255.255.255.0
gateway 192.168.0.254
mtu 9000

Note: That is LOWER CASE “mtu”

To save and exit VI press the Escape key and then type :wq (colon, w, q) and press return

Finally type reboot to restart the ReadyNAS

Step 9

Repeat the ping test and you should find Jumbo Frames working

Everything went wrong and now I cannot access my ReadyNAS

Don’t panic. Just access the boot menu and put it into OS Reinstall mode

  1. Turn the ReadyNAS off
  2. Use a paper clip to hold in the reset button on the back
  3. Keep the clip held in place and turn the ReadyNAS on
  4. Hold the paper clip in for 10 seconds
  5. Release the paperclip
  6. Push the backup bottom on the front of the ReadyNAS until the Disk 2 LED is the only one illuminated (be very careful that it is Disk 2 and not Disk 1. Disk 2 reinstalls the OS, Disk 1 factory resets the device and deletes all of your data)
  7. Use the paperclip one more time and single press the reset button to execute the boot menu mode
  8. Come back in 20 minutes and use RAIDar to find your ReadyNAS again (you will have to reconfigure the settings)

Does is make a difference?

Comparing the transfer speed before and afterwards does yield a significant improvement in write speed on the device.

The test configuration

The Source

  • 3.4GB ISO
  • Windows Server 2012 R2
  • From an NTFS formatted 5 disk RAID 5 array on a LSI MegaRaid 8260-8i with caching an optimisation’s enabled
  • A Quad Port Intel I350-T4 Gigabit Server NIC with Jumbo Frames set to 9014 bytes
  • Cat 5e cabling
  • 3 intermediate switches all supporting Jumbo Frames

The Destination

  • ReadyNAS firmware 5.3.12
  • Dual WD Red 3TB WD30EFRX-68AX9N0 with 64MB cache (only 5400 RPM)
  • Write caching enabled on the ReadyNAS
  • The ReadyNAS is running in Flex-RAID RAID 0

To enable write caching over SSH:

hdparm -W1 /dev/sda
hdparm -W1 /dev/sda1
hdparm -W1 /dev/sda2
hdparm -W1 /dev/sda3

Transferring the same 3.4GB ISO from the same Windows Server 2012 R2 NTFS volume over the same NIC/Network/Switches with the MTU set at the default 1500 resulted in a non-burst transfer speed of around 43MB/s (344Mbps).

Repeating the transfer with Jumbo Frames set to 9000 enabled increased this noticeably 53MB/s (424Mbps); a 18.86% increase in write speed.

Given that 53MB/s isn’t that stellar in the first place, this improvement is certainly worth having.