When you are designing your storage subsystem. On modern hardware, you will often be asked to choose between formatting using 512 byte or 4K (4096 byte) sectors. This article discusses whether there is any statistically observable performance difference between the two in a 512 vs. 4K performance test.
NB: Do not get confused between the EXT4 INODE size and the LUN sector size. The INODE size places a mathematical cap on the number of files that a file system can store, and by consequence how large the volume can be. The sector size relates to how the file system interacts with the physical underlying hardware.
Method
- A QNAP TS-1277XU-RP with 8x WD Red Pro 7200 RPM WD6003FFBX-68MU3N0 drives running firmware 83.00A83 were installed with 8 drives in bays 5 – 12
- Storage shelf firmware was updated to QTS version 4.3.6.0923, providing the latest platform enhancements
- A Storage Pool comprising all 8 disks in RAID 6 was configured, ensuring redundancy
- A 4GB volume was added allowing QNAP app installation so that the systme could finish installing
- The disk shelf was rebooted after it had completed its own setup tasks
- RAID sync was allowed to fully complete over the next 12 hours
- Two identical 4096 GB iSCSI targets were created with identical configurations apart from one having 512 byte and the other 4k sector sizes
- SSD caching was disabled on the storage shelf
- 2x10Gbps Ethernet, dedicated iSCSI connections were made available through two Dell PowerConnect SAN switches. Each NIC on its own VLAN. 9k jumbo frames were enabled accross the fabric
- A Windows Server 2016 hypervisor was connected to the iSCSI target and mounted the storage volume. iSCSI MPIO was enabled in Round Robin mode. Representing a typical hypervisor configuration
- The two storage LUNs were formatted with 64K NTFS partitions (recommended for dedicated VHDX volumes)
- A Windows 10 VM was migrated onto each of the targets and the test performed using Anvil’s Storage Utilities 1.1.0.20140101. The VM had no live network connections. The Super Fetch and Windows Update services were disabled, preventing undesirable disk I/O. The VM was not rebooted between tests, had no other running tasks and had been idling for 6 hours prior to the test
- No other tasks, load or data were present on the storage array
512 vs. 4K Performace Results
The results of the two tests are shown below.
| IOPS | |||||
| 512 byte | 4K | 4K Diff | 4K Diff % +/- | ||
| Read | Seq 4MB | 417.45 | 403.3 | -14.15 | -3.51 |
| 4K | 3001.56 | 3164.56 | 163 | 5.15 | |
| 4K QD4 | 6021.45 | 6006.7 | -14.75 | -0.25 | |
| 4K QD16 | 24228.16 | 24062.61 | -165.55 | -0.69 | |
| 32K | 2742.39 | 2807.47 | 65.08 | 2.32 | |
| 128K | 2628.86 | 2620.8 | -8.06 | -0.31 | |
| Write | Seq 4MB | 233.2 | 230.79 | -2.41 | -1.04 |
| 4K | 2090.79 | 2165.45 | 74.66 | 3.45 | |
| 4K QD4 | 5976.18 | 5983.65 | 7.47 | 0.12 | |
| 4K QD16 | 8254.84 | 7874.67 | -380.17 | -4.83 | |
Analysis and Recommendations
The results show that there is little difference between the two. Repeating the tests multiple times showed that the figures for both the 512 byte and 4K LUNs are within the margin of error of each other. A bias towards 512 byte was consistently present, but was not statistically significant.
The drives in the test disk array are 512e drives. 512e is an industry transition technology between pure 512 byte and pure 4K drives. 512e drives use physical 4K sectors on the platter, but that the firmware uses 512 byte logic. A firmware emulation layer converts between the two. This creates a performance penalty during write operations due to the computation and delay of the re-mapping operation. Neither sector size will prevent this from occurring.
My recommendations are
- If all of your drives are legacy 512 byte drives, only use 512
- Should you intend to mount the LUN with an operating system that does not support 4K sectors. Only use 512
- In situations where you have 512e drives, you can use either. Unless you intend to clone the LUN onto 4K drives in the future, stick with 512 for maximum compatibility
- Never create an array that mixes 512 and 4K disks. Ensure that you create storage pools and volumes accordingly
- Where all of your drives are 4K, only use 4K