Tag: Troubleshooting

System Requirements:

• QNAP TVS-1271U
• QTS 4.3

The Problem:

A brand new, out of the box 12 drive 2u NAS appliance, with the “firmware” up to date gets thrown into production. During its first maintenance cycle, the firmware is updated again and it is rebooted.

On completion of the reboot, the array is offline. Visual inspection of the appliance reveals that the top row of drives in the array, drives 9, 10, 11 and 12 are offline. They just are not powered. Drives 1-8 are online and all green.

Rebooting the array shows the diagnostic LED’s on all 12 drives flash red, but the 4 drives in question then go into a powered down state and the boot completes with an inconsistent volume layout.

1. Hot swapping the drives did not make a difference
2. Rebooting the array did not make a difference.
3. Testing all of the top rows worth of drives externally showed no errors in any drives
4. Performing a couple of hard power downs and cold booting did solve the problem and allowed the array to start normally.

The same thing happened during the next test and diagnostics window.

The Fix

Disgruntled customer note: I want to add at this point that trying to phone QNAP UK is an exercise in futility. I sat in their queue system for more than half an hour listening to the same loop of music without getting anywhere. By the time that I gave up, I’d managed to get the thing to start through cold boot cycles and done quick SMART tests on all of the drives. When it came to this occurring the second time, I used the online chat feature with their people in Taiwan, who were more responsive and most helpful, but would not entertain speaking on the phone come what may.

My bigger issue is that QNAP support seems to have little sympathy for the needs of a production IT department, where as you would have thought that QNAP would be the kings of working to such constraints.

Once I had got to second line, the fix in itself was quite simple in the end, but its implementation leaves something to be desired. The system needed a BIOS update.

You might assume that when you are updating your “firmware“, this sort of things is being accommodated for in those updates. Apparently not! It seems as this new device may have been sitting in a warehouse for some time so was out of date, but it was immediately firmware serviced as soon as it was first booted. You would have expected that this sort of well know about, intermittent issue was being dealt with through the update delivery mechanism.

As soon as the BIOS of the TVS-1271U was updated to version QW10IR12 and rebooted, the problem was fixed. Why QNAP have not put information on this on their website knowledge base I do not know. This would seem more than sensible. QNAP does itself more reputational damage by trying to hide the issue and hope that only a few people see it. The reality is that they are likely causing stress and grief to end users and unnecessary RMA’s to their suppliers.

After checking power rails, our initial response was that it was a dead backplane and we were assuming it would have to be an RMA. Fortunately, it wasn’t and fortunately I went to QNAP before I went back to the supplier, but you do not always, especially with companies increasingly insisting that you deal with suppliers for RMA processes. A little public disclosure about this known issue would have just saved a lot of headaches (and disclosure makes you look good QNAP!).

The execution of the BIOS update itself was unfortunately quite cumbersome, requiring the support tech to back-up and then re-programme all of the NIC MAC addresses built into the motherboard. Something that could have easily have been sorted in a shall script and then transparently bundled into the firmware delivery, saving QNAP Taiwan more than an hours’ worth of time on this and some 8 emails.

It did however fix the problem and the Taiwan support team were pretty accomodating about doing it at 7am UK time.

Here for the benefit of the rest of the world, is the process that the tech went through to flash the BIOS. I have substituted real MAC addresses with fake ones below.

Note: I strongly recommend that you do not try this yourself and that you contact QNAP via their web chat support if you have a need to perform this procedure. If you try it, it is entirely at your own risk.