WorkFolders Folder shows Sync Error even though its contents are fully synchronised

WorkFolders allows you to perform policy based file HTTPS synchronisation between corporate servers and BYOD devices or teleworker devices. This article discusses a workaround to a problem where an anonymous sync error appears on a directory despite all of its contents synchronising successfully.

Outline of the Problem

Assume the following directory structure

C:\Users\CompanyUser\WorkFolders\Documents\WorkFolders Test

The following files/folders are present within WorkFolders Test:

WorkFolders Test\Problem Folder
WorkFolders Test\Problem Folder\File in Problem Folder.docx
WorkFolders Test\This file is OK.txt

Windows Explorer will display a Green circle with a tick for file/folder object that is synchronised and a Red circle with a cross for a faulted file/folder. After synchronising, the sync results will display as follows:

WorkFolders Test\Problem Folder [CROSS]
WorkFolders Test\Problem Folder\File in Problem Folder.docx [TICK]
WorkFolders Test\This file is OK.txt [TICK]

No errors are displayed in the Control Panel WorkFolders applet. There are no related errors in the clients WorkFolders Management/Operational Event Viewer logs. No relevant errors are present in the file servers SyncShare Operational/Reporting logs.

WorkFolders Error Screenshot - outer folder
The parent folder shows that its sub-folder has a sync error
WorkFolders Error Screenshot - inner folder
The contents of the error’d folder are however correctly synchronised.

Analysis

Although WorkFolders is indicating that the issue is being caused by the “Problem Folder” directory. The issue is being caused by the “File in Problem Folder.docx”.

The following symptoms will be true:

  1. Renaming “Problem Folder” will not fix the issue
  2. Altering the filename of “File in Problem Folder” will not fix the issue
  3. Changing the “File in Problem Folder.docx” file extension (e.g. to .txt) will not fix the issue
  4. Opening and saving the “File in Problem Folder.docx” will not solve the issue
  5. Moving “File in Problem Folder.docx” out of “Problem Folder” will clear the sync error, but the error will immediately migrate to the new location
  6. Rebooting the client computer will not help
  7. Restarting the server will not help
  8. The file does not have any connected temp files or lock files associated with it in the client file system

 

There is nothing wrong with the file itself. It is not corrupt, pay-loaded with a virus or violating any policy. It is my (unproven) belief that the record for the file in the WorkFolders synchronisation database is corrupt. Performing any of the above steps will not alter the record in the WorkFolders client database, thus the problem cannot be ameliorated.

 

Fixing the problem

One you have identified the problem file(s). You can use one of the methods below to correct the error.

Save the file as a completely new file

  1. Open the file in its associated editor (e.g. Microsoft Word for docx files)
  2. File > Save as…
  3. Save the file in its original location, but with a different file name (do not overwrite the original)
  4. Delete the original file
  5. Allow WorkFolders to re-sync
  6. Rename the new file as required

This approach is easy for an end-user to perform, but can be very time consuming if you are troubleshooting a large number of such issues. It requires you to know which file is causing the problem in the first place.

 

Compression

  1. Compress the file using Windows Compressed folders (Right click > Send to… > Compressed (zipped) file)
  2. Delete the original file
  3. Wait for the folder to re-sync and clear the error
  4. Extract the original file from the zip back into the desired location

This method will create a new record in the WorkFolders synchronisation database and the error will not reappear. You can use the technique to fix an entire folder structure without having to first identify the problem file. It is also easy for an end-user to perform.

 

Move the file

  1. Move the file outside of the WorkFolders monitored file system. For example, move the file to C:\ or into the Recycle Bin
  2. Allow the original folder to re-sync and clear the error
  3. Return the original file to its original location and allow it to re-sync

Again, this method will create a new synchronisation record. In a managed environment this may be harder for an end-user to perform due to permissions. It is however easier for an administrator to perform as you can cut/paste the entire file structure out and then back into the WorkFolders sync root.

If you use this method, remember to move it to a location within the same drive letter. If you do, the move will preserve permissions, file dates and will not physically copy the underlying data to the new location (just update the MFT).

“RPC server unavailable. Unable to establish communication between and ” when connecting to Hyper-V 2008, 2008 R2, 2012, 2012 R2 from Hyper-V Manager version 1709

System Requirements:

  • Windows 10 1709
  • Windows Server 2016
  • Hyper-V Management Console
  • RSAT 2016/1709 for Windows 10 version 1709

The Problem:

After upgrading to Windows 10 version 1709 and installing the updated Windows Server 2016 (version 2016 or version 1709) RSAT tools for Windows 10 1709. On attempting to connect to a down-level Windows Server 2012 R2, 2012, 2008 R2, 2008 Hyper-V Server via the Hyper-V Manager MMC snap-in. You receive the error even though no configuration changes have been made on the Hyper-V hosts:

"RPC server unavailable. Unable to establish communication between <management host> and <Hyper-V host>"

At this point you are unable to manage down-level version of Hyper-V from Windows 10. This issue does not impact the management of remote Windows Server 2016 or Windows Server 1709 Hyper-V instances.

View: Remote Server Administration Tools for Windows 10 (RSAT)

The Fix

This appears to be related to a change in the default firewall behaviour on Windows 10 1709 installs. to fix the problem. On the client system, where you have installed RSAT to remote manage the hypervisor (i.e. not on the hypervisor itself):

  1. Open ‘Administrative Tools’ in the Windows Control Panel
  2. Open ‘Windows Defender Firewall with Advanced Security’
  3. Select ‘Inbound Rules’ from the left hand side
  4. Scroll down until you get to ‘Windows Management Instrumentation (ASync-In)’
  5. Enable the rule for domain/private/public networks as required
    Note: By default the Windows firewall MMC will only display WMI rules for domain and private networks. If you are not running against a domain and Windows has not been explicitly told that you are on a private network, Windows will assume that you are on a public network. Check in network settings in the settings app to ensure that you are not running on a public network, or if you are edit the firewall rule to include public networks. In general, it is a bad idea to open WMI up to traffic on public networks.
  6. Restart Hyper-V Manager

You should now find that you can connect to down-level versions of Hyper-V from Windows 10 1709.

“Some migration network settings cannot be modified or removed because these migration network settings are in use by a cluster” in Hyper-V manager, Hyper-V Settings after a node is non-cleanly evicted from a failed cluster

System Requirements:

  • Windows Server 2008, 2008 R2, 2012, 2012 R2, 2016
  • Hyper-V

The Problem:

After a total cluster failure occurs, or a node is removed from a cluster before it has been cleanly removed. When attempting to locally manage the host as a stand-alone Hyper-V server, you are unable to edit the live migration settings in Hyper-V Settings via Hyper-V Manager with the error

“Some migration network settings cannot be modified or removed because these migration network settings are in use by a cluster”

The IP/Subnet addresses shown on the form will be greyed out and you will be unable to edit the live migration network settings.

Hyper-V Settings: Unable to edit Live Migration Networks

More Info

While you should ensure that you have performed a cluster clean-up on the host

Server 2012 +:
Clear-ClusterNode -Force -CleanupA

Server 2008/R2:
cluster node <hostname> /forcecleanup

This will not solve the live migration settings issue highlighted above.

The Fix

  1. Close Hyper-V Manager on the management workstation
  2. On the hypervisor experiencing these symptoms (not necessarily the management console). Navigate to HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Virtualization\Migration\NetworkSettings
  3. Underneath this key there will be a key for each entry shown on the Live Migration Settings screen. The keys will be named Network# e.g. Network0, Network1, Network2 and so on.
  4. Go through each of these Network# keys and locate the ‘Tags’ REG_MULTI_SZ. This will have a value of “Microsoft:ClusterManaged”
    Registry Screenshot: Error
  5. Change the data value to “Microsoft:UserManaged”
    Registry Settings: Fixed
  6. Complete the process for each Network# sub-key
  7. Restart Hyper-V Manager

You will now be able to add, edit and delete the live migration settings.

QNAP TVS-1271U Top Row of Drives (drives 9-12) do not start on boot/reboot of the NAS appliance

System Requirements:

  • QNAP TVS-1271U
  • QTS 4.3

The Problem:

A brand new, out of the box 12 drive 2u NAS appliance, with the “firmware” up to date gets thrown into production. During its first maintenance cycle, the firmware is updated again and it is rebooted.

On completion of the reboot, the array is offline. Visual inspection of the appliance reveals that the top row of drives in the array, drives 9, 10, 11 and 12 are offline. They just are not powered. Drives 1-8 are online and all green.

Rebooting the array shows the diagnostic LED’s on all 12 drives flash red, but the 4 drives in question then go into a powered down state and the boot completes with an inconsistent volume layout.

  1. Hot swapping the drives did not make a difference
  2. Rebooting the array did not make a difference.
  3. Testing all of the top rows worth of drives externally showed no errors in any drives
  4. Performing a couple of hard power downs and cold booting did solve the problem and allowed the array to start normally.

The same thing happened during the next test and diagnostics window.

The Fix

Disgruntled customer note: I want to add at this point that trying to phone QNAP UK is an exercise in futility. I sat in their queue system for more than half an hour listening to the same loop of music without getting anywhere. By the time that I gave up, I’d managed to get the thing to start through cold boot cycles and done quick SMART tests on all of the drives. When it came to this occurring the second time, I used the online chat feature with their people in Taiwan, who were more responsive and most helpful, but would not entertain speaking on the phone come what may.

My bigger issue is that QNAP support seems to have little sympathy for the needs of a production IT department, where as you would have thought that QNAP would be the kings of working to such constraints.

 

Once I had got to second line, the fix in itself was quite simple in the end, but its implementation leaves something to be desired. The system needed a BIOS update.

You might assume that when you are updating your “firmware“, this sort of things is being accommodated for in those updates. Apparently not! It seems as this new device may have been sitting in a warehouse for some time so was out of date, but it was immediately firmware serviced as soon as it was first booted. You would have expected that this sort of well know about, intermittent issue was being dealt with through the update delivery mechanism.

As soon as the BIOS of the TVS-1271U was updated to version QW10IR12 and rebooted, the problem was fixed. Why QNAP have not put information on this on their website knowledge base I do not know. This would seem more than sensible. QNAP does itself more reputational damage by trying to hide the issue and hope that only a few people see it. The reality is that they are likely causing stress and grief to end users and unnecessary RMA’s to their suppliers.

After checking power rails, our initial response was that it was a dead backplane and we were assuming it would have to be an RMA. Fortunately, it wasn’t and fortunately I went to QNAP before I went back to the supplier, but you do not always, especially with companies increasingly insisting that you deal with suppliers for RMA processes. A little public disclosure about this known issue would have just saved a lot of headaches (and disclosure makes you look good QNAP!).

The execution of the BIOS update itself was unfortunately quite cumbersome, requiring the support tech to back-up and then re-programme all of the NIC MAC addresses built into the motherboard. Something that could have easily have been sorted in a shall script and then transparently bundled into the firmware delivery, saving QNAP Taiwan more than an hours’ worth of time on this and some 8 emails.

It did however fix the problem and the Taiwan support team were pretty accomodating about doing it at 7am UK time.

Here for the benefit of the rest of the world, is the process that the tech went through to flash the BIOS. I have substituted real MAC addresses with fake ones below.

Note: I strongly recommend that you do not try this yourself and that you contact QNAP via their web chat support if you have a need to perform this procedure. If you try it, it is entirely at your own risk.

login as: admin
admin@192.168.1.1's password:
[~] # md_checker Welcome to MD superblock checker (v1.4) - have a nice day~ Scanning system... HAL firmware detected!
Scanning Enclosure 0...
RAID metadata found!
UUID: 3e5d7c85:95d82d3e:42647860:c0aaec32
Level: raid6
Devices: 12
Name: md2
Chunk Size: 512K
md Version: 1.0
Creation Time: Apr 19 11:57:42 2017
Status: ONLINE (md2) [UUUUUUUUUUUU]
===============================================================================
Disk | Device | # | Status | Last Update Time | Events | Array State
===============================================================================
1 /dev/sdc3 0 Active Jun 16 07:15:08 2017 156 AAAAAAAAAAAA
2 /dev/sdd3 1 Active Jun 16 07:15:08 2017 156 AAAAAAAAAAAA
3 /dev/sde3 2 Active Jun 16 07:15:08 2017 155 AAAAAAAAAAAA
4 /dev/sdf3 3 Active Jun 16 07:15:08 2017 155 AAAAAAAAAAAA
5 /dev/sdk3 4 Active Jun 16 07:15:08 2017 155 AAAAAAAAAAAA
6 /dev/sdl3 5 Active Jun 16 07:15:08 2017 155 AAAAAAAAAAAA
7 /dev/sdm3 6 Active Jun 16 07:15:08 2017 155 AAAAAAAAAAAA
8 /dev/sdn3 7 Active Jun 16 07:15:08 2017 155 AAAAAAAAAAAA
9 /dev/sdg3 8 Active Jun 16 07:15:08 2017 155 AAAAAAAAAAAA
10 /dev/sdh3 9 Active Jun 16 07:15:08 2017 155 AAAAAAAAAAAA
11 /dev/sdi3 10 Active Jun 16 07:15:08 2017 155 AAAAAAAAAAAA
12 /dev/sdj3 11 Active Jun 16 07:15:08 2017 155 AAAAAAAAAAAA
=============================================================================== [~] # cd /share/CACHEDEV2_DATA/Public/
[/share/CACHEDEV2_DATA/Public] # ls
@Recycle/ messages
[/share/CACHEDEV2_DATA/Public] # wget http://download.qnap.com/Storage/tsd/bios/TVS-1271U_QW10IR12.zip
--2017-06-16 07:16:51-- http://download.qnap.com/Storage/tsd/bios/TVS-1271U_QW10IR12.zip
Resolving download.qnap.com (download.qnap.com)... 2.17.149.135, 2a02:26f0:ec:38c::1b52, 2a02:26f0:ec:398::1b52
Connecting to download.qnap.com (download.qnap.com)|2.17.149.135|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5931607 (5.7M) [application/zip]
Saving to: ‘TVS-1271U_QW10IR12.zip’ TVS-1271U_QW10IR12.zip 100%[=====================================================================>] 5.66M 5.50MB/s in 1.0s 2017-06-16 07:16:52 (5.50 MB/s) - ‘TVS-1271U_QW10IR12.zip’ saved [5931607/5931607] [/share/CACHEDEV2_DATA/Public] # chmod +x TVS-1271U_QW10IR12.zip
[/share/CACHEDEV2_DATA/Public] # unzip TVS-1271U_QW10IR12.zip
Archive: TVS-1271U_QW10IR12.zip
creating: BIOS_QW10IR12/
inflating: BIOS_QW10IR12/flashrom
inflating: BIOS_QW10IR12/QW10IR12.bin
[/share/CACHEDEV2_DATA/Public] # ls
@Recycle/ BIOS_QW10IR12/ TVS-1271U_QW10IR12.zip* messages
[/share/CACHEDEV2_DATA/Public] # dmidecode -t bios | grep version
[/share/CACHEDEV2_DATA/Public] # cd
[~] # cd /
[/] # dmidecode -t bios | grep version
[/] # cd -
/root
[~] # cd /share/CACHEDEV2_DATA/Public/
[/share/CACHEDEV2_DATA/Public] # head /etc/config/uLinux.conf
[System]
Model = TS-X71U
Internal Model = TS-X71
Server comment =
Version = 4.3.3
Build Number = 20170606
Number = 0224
Time Zone = Europe/London
Enable Daylight Saving Time = TRUE
Workgroup = QNAP
[/share/CACHEDEV2_DATA/Public] # cat /etc/model.conf | grep INTERNAL_NET_PORT_NUM
INTERNAL_NET_PORT_NUM = 4
[/share/CACHEDEV2_DATA/Public] # hal_app --se_sys_get_mac obj_index=0
35:35:35:35:36:31
[/share/CACHEDEV2_DATA/Public] # hal_app --se_sys_get_mac obj_index=1
35:35:35:35:36:32
[/share/CACHEDEV2_DATA/Public] # hal_app --se_sys_get_mac obj_index=2
35:35:35:35:36:33
[/share/CACHEDEV2_DATA/Public] # hal_app --se_sys_get_mac obj_index=3
35:35:35:35:36:34
[/share/CACHEDEV2_DATA/Public] # cd
[~] # cd /share/Public
[/share/Public] # ls
@Recycle/ BIOS_QW10IR12/ TVS-1271U_QW10IR12.zip* messages
[/share/Public] # cd BIOS_QW10IR12/
[/share/Public/BIOS_QW10IR12] # ls
QW10IR12.bin flashrom
[/share/Public/BIOS_QW10IR12] # ls
QW10IR12.bin flashrom
[/share/Public/BIOS_QW10IR12] # chmod +x *
[/share/Public/BIOS_QW10IR12] # ls
QW10IR12.bin* flashrom*
[/share/Public/BIOS_QW10IR12] # ./flashrom -c MX25L128050 --programmer internal -w QW10IR12.bin
flashrom v0.9.8-unknown on Linux 4.2.8 (x86_64)
flashrom is free software, get the source code at http://www.flashrom.org Error: Unknown chip 'MX25L128050' specified.
Run flashrom -L to view the hardware supported in this flashrom version.
[/share/Public/BIOS_QW10IR12] # ./flashrom -c MX25L12805D --programmer internal -w QW10IR12.bin
flashrom v0.9.8-unknown on Linux 4.2.8 (x86_64)
flashrom is free software, get the source code at http://www.flashrom.org Calibrating delay loop... OK.
Found chipset "Intel C226".
This chipset is marked as untested. If you are using an up-to-date version
of flashrom *and* were (not) able to successfully update your firmware with it,
then please email a report to flashrom@flashrom.org including a verbose (-V) log.
Thank you!
Enabling flash write... Warning: SPI Configuration Lockdown activated.
OK.
Found Macronix flash chip "MX25L12805D" (16384 kB, SPI) mapped at physical address 0xff000000.
Reading old flash chip contents... done.
Erasing and writing flash chip... Erase/write done.
Verifying flash... VERIFIED.
[/share/Public/BIOS_QW10IR12] # hal_app --se_sys_set_mac obj_index=0,value=35:35:35:35:36:81
eth port = 0, Set MAC address = 35:35:35:35:36:31,ret = 0
[/share/Public/BIOS_QW10IR12] # hal_app --se_sys_set_mac obj_index=1,value=35:35:35:35:36:82
eth port = 1, Set MAC address = 35:35:35:35:36:32,ret = 0
[/share/Public/BIOS_QW10IR12] # hal_app --se_sys_set_mac obj_index=2,value=35:35:35:35:36:83
eth port = 2, Set MAC address = 35:35:35:35:36:33,ret = 0
[/share/Public/BIOS_QW10IR12] # hal_app --se_sys_set_mac obj_index=3,value=35:35:35:35:36:84
eth port = 3, Set MAC address = 35:35:35:35:36:34,ret = 0
[/share/Public/BIOS_QW10IR12] # reboot