is article is an update on the Hyper-V Virtualised DVT-T2 Streaming TV / IPTV server and is part 5 in my series on the design and evolution of my implementation.
If you have not see the previous parts, then please use the links below to navigate to the historic articles.
Part 1: Creating a Virtual TV Streaming Server
Part 2: Redesigning the Hardware for the Virtual TV Streaming Server
Part 3: Discrete Device Assignment & PCI Express Bifurcation in the Hyper-V TV Server
Part 4: Replacing the TV Server USB Controller with a Single Channel PCIe Device
Part 5: Virtual TV Server: Changes & Issues after 4 years
Change is inevitable
This April will represent the 4th anniversary of the Hyper-V Virtualised TV server. The server hardware has run exemplarily well during this time. The server motherboard, with its remote IPMI management has been a breeze to work with; although SuperMicro’s firmware update processes continue to frustrate.
SuperMicro’s BIOS/UEFI updates always perform a factory reset on the BIOS, which resets the motherboard to legacy/CSM mode and renders the system non-bootable. Requiring one to trudge through a plethora of BIOS sub-screens to set everything back. This is quite frustrating when you just want things to go smoothly. That said, I haven’t had any issues with the BIOS update process. The same cannot be said for the IPMI firmware, which failed completely at one point and required a convoluted recovery to get it working again.
I know that SuperMicro have always warned end-users off of flashing firmware – having been a user since the 1990’s, their stance has never changed. Yet, if Dell, HP and more can do it – and maintain settings – then surely at this juncture, it is SuperMicro who are doing something wrong?
System Stability
The performance of the TV Server’s VM has been, absolutely rock solid. In 4 years it has never blue screened. It has survived one or two failures of the electricity grid that were pushed over the edge by (then unknown) dead UPS batteries. Each time, the VM has come back up without complaint.
The Windows Server 2019 Hyper-V Hypervisor has been almost as reliable. It itself has not blue screened, although it has invisibly snarled up the Discrete Device Assignment (DDA) and then refused to reboot on two occasions.
On each occasion, the VM reported the presence of the USB controller as well as each of the TV Tuners. The VM was however unable to tune to any channel using that hardware. Multiple reboots of the VM did not change this. Upon rebooting the Hypervisor itself, the server was unable to correctly POST.
On the first occasion (Spring 2020) a CMOS reset was required. On the second (January 2023) turning the system off and disconnecting the power for long enough to drain all of its capacitors was sufficient to get it working. With that said, after resetting the BIOS following the first occurrence, DDA while still registered, was ineffective within the VM. It was necessary to dismount and then re-attach the device pass-through to return the system to a fully working state. No such action was required the second time.
With those two exceptions, and a couple of instances of Windows Updates breaking Hyper-V Server 2019. There have been no stability problems.
Software Stability
The stability of the application software and drivers running within the VM have not faired quite as well.
A mistake of three tuners
By far the biggest mistake that I made with the system was in purchasing three identical tuners. I was aware that it was a risk when I set it up, however initial tests didn’t show there to be any problems. Yet it only took a matter of weeks for the mistake to become apparent.
Sometimes as a result of a reboot and sometimes just randomly out of nowhere (presumably related to power management) TV Mosaic would refuse to playback live TV. A trip to the web UI would reveal that one or two, but never all three of the existing Tuners (rebadged ITE IT9300’s) had been changed to “Unknown” with the one or two missing tuner(s) now displaying as a new, unused device.
In this state, TV Mosaic could not playback TV and neither would it record. Removing the unknown devices and re-attaching the ‘new’ ones solved the problem. Ultimately, for the last few years it has been necessary to visit the web UI after every reboot and to try to remember to check on it several times a week so I could catch the error before it became a problem.
Analysis of the problem
The temptation would be to blame TV Mosaic, however on switching to Plex (more on this later) the same problem occurs. Plex also drops the tuner into its ‘available devices’ pool and refuses to playback or record.
The mistake could ultimately be blamed on DVBLogic, or perhaps more specifically its Tuner hardware manufacturing partner ITE.
The detaching Tuner problem is caused by Windows’ USB enumeration. After a reboot (say) Windows is aware that there are three BDA Tuner Devices connected to its USB bus. It is also aware that all three tuners use the same driver. Windows accordingly assigns each USB device a device ID. These are consecutively numbers representing the order that they appear on the USB bus – originally: 1, 2 and 3.
On a reboot, the may come back up in the same order – they often will. However sometimes they will not and 3 will become 2 and 2, 3. Except the Windows driver database will have a device 2 and 3 already registered. So 3 will become device 4 and 2 will become device 5. To TV Mosaic and Plex, these are seen as new hardware devices. Deleting the phantom tuner will allow it to work again – for a time. Until 4 drops out and returns to 3. At which point you have the same issue – a missing Tuner preventing TV playback.
The ultimate reason for the problem (and the error made by DVBLogic/their hardware partner) is that the TV Tuner firmware has no unique device identifier burned into it. Windows would ordinarily look a the device serial number to ascertain uniqueness. This allows it to consistently re-map the physical device to its virtual device ID – and in turn for TV Mosaic and Plex to do likewise. The TV Butler Tuners have no such serial number (or unique ID written to ROM). As a result, Windows is forced to guess which device is which and gets it wrong a fair percentage of the time.
Either through general fatigue, or actual increased regularity, in recent months the occurrences of phantom Tuner faults have felt like they becoming more frequent. Consequently, I have replaced two of the three tuners. One with a PCTV tripleStick 292e and the other with a Hauppauge WinTV-soloHD. Each having its own unique Plug and Play ID. It is my hope that this will prevent the race condition and finally put an end to the problem.
TV Mosaic Reliability
Most of my issues with TV Mosaic have been related to the USB enumeration problem; but not all.
My main gripe has been with the EPG. I feel that DVBLogic intentionally limited the EPG to a maximum of 1 week in order to sell their own 2 week subscription service. Unfortunately as time has gone on, TV Mosaic has been able to acquire less and less OTA EPG data. First a week became 5 days, then 3, 2 to the point that now you cannot go much past midnight of the day you are currently in before channels start to show “No data available”.
When the Tuners are working, the automatic re-scan does consistently keep data available until midnight, however this is utterly useless if you want to scan ahead and set a recording schedule for something in the further future.
Even a complete reset and clean re-install failed to fix this problem.
On that occasion, the reset was compelled by the so called ‘TV Recordings database’ becoming corrupt. This wipes out the meta data for all recordings as well as tuned channel data. For whatever reason, DVBLogic chose not to write recording metadata into the MPEG recording file. This to me was a huge mistake and a missed opportunity. Without meta data from the database orphaned recordings are impossible to work with. If the data had been duplicated in both the database and on the file (or even in XML/JSON), then it would have easily been possible to re-import recordings after a failure.
I manually reconstructed most of the database using a MySQLite database editor and a backup file of the database after a full clean re-install of TV Mosaic, so at least I got back up and running.
Interestingly, having replaced the three TV Butlers with three different TV Tuners, much to my surprise, the EPG immediately went back to displaying 7 days worth of programming. This indicates that the issue was being caused by the TV Butler itself, or the presence of more than one of them – an additional, unseen bug in TV Mosaic!
The final issue is a rather sad one. That with the dawn of the COVID-19 crisis, after much silence and no product development for many years, DVBLogic announced that they were closing down the business. This decision ended my hopes of ever seeing a resolution to any of my issues. So while I have continued to tease the TV Server to stay working for all these years, its failings have for me become beyond tiresome.
Plex
Plex – and a January sales discounted Plex Pass Lifetime Membership – may, I hope, provide the answer.
Plex premium works with the DVBLogic TV Butler, as it does with the PCTV tripleStick 292e and the Hauppauge WinTV-soloHD. As mentioned above, the USB enumeration problem remains when running three identical TV Butler USB Tuners, however – so far – there have been no such issues when using three different USB tuner devices.
Tuning Nightmares
Plex’s configuration options are not as comprehensive as TV Mosaic’s and its tuning system leaves a lot to be desired.
You might expect that having three tuners listening to the same transmitter, you would have to map the EPG and channel data once and it be applied to each tuner. Sadly not!
Instead you have to manually map the EPG and channel data three times, once for each tuner. To achieve this, you have to select/deselect which channels you do or do not want, three times. Finally you have to link the channel to its EPG data and number. It does have an automatic mapping feature, however the auto mapping is fairly poor. This is made worse by the Plex UK transmitter list only having a few of the UK’s TV transmitters in it – mine not included. Consequently, performing an all frequencies scan takes more than 25 minutes!
Plex’s EPG is also missing a lot of (minor) UK TV and Radio channels. This is frustrating as Plex actively prevents the use of a channel unless there is a codified EPG entry. Rendering these unwatchable!
To add insult to injury, even though the three tuners may scan the same channels. The UI to map channel to EPG data doesn’t display them in the same order.
As you can see above, all three interfaces display channels in a different order, with different default sections depsite the fact that they are all connected to the same aerial, on the same transmitter.
NB: The weak signal on tuner 2 is due to a damaged cable. A replacement has been ordered.
There is also no way to re-sort the channels to display by name or number. This makes the mapping process a fiddly logic problem, as well as time consuming chore. It is very difficult to synchronise all three tuners to use precisely the same data.
Worse still, the UI is buggy requiring you to map channels a few at a time and save otherwise it may randomly refuse to save. No error will be presented and there is no indication as to why the save button does nothing what so ever. Naturally having just mapped 130+ channels, this will result in you losing all of your arduous work.
Once you finally have a channel mapping in place. It then takes Plex a further half an hour to “build the guide”, something that has to be repeated every single time you edit any one of the EPG mappings on any of the tuners.
… and be warned: if Plex drops a tuner (for example due to the USB enumeration problem), you have to do all of the above all over again!
As can be seen below, once all three tuners are scanned, we receive a very different set of mapped channels on each of the three tuners (highlighted in red).
Manually changing any of these to try and synchronise them triggers a new 30 minute guide rebuild, making the activity even more arduous.
With that said, once setup, the EPG experience on Plex is far better than TV Mosaic has been. Data is available for an extended period – I can go forward at least a week and a half before spotting a “To be announced” on any of the main channels (with many channels having data for longer). Compare this to TV Mosaic’s maximum of 24 hours.
The downside is that navigating forward in time is a lot slower and the jump menu only allows you to hop up to 7 days ahead. As there is no keyboard navigation, you have an awful lot of manual x/y mouse-work to do to simply ‘look around’. If you know what you are looking for on the future EPG however, Plex shines! Its integrated search is exceptionally punchy and the category/programme type views are nice additions.
Other features of Plex that are welcome additions over TV Mosaic are:
- Commercials detection and their removal during transcoding.
- If Plex misses a recording, it stores it in a queue to automatically re-record should it appear on the EPG again.
- Automatic all episodes recordings of a series (TV Mosaic has this too, but it is not as advanced).
- Programme filters – genre, year, sport, movie, TV show, series and episode that integrate well into the wider Plex ecosystem.
- You can now assign permissions for managed users to be able to record TV, not just the main account holder.
Some functional details that I dislike are:
- While vertical scrolling using the mouse wheel/track pad works to move between channels on the EPG, horizontal scrolling does not allow you to move forward/backwards in time.
- It is not possible to persist the visibility of channel numbers (you have to hover the mouse over the channel logo in the EPG to see the number).
- There is no way to make the EPG text size bigger/bolder – this is a very real problem for accessibility.
- You cannot create a favourite channels list or pin channels to filter the EPG
- When you start TV playback via Plex, it will take a moment to load, play for a second, pause and then resume after a few seconds. Reducing this to a single – if longer – buffer interval would be preferred.
- Programme/channel groups are not being maintained properly. For example, the news category doesn’t have news channels in it which have been running since 2021.
Another nice surprise has been that Plex and TV Mosaic can co-exist with the same tuners on the same virtual machine, at least in a light-use, limited capacity. All three tuners can be tuned and configured on both platforms, but if the other platform is using the tuner the first will fail to start playback or record. This has made testing and transitioning easier. Previously, I had seperate VM’s for TV Mosaic and Plex. As part of this migration, I copied some 500,000 Plex meta data files over to the TV Mosaic VM and was pleased to say that everything “just worked”.
Wrapping Up
The Hyper-V Virtual TV Server has been working for the last few years, but it has not been a smooth process. Had I been able to use PCIe TV tuners with Hyper-V, then I would have been spared these frustrations. Sadly, until newer generation PCIe cards are released, this remains impractical.
With the exception of replacing a damaged MCX to IEC RF cable, this cpompletes all of the planned work on the TV Server here at the beginning of 2023. It will continue to run both TV Mosaic and Plex for the forseeable future, with a desire to switch fully to Plex if testing is successful.
The VM continues to get regular use and now that Plex is consolidated on to the same VM, its utility will continue to grow. Fingers crossed for the rest of 2023!