SQL Server combining LIKE with IN (SELECT LIKE IN query)

System Requirements

  • SQL Server

 

The Problem

This was one of those “it should be obvious” moments.

Dating back to 2004 in its current iteration, the hpcfactor.com membership database has been blighted by SPAM. We are no different to any web community in this regard. A can be a particular problem as many accounts that we may miss are registered and (if they survive) become a valuable commodity to sell as trojan accounts for use in mischief at a later date.

The moderation team at HPC:Factor does a great job of ensuring that these ‘pay per post’ (usually from Indian IP’s) urchins are wasting their time, but it does not stop them from trying. At one point we were having to content with several hundred a day, and had to shutdown registration completely. A custom layer that I wrote to sit on-top of the BBS registration system has done a good job to convert that SPAM torrent into a trickle of a “couple of times a month”.

Over the years, I have added more and more banned email domains to a ban list that is maintained in the user registration database. It occurred to me that as that list has increased, the historic pool of user account has never been re-parsed against the updated banned domains list.

Sounds simple right? Fire up SQL Server Management Studio and create a SQL SELECT LIKE IN query… is it that simple?

 

The Solution

Yes actually it is.

The obvious logic that you want to follow is to take this syntax

SELECT id, username, emailaddress
FROM members
WHERE emailaddress IN (
   SELECT emailaddress FROM bannedemails
)

… and merge it with a LIKE statement to create a SELECT LIKE IN query, something like

SELECT id, username, emailaddress
FROM members
WHERE emailaddress LIKE IN (
   SELECT '%' + emailaddress FROM bannedemails
)

Unfortunately however this syntax is completely illegal.

The answer is to perform the LIKE/IN as part of an INNER JOIN

SELECT members.id, members.username, members.emailaddress
FROM members
INNER JOIN bannedemails ON members.emailaddress LIKE '%' + bannedemails.emailaddress

 

Performance

This is a not a “real time” friendly SQL expression. The inefficiency of using LIKE evaluation becomes very apparent as the record set scales. The HPC:Factor database had 2,271 banned email domains and 30,336 user. Giving a total record iteration and evaluation count of 68,893,056 rows.

The total query execution time for this on was 2 minutes 13 seconds (ouch). Having said that, on reviewing why it took that long, I realised that at some point in time I was being extremely stingy with the SQL Server VM. It only had access to 2 CPU cores and 12 GB of RAM. The hypervisor server itself has access to a 10 core hyper-threaded Xeon.

 

The Result

I have a lot of respect and sympathy for people who use VPNs and temporary email addresses and I genuinely understand why they feel the need (or in some cases must, depending on what part of the world you are in).

I saying that, I also have to deal with the issue from the other side of the equation. If the website is having a bad day, it’s usually because of something coming from one or both of these camps. As ever in life, a minority will insist on spoiling a good thing for everyone else.

After running the query, we had only missed 8 user accounts – credit to the moderation team. The oldest account was 26/09/2006 and my favourite that we probably should not have missed was “mrputin” from January 2013.

Windows NT 4.0 on Hyper-V 2016

System Requirements:

  • Windows Server 2016, Hyper-V Server 2016
  • Windows 10
  • Windows NT 4.0 Advanced Server, Server, Terminal Server Edition, Workstation

The Problem:

For reasons that defy any sane logic, I decided that I needed to install NT 4.0.

It’s 2018 and over the last few years I have been slowly clearing out all of my old IT hardware, to the point now that I no longer have any legacy motherboards or systems in the house or office. So when I recently needed to fire up Windows NT 4.0 once again – for reasons that defy logic – you would assume that Virtualisation was the easy win.

Sadly – and especially with Hyper-V – this is not the case. Microsoft’s virtualisation solution is (and always has been) designed around its currently supported operating systems, with a little Linux added in to the mix in more recent times. Down-level operating systems are not supported and by default, are not going to work. This is especially true of what in effect is Windows 1996, the workhorse wonder that was Windows NT 4.0.

I am sure that the non-masochists of you will just use something like VMWare or Virtual Box to do thy bidding and carry on with their day… but I digress….

Note: This process is will be very similar for Windows NT 3.5 and NT 3.51 as it will be for Windows 2000 – however Windows 2000 does not have the 8GB disk/2GB partition initial size limitation.

The Fix

The following procedure will get you up and running with a working NT 4.0 install under Hyper-V 2016. I am assuming that you know your way around Hyper-V and this article is intended as a results based guide, not a step-by-step ‘click here, go here’ guide.

Create the VM

Use the following configuration when creating your VM:

  1. Create a generation 1 Virtual Machine. In our case this will be “NT 4.0 Server”
  2. Set the RAM to 512 MB (or lower)
  3. You can set it to 1 or 2 CPU cores as required
  4. Do not connect to a network. Remove the default network adapter completely. Add a new Legacy Network Adapter
  5. Create a new virtual hard drive. The drive can be fixed or dynamically expanding, however set the maximum disk size to 6GB or lower. Ensure that both the VHDX and the virtual DVD drive are connected to the IDE bus, not the SCSI bus
  6. Attach your NT 4.0 install CD/ISO to the virtual DVD drive
  7. [If applicable] attach the NT 4.0 virtual floppy boot disk to the virtual floppy drive
  8. Set the required boot order (Floppy or CD ahead of HDD)

Pre-configure Hyper-V

By default, Hyper-V will attempt to run the VM under its default modern architectures mode, compatible with Windows Vista+ systems. The 1996 Windows NT 4.0 code-base is not compatible with modern platforms or CPU instruction sets and if you attempt to boot to the NT text mode installer without addressing this issue, NT 4 will blue screen while attempting to bootstrap the installer.

To fix this, you need to enable the legacy CPU compatibility. This used to be a GUI option in Hyper-V 1.0 under Windows Server 2008, but the option was removed in later releases. Despite being removed from the GUI, the option does still exist in the Hyper-V core and can be re-enabled for the VM using PowerShell.

To enable compatibility mode, open an elevated PowerShell sessionon the hypervisor and enter the following command:

Set-VMProcessor "NT 4.0 Server" -CompatibilityForOlderOperatingSystemsEnabled $true
Get-VMProcessor "NT 4.0 Server"

Text Mode Setup

Boot your Virtual Machine from the floppy/CD and enter text mode and follow through the setup process.

  1. You do NOT need to add any additional mass storage device drivers (this includes the NT 4.0 SP4 ATAPI update, which if you attempt to add the updated driver, the installer will ignore).
  2. When prompted to choose the keyboard layout, language and confirm the computer type. Change the computer type to “Standard PC” for a single core VM or “MPS Multiprocessor PC” if you require access to two cores. Enter your preferred keyboard settings as required.
  3. In the drive partitioning section, create an NTFS partition of less than 2048MB. I would suggest 1024MB for simplicity. Do not attempt to create a larger partition. The reason for this is that NT 4.0 will initially format the VHDX as FAT16, which has a maximum partition size of 2GB. During the later installer process and before entering GUI mode setup, NTFS conversion will be run over the FAT partition, converting it into an NTFS 1.2 file system. You will patch it to NTFS version 3.0 after installing NT 4.0 SP4 or later.

If you receive an installation failure because setup cannot write to the Windows folder or a setup error stating that permissions could not be created, this is most likely caused by you creating the initial VHDX larger than 8GB.

GUI Mode Setup

There are no special requirements or steps to perform during GUI Mode Setup.

Auto detection of the Network card will work with the Hyper-V Legacy Network Adapter. Ensure that you properly configure TCP/IP and remove IPX/SPX from the protocol list (unless you specifically need it).

Post Install

  1. Install SP6a (SP4 at a minimum).
  2. Turn off the machine.
  3. Increase the RAM from 512MB if required.
  4. In Hyper-V Manager/PowerShell edit the virtual disk and set the maximum size to your required size (e.g. the default 127GB).
  5. In Windows Server 2016, locate the VHDX and mount the disk. Using PowerShell or Disk Manager, expand the partition to fill the entire size of the disk.
  6. [1/2] If you want Windows NT 4.0 to turn off automatically when you click the shutdown button (instead of telling you it is now safe to turn off your computer):
    1. Use 7-zip (or similar) and extract the hal.dll.softex file from the SP6a installer, rename it HAL.dll and copy it into C:\WinNT\System32\
      Note: If you are using a multi-processor VM, rename the halmps.dll.softex to HAL.dll and do the same.
  7. Unmount the VHDX.
  8. Reboot the Virtual Machine.
  9. [2/2] If you want Windows NT 4.0 to turn off automatically when you click the shutdown button (instead of telling you it is now safe to turn off your computer):
    1. Add the following to the registry:
      REGEDIT4[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon\]
      "PowerdownAfterShutdown"="1"
  10. Install Internet Explorer 6.0 SP1, patch and update the Windows install, install application and configure to your needs.

Things the inexperienced user may not know

If you are playing with NT 4.0 for the first time, then there may be some things that you are not aware of. Here are a list of a few points that are worth noting should you be the Windows equivalent of a millennial (pun intended).

  1. The total install size (less the page file), after patching and cleaning up uninstall data was ~320MB. They don’t make them like that any more!
  2. NT 4.0 does not support plug and play. If you want to add hardware, you have to do it manually via a plethora of different places in the control panel – there is no device manager!
  3. NT 4.0 is extremely insecure by default. Know that it has no built-in firewall and that the base system policy and security configuration is insecure by default (even file system permissions are a free for all). You should keep this in mind when attempting to do anything at all with NT. If it really needs to be on the network you should at a minimum harden system policy and add a firewall (ZoneAlarm Free was the go to back in the day).
  4. There are no display drivers for Hyper-V. This means that there is no mouse integration and as such you will be unable to install NT 4.0 over a Remote Desktop session. It also means that you will be stuck at a max resolution of 800×600 in 16 colour using official means.
    1. Unofficially, you can make use of the great work of the VBEMP NT project to increase the resolution and get NT 4.0 running at modern resolutions and up to ‘True Colour’ (24-bit). This does not offer any cursor integration between the VM and the Hyper-V Manager, preventing mouse use over a Remote Desktop connection and requiring the ctrl +  alt + left cursor to escape the Hyper-V Connection window.
      View: VBEMP NT (tested with NT 4.0 stable version 3.0)
  5. There are no sound drivers for NT 4.0 in Hyper-V (unlike there used to be in Virtual PC) as Hyper-V does not emulate any sound adapters.
  6. The disk performance is fairly poor, until you have patched up to SP6a + the SP6a URP (Q299444). You can further improve performance by enabling DMA Mode on the IDE adapter and write caching on the VHDX.
  7. NT 4.0 by default does not use SMB signing and uses LAN Manager authentication instead of NTLM. It can use NTLM v1/2 once it has been fully patched. However, be aware that this means that it will be unable to communicate with Windows XP SP2+ or Windows Server 2003 in their default configurations. You will have to perform some security hardening on NT 4.0 or security weakening on XP+ to get SMB working. Hint: It’s the same process in the registry on both, so security harden NT 4.0 after installing SP6a.
  8. NT 4.0 ONLY supports SMB 1.0 / CIFS (“SMB 1.5”). Microsoft have been removing support for SMB 1.0 with each successive Windows release. Under Widows Server 2016 and Windows 10 SMB 1.0 support is an optional component/feature that you may need to install manually.
    Note: You should not be using SMB 1.0 at all in 2018 as it is a 100% exploitable security risk.
  9. After you have performed the install, you may be looking for the easiest way to copy SP6a, Internet Explorer, patches and app installers to the VM. To do this as fast as possible without having to pre-harden the OS either burn the updates into an ISO or do it via Guest authenticated SMB by:
    1. Enable the guest account
    2. Create a SMB share on the root of the C Drive and set Guest access to read/write (modify) under NTFS and Share permissions.
      Note: before it is patched, you will struggle to SMB into NT 4.0 using a username and password combination unless you weaken the security policy on the calling client. Using the guest account bypasses the problem.
    3. Use an intermediate level VM/system as a bridge between newer and older SMB versions. For example I used a Windows Server 2008 VM to pull data from third server with a SMB 2.x file share of updates and drop them onto the NT 4.0 SMB 1.0 share found at c:\shared.
    4. Once NT 4.0 is patched, you should disable the guest account again, remove its permissions to the file share and authenticate into NT 4.0 using a normal user account found in the SAM database. Do note my warning above about SMB signing however, which will scupper you unless you have made mitigations via hardening.
Once you have done all of the above, and have a fully patched system. You will have something resembling the below running in Hyper-V 2016.
NT 4.0 in Hyper-V 2016
NT 4.0 on Windows 10 via Windows Server 2016 Hyper-V install

PowerShell – Convert DER Encoded Certificate file into a Base64 .cer

System Requirements:

  • Windows PowerShell

The Problem:

If you have a binary encoded .cer (certificate) file that you need to get into a Base64 format, you can either follow the advice and use OpenSSL to convert it or you can import it into the Windows Certificate Store and re-export it.

If you want to do it automatically, without having to download and install anything else, neither option is particularly appealing.

The Fix

You can use the following function to convert the binary encoded file into a Base64 ASCII encoded file

function Convert-CertificateBinaryToBase64 {
param( [string]$SourceFile, [string]$DestinationFile )
$cert = get-content "$SourceFile" -Encoding Byte
$content = @(
'-----BEGIN CERTIFICATE-----'
[System.Convert]::ToBase64String($cert, 'InsertLineBreaks')
'-----END CERTIFICATE-----'
)$content | Out-File -FilePath "$DestinationFile" -Encoding ASCII
}

Example usage

Convert the file, retaining the source file

Convert-CertificateBinaryToBase64 -Sourcefile 'C:\myBinary.cer' -DestinationFile 'C:\myBase64.cer'

Convert the binary file, overwriting it with the Base64 file

Convert-CertificateBinaryToBase64 -Sourcefile 'C:\myCertificate.cer' -DestinationFile 'C:\myCertificate.cer'

“RPC server unavailable. Unable to establish communication between and ” when connecting to Hyper-V 2008, 2008 R2, 2012, 2012 R2 from Hyper-V Manager version 1709

System Requirements:

  • Windows 10 1709
  • Windows Server 2016
  • Hyper-V Management Console
  • RSAT 2016/1709 for Windows 10 version 1709

The Problem:

After upgrading to Windows 10 version 1709 and installing the updated Windows Server 2016 (version 2016 or version 1709) RSAT tools for Windows 10 1709. On attempting to connect to a down-level Windows Server 2012 R2, 2012, 2008 R2, 2008 Hyper-V Server via the Hyper-V Manager MMC snap-in. You receive the error even though no configuration changes have been made on the Hyper-V hosts:

"RPC server unavailable. Unable to establish communication between <management host> and <Hyper-V host>"

At this point you are unable to manage down-level version of Hyper-V from Windows 10. This issue does not impact the management of remote Windows Server 2016 or Windows Server 1709 Hyper-V instances.

View: Remote Server Administration Tools for Windows 10 (RSAT)

The Fix

This appears to be related to a change in the default firewall behaviour on Windows 10 1709 installs. to fix the problem. On the client system, where you have installed RSAT to remote manage the hypervisor (i.e. not on the hypervisor itself):

  1. Open ‘Administrative Tools’ in the Windows Control Panel
  2. Open ‘Windows Defender Firewall with Advanced Security’
  3. Select ‘Inbound Rules’ from the left hand side
  4. Scroll down until you get to ‘Windows Management Instrumentation (ASync-In)’
  5. Enable the rule for domain/private/public networks as required
    Note: By default the Windows firewall MMC will only display WMI rules for domain and private networks. If you are not running against a domain and Windows has not been explicitly told that you are on a private network, Windows will assume that you are on a public network. Check in network settings in the settings app to ensure that you are not running on a public network, or if you are edit the firewall rule to include public networks. In general, it is a bad idea to open WMI up to traffic on public networks.
  6. Restart Hyper-V Manager

You should now find that you can connect to down-level versions of Hyper-V from Windows 10 1709.