Memory Leak in SvcHost.exe on Microsoft.XmlHttp (IXMLHTTPRequest) .Send() when called from CScript.exe or WScript.exe

System Requirements:

  • Windows Server 2008 R2

The Problem:

Svchost.exe, that black box amongst many other black boxes. If you ever happened to be in the business of watching what your scripts are getting up to on a Sunday morning and you are using Microsoft.XmlHttp, then you might be in for a surprise.

Every 2 hours a batch process on a group of servers fires off a script that in turn iteratively runs a second VBS script some 200-300 times. The script, calls a Web Service and performs a push/pull of instructions. Within a few days of the patch Tuesday reboot, you start noticing that memory use is going up, and up, and up.

You’ve done all of your deallocations, right? “set xmlHttp = nothing”? Yep, but despite that, memory use continues to grow. The culprit, svchost.exe. It grows until it’s into the page file and then grows a little bit more. Every run of the script puts between 4 and 100KB onto the memory footprint. At the end of the month, the servers are groaning because of memory starvation and your SAN array’s are not happy because of all of the paging.

True story.

More Info

I have been able to reproduce this on 3 separate and wholly independent Server 2008 systems (read different clients, enterprise/retail licensing, server hardware and install images) as well as on related servers (read from the same image on same or similar hardware). I have attempted to reproduce it on Windows Server 2012 R2 and I was not successful. Server 2012 R2 does not appear to be impacted by the issue. Running the iterator loop below for 10 minutes yields no increase in the memory use curve on the operating system, just a constant cycle of assign, release, assign, release that you would expect to see.

After a lot of diagnostics and a lot of me initially assuming that the problem was the web service (many, many wasted hours… although I did find a few bugs in the service code itself…) I managed to narrow it down to Microsoft.XmlHttp. More specifically, it’s in the way that CScript or WScript interfaces with Microsoft.XmlHttp at initialisation.

As you probably know, svchost itself is just a service wrapper. Inspection of the wrapper reveals a number of services running inside the wrapper. In this case the specific services are:

  • COM+ Event System
  • Windows Font Cache Service
  • Network List Service
  • Network Store Interface Service
  • Secure Socket Tunneling Protocol Service
  • WinHTTP Web Proxy Auto-Discovery

There are two things here that could be interesting, COM+ Event System and WinHTTP Web Proxy. Microsoft.XMLHTTP itself relies upon the WinHTTP stack for operation, but we are also using a COM interface to call it from VBScript.

While we cannot shutdown the COM+ Event Service and expect the operating system to survive for long, we can the WinHTTP Web Proxy Auto-Discovery Service. Did it release the memory consumed in the leak? No. So in the balance of probabilities, it’s coming from COM+.

The problem with that is in the need to reboot the server to safely clear the memory leak, hence why Patch Tuesday has been the true savior in keeping a gradual performance bottle neck from becoming a full scale meltdown. So what is going on?

I stripped off all of the web service and customisation parts and went back to vanilla Microsoft implementation examples. We cannot get much simpler than this.

Option Explicit
Dim xmlset xml = CreateObject("Microsoft.XmlHttp")
xml.open "POST", "http://127.0.0.1", false
xml.send "he=llo"
set xml = nothing

Save it to a VBS and run it via CScript, run it a lot. Run it in a BAT file loop

:start
ccscript.exe testfile.vbs
goto start

Watch the svchost.exe processes until you spot the instance with the rising service working set (or private set). Now you know which one to focus on.

It’s memory leaked. Hold on, we’ve created the instance of Microsoft.XmlHttp (which is actually an instance of IXMLHTTPRequest), done something and told CScript to deallocate it (set xml = nothing). Why is it leaking memory?

The third parameter on .Open() is bAsync – is it an asynchronous request? It’s false above, meaning that the request is synchronous. It continues to leak. It would be more likely to leak asynchronously than synchronously, however changing that to true makes no difference.

So where is the leak being triggered? By process of line elimination we can reveal that the memory is committed into the svchost wrapper during xml.send(). Run it without .Send() as below and there is no growth in the scvhost process memory footprint no matter how many times you run it..

Option Explicit
Dim xmlset xml = CreateObject("Microsoft.XmlHttp")
xml.open "POST", "http://127.0.0.1", false
' COMMENTED OUT      xml.send "he=llo"
set xml = nothing

In the MSDN documentation for the .Send() method, it states

“If the input type is a BSTR, the response is always encoded as UTF-8. The caller must set a Content-Type header with the appropriate content type and include a charset parameter.”

So far we haven’t done that and we are sending a VBString – which is ultimately a BSTR in C++, so add in the necessary setRequestHeader beneath the .Open() method call in case it is a case of not following the documentation:

Option Explicit
Dim xmlset xml = CreateObject("Microsoft.XmlHttp")
xml.open "POST", "http://127.0.0.1", false
xml.setRequestHeader "Content-Type", "application/x-www-form-urlencoded; Charset=UTF-8"
xml.send "he=llo"
set xml = nothing

It isn’t. There is no change, it still results in an increase in process memory after cscript.exe has shutdown.

We have confirmed that there is a memory leak, where it is and what is triggering it. We can also be confident that given the extremely simple nature of the sample code printed above – and its match to the samples documentation – that it is being implemented correctly.

So the next step is to try and prove that there is an issue in the COM implementation between CreateObject and set nothing. This is achieved by running the allocate/deallocate (set/set nothing) in a loop as shown below

Option Explicit
Dim i
Dim xmlwscript.echo TypeName(xml)              ' This returns "empty" on this testfor i = 0 to 999

set xml = CreateObject("Microsoft.XmlHttp")
xml.open "POST", "http://127.0.0.1", false
xml.setRequestHeader "Content-Type", "application/x-www-form-urlencoded; Charset=UTF-8"
xml.send "he=llo"
wscript.echo xml.responsexml.xml    ' This returns nothing on this test
wscript.echo xml.statusText         ' This returns "OK" on this test
set xml = nothing

next

wscript.echo TypeName(xml)              ' This returns "nothing" on this test

At this point you would expect to see a large increase in the svchost.exe memory footprint.

It does not happen.

1000 iterations and instantiation of the IXMLHTTPRequest later and there is no obvious exponential increase in the memory footprint of svchost.exe. It simply increments once i.e. the additional memory consumption is no worse than running the script with only 1 call to CreateObject/set nothing despite the fact that .send() has been called 1000 times.

What does that mean? Well, it would seem to suggest that the fault isn’t actually in IXMLHTTPRequest (Microsoft.XMLHTTP), but actually in VBScript itself. As a speculative suggestion, I would suggest that VBScript is registering event callbacks with COM+’s Event Management System on the first call to .Send() which are not being cleaned up by the garbage collector when “set nothing” is called in the code. So either there is a bug in VBScript or there is a bug in the event handling interface for COM+ event registration through which IXMLHTTPRequest is registering its own actions.

Most people aren’t going to notice this problem, they are morelikely to iterate instance of Microsoft.XmlHttp inside VBScript than they are to repeatedly externally iterate accross it. It just so happens that I need to fire it externally to the script processor via the command shell. The chances are that if you are reading this, so do you.

The Fix

As of writing, I have not found a direct way to force VBScript to release the memory from scvhost, short of rebooting (or migrating to Windows Server 2012). Calling Microsoft.XmlHttp from WScript or CScript seems to be the problem and the fact that the web service scripts are using an external iterator to repeatedly call n new instances of CScript are exacerbating the situation. Simply put, the transaction load is the catalyst for spotting the leak. In most cases growth would be very subtle as would growth were the iteration internal to the CScript.exe script instance.

While not necessarily ideal, if you are in the position of being able to change provider, you can substitute Microsoft.XMLHTTP for MSXML2.ServerXmlHttp, which provides most of the functionality without making use if WinHTTP. This provider does not exhibit the memory growth issue as in its client counterpart, however its use requires MSXML 3 or 6 and you lose some functionality.

The fact that I could not reproduce the issue under Windows Server 2012 R2 suggests that the culprit has been fixed – either intentionally or inadvertantly. By default, Microsoft.XMLHTTP is a COM Class ID reference to msxml3.dll. Under Windows Server 2008 R2 the file version is SP11 at 8.110.7601.18334, under 2012 R2 the file version is simply 8.110.9600.16483. Yet oddly, with all systems fully patched, vbscript.dll under Windows Server 2008 R2 is version 5.8.9600.17041 (KB2929437) while its counterpart under Server 2012 R2 is 5.8.9600.17031.

What I can tell you is that these systems have been running this recursion script every 2 hours since the beginning of 2012 and the issue has only been observed in more recent months, therefore I suspect that Microsoft have a regression bug on their hands. Until it is fixed however, I have a load of (thankfully firewalled, private network) web service that have a DOS vulnerability. So do you.

Compacting a Linux Hyper-V, Virtual PC 2004 or 2007 Dynamic VHD/VHDX file

System Requirements:

  • Virtual PC 2004, SP1
  • Virtual PC 2007, SP1
  • Hyper-V

The Problem:

The Windows Integration components ISO contains a tool for performing a VHD pre-compact. Once completed, this tool allows you to shut down a VHD and compact it to reclaim disk space previously used by the VM which has actually marked as being free space by the VHD FAT/MFT.

More Info

As Linux operating systems do not formally include integration components for Virtual PC, there is no standard Microsoft way to pre-compact the VHD. The fundamental process behind pre-compaction is however simple: write 0’s all sectors of the hard drive that the FAT/MFT claim to be free (available) space.

Once you have done this, the Virtual PC/Hyper-V management UI will flush the contents of the VHD into a new VHD, skipping any 0’d sectors from the migration process, thus reducing the size of the VHD as seen by the hypervisors host partition.

The Fix

I’ve used two different methods to achieve Linux pre-compaction.

Note: This was written against the steps that need to be performed on Debian. Other distributions may require additional steps to install software.

Open Terminal and execute the following:

su
cd /
cat /dev/zero > zero.dat ; sync ; sleep 1 ; sync ; rm -f zero.dat

Although simpler and not requring any software installation, the compaction rate achieved using the method was a little hit and miss, often resulting in a large VHD file after the compaction process.

A higher success rate was achieved by installing a small app package called secure-delete. To install and use this issue the following commands from a shell.

su
apt-get install secure-delete
sfill -f -z /

While I have realised better results from this, the process takes far longer as the secure-delete packages sfill is in fact performing an “insecure” erase on the hard drive, not simple zeroing sectors. This means that it actually passes over the disk more than once, with the concluding pass being the zero pass. The cumulative effect of the multiple passes means that it takes far longer to complete. The last time that I ran with this method, a 16 GB Debian 6 VHD was reduced by 1.2 GB. Small change given the size of today’s hard drive, but a significant percentage of the 16GB disk none the less.

Please also note that there are other elements to the secure-delete package, including tools to wipe the SWAP partition (which may further reduce the size of the VHD if used) as well as tools to perform a full, secure disk erase (not just empty sector erasing). So do ensure that you use this package carefully.

Why releasing the Start Menu for Windows 8.1 is great, but also a mistake

System Requirements:

  • Windows 8

The Problem:

It’s all many of us ever wanted, right? The option to choose between the old way and the new way, to transition over to what Microsoft was telling us was a better way and to do it in our own time? After all, we could do it freely with Windows 95 and the transition between File Manager and Windows Explorer. We could do it in XP and even Vista between the Windows 2000 cascading start menu and the XP style ‘most recently used, shortcuts plus cascading view’ that so many of us yearn to have back in Windows 8.

It was an own goal to try and forcibly advance the entire Windows ecosystem onto a new paradigm in the first place. To have tried to force it onto one that wasn’t evolved and had significant if not glaring gaps in its usability, documentation and general design was an even bigger, if not typically Microsoft own goal.

Since the announcement of the return of the start menu to Windows 8.1, I’ve been internalising thought over finally migrating from Windows 7 – with the demo video displaying the return of my much used start menu ‘application most recently used’ jump lists and a way to avoid the start screen completely. I’ll happily put up with live tiles, apps and Microsoft accounts in the knowledge that after 10 minutes I will have turned it all off.
Yet I also think that it is a mistake for Microsoft to release it into Windows 8.1. Windows 8 and its hastily rushed out 8.1 successor are nothing more than a consume psyche car crash. It’s been long stated and to say that it was a slow motion car crash is wholly wrong – it was blatantly obvious from the release candidates. To release the start menu against Windows 8.0, 8.1 or even as 8.2 would not be a commercially optimal decision.

“Windows 8.x sucks” is now vehemently burned into the psyche of consumer and business decision maker alike. I myself have sat in meetings and outright banned it from client deployment and purchasing. Windows 8 quite simply cannot be given to the average user. It isn’t fit for purpose and where funds permit the reaction that I am increasingly seeing from end users in both consumer and enterprise settings is simply to go and get Mac if they are forced into a corner over Windows 8 adoption. The reasons for this are multiple, varied and well trodden in all corners of the IT press. The common ground (which does not include the UI) usually include the app store eco system being weak, under developed and problematic while for enterprise the entire concept of “Microsoft account” is laughable if not somewhat sinister. That’s just the way it is.

Yet we only have to look back to late 2006 to see the reason why releasing the start menu back against 8.1 is a bad idea.

The RTM of Vista did more damage to Microsoft’s consumer reputation than quite possibly Windows 8 has or will ever manage. It didn’t matter that SP1 fixed Vista’s issues and that with its release, device manufacturers – who had procrastinated in developing drivers during the beta phases – had finally woken up and starting writing / offering better quality drivers for NT 6. It didn’t matter that RAM prices had come down and all those cheap, bad decision netbooks with 512MB of RAM could now take 10 minutes to boot instead of 25. Even by SP2 and Vista finally maturing, it didn’t matter. The damage was done. “Vista sucks”, that was all there was to it.

Windows 8 is tarred by the same brush; be it for different reasons. It is doomed to have an ever tightening noose around its neck caused by the decisions made ahead of its RTM. Sure, once integrated images are available Microsoft may start to sway business decision makers as well as technologists like myself towards offering up a more positive stance… but the consumer market?

Regrettably Microsoft learnt from the mistakes of the Vista release. To date they’ve stopped as many people as they possibly can from getting hold of a Windows 7 device. Rewind back to 2006 with Vista and Windows XP OS shipments at the point of purchase lingered right up until the release of Windows 7. Was this a good thing for Microsoft to learn? Sure, Windows 8 is now the highest selling OS Microsoft has ever produced – second only to Windows 7 that is. Does the consumer care? Well, no. Not really. The computing market is far larger, far lower cost now compared to 8 years ago. High sales volume was inexorable, especially once Microsoft closed the taps on the supply of Windows 7. It doesn’t mean that any of these increased numbers of people like what they have, it is simply a reflection of not knowing any better and the free market alternatives still being too niche at this point in time.

Since its release, I personally have only encountered one user who said they really like Windows 8 and in that would’t contemplate moving back down to Windows 7 (and they were a Surface user, not a laptop or desktop one). By far the majority of people that I speak to would like Windows 7 back if they had the option and could cost rationalise it and will ultimately suffer Windows 8 by ignoring most of its innovations and keeping their taskbar come desktop cluttered with launcher shortcuts.

The benefit of all of this has of course been to Apples favour. People are either paying to have 8 removed (difficult and occasionally impossible due to UEFI restrictions on consumer grade hardware), avoiding purchase altogether, going to non-Windows tablet devices and in the rapidly shrinking desktop space… I’m seeing an awful lot more Apple out there.

Apple may be many things; I am not a fan in any size, shape or form. My contempt for their software solutions is well known. Yet you would never see Apple committing this kind of carnal sin against their consumer base. They are more subtle, more discreet. Apple looks at the long term transitioning picture instead of a monolithic sledgehammer approach used by Microsoft to bully consumers into their own way of thinking.
Don’t get me wrong here, Apple always gets it way. The difference is that by the time they do get their own way, hardly anyone who notices is going to complain, the changes smooth, sleek, refined. Those who do complain are by this point a distinct minority. Progress made. Consumer base increased. Negative publicity: nil to irrelevant.

Of course I would personally like to see Windows 8.1 with the option to toggle the start menu / start screen on and off. Of course the Start Screen should continue to exist – on the Surface RT it works, it makes sense, it’s good. Here it is certainly better than using the desktop environment. Yet for Microsoft to have a happy release, to regain their consumer reputation as well as their business reputation the answer is simple.

Don’t like Windows 8/8.1? Here is Windows 9. We’ve listened to YOU, we’ve refined the offering based upon YOU, your wishes, your feedback. If at this point you are feeling an innate sense of déjà vu, this strategy was itself the drum beat behind the marketing strategy for Windows 7 – Windows that you (I) created.

This focus, strategy and tone based around the concept of ‘you’ was what was missing at the Windows 8.1 start menu announcement a week or so ago – a conciliatory tone, even an apology was itself absent. If they had given one, I think many (including myself) would have respected them far more for giving it than anything that the wet attempt to make it look like it was a planned change has done.

An even more critical message would be for people to hear that if they have an 8.0 or 8.1 system, their investment will run Windows 9 with no performance loss. For Surface users, I’m sure they’d like to hear that they’ll be getting it for free. For everyone else, how about a Mac of old style “£19.99” upgrade cost from 8.0 or 8.1 up to Windows 9. Even for Windows 7 users, for which a discounted offering which would make sense as a way to eliminate platform fragmentation and increase exposure to the Windows App Store ecosystem – in turn making the Metro environment that little bit sweeter for developers and consumers alike. If that idea sounds overtly familiar it’s because it is. This was Apple’s greatest decision of 2013; defragment their desktop ecosystem in one hit, open up their app store and APIs to as many people as possible in order to make lots more money and wrap it up in a double edged goodwill gesture named Mavericks.

At this stage does it matter if Windows 9 is Windows 8.1 with a start menu and not much else? Microsoft has done a lot of damage to the Windows brand on its own over the last 8 years when factoring in both the Vista and the Windows 8 debacle. It will take time to undo that. Yet if letting people have their moment in slamming Windows 8 into becoming a distant memory is the thing which allows the healing to take place, isn’t that commercially better for everyone in the Microsoft hegemony?

You can send audio but not receive it when using Skype for Android

System Requirements:

  • Android
  • Skype for Android 4.5.0.39600

The Problem:

When you make a Skype call on an Android device (Skype for Android 4.5.0.39600 at the time of writing) your partner can hear you and see you, however you cannot hear them.

This problem was observed on a Barnes & Noble Nook HD+ running ICS 4.0.4, but should by no means be exclusive to this equipment.

More Info

Quite simply, the fix to this one was that the Android device had been left with a pairing state with a Bluetooth Audio device – one that was not connected. This occurred irrespective of whether Bluetooth was on or off or the speaker in range.

By unpairing the device from the bluetooth partners list in Android Settings, the sound started to work once again at the next establishment of a call.

As other apps were using Audio correctly via the built-in speakers, I suspect this to be a bug in Skype rather than Android or Barnes & Noble’s firmware.