Memory Leak in SvcHost.exe on Microsoft.XmlHttp (IXMLHTTPRequest) .Send() when called from CScript.exe or WScript.exe

System Requirements:

  • Windows Server 2008 R2

The Problem:

Svchost.exe, that black box amongst many other black boxes. If you ever happened to be in the business of watching what your scripts are getting up to on a Sunday morning and you are using Microsoft.XmlHttp, then you might be in for a surprise.

Every 2 hours a batch process on a group of servers fires off a script that in turn iteratively runs a second VBS script some 200-300 times. The script, calls a Web Service and performs a push/pull of instructions. Within a few days of the patch Tuesday reboot, you start noticing that memory use is going up, and up, and up.

You’ve done all of your deallocations, right? “set xmlHttp = nothing”? Yep, but despite that, memory use continues to grow. The culprit, svchost.exe. It grows until it’s into the page file and then grows a little bit more. Every run of the script puts between 4 and 100KB onto the memory footprint. At the end of the month, the servers are groaning because of memory starvation and your SAN array’s are not happy because of all of the paging.

True story.

More Info

I have been able to reproduce this on 3 separate and wholly independent Server 2008 systems (read different clients, enterprise/retail licensing, server hardware and install images) as well as on related servers (read from the same image on same or similar hardware). I have attempted to reproduce it on Windows Server 2012 R2 and I was not successful. Server 2012 R2 does not appear to be impacted by the issue. Running the iterator loop below for 10 minutes yields no increase in the memory use curve on the operating system, just a constant cycle of assign, release, assign, release that you would expect to see.

After a lot of diagnostics and a lot of me initially assuming that the problem was the web service (many, many wasted hours… although I did find a few bugs in the service code itself…) I managed to narrow it down to Microsoft.XmlHttp. More specifically, it’s in the way that CScript or WScript interfaces with Microsoft.XmlHttp at initialisation.

As you probably know, svchost itself is just a service wrapper. Inspection of the wrapper reveals a number of services running inside the wrapper. In this case the specific services are:

  • COM+ Event System
  • Windows Font Cache Service
  • Network List Service
  • Network Store Interface Service
  • Secure Socket Tunneling Protocol Service
  • WinHTTP Web Proxy Auto-Discovery

There are two things here that could be interesting, COM+ Event System and WinHTTP Web Proxy. Microsoft.XMLHTTP itself relies upon the WinHTTP stack for operation, but we are also using a COM interface to call it from VBScript.

While we cannot shutdown the COM+ Event Service and expect the operating system to survive for long, we can the WinHTTP Web Proxy Auto-Discovery Service. Did it release the memory consumed in the leak? No. So in the balance of probabilities, it’s coming from COM+.

The problem with that is in the need to reboot the server to safely clear the memory leak, hence why Patch Tuesday has been the true savior in keeping a gradual performance bottle neck from becoming a full scale meltdown. So what is going on?

I stripped off all of the web service and customisation parts and went back to vanilla Microsoft implementation examples. We cannot get much simpler than this.

Option Explicit
Dim xmlset xml = CreateObject("Microsoft.XmlHttp")
xml.open "POST", "http://127.0.0.1", false
xml.send "he=llo"
set xml = nothing

Save it to a VBS and run it via CScript, run it a lot. Run it in a BAT file loop

:start
ccscript.exe testfile.vbs
goto start

Watch the svchost.exe processes until you spot the instance with the rising service working set (or private set). Now you know which one to focus on.

It’s memory leaked. Hold on, we’ve created the instance of Microsoft.XmlHttp (which is actually an instance of IXMLHTTPRequest), done something and told CScript to deallocate it (set xml = nothing). Why is it leaking memory?

The third parameter on .Open() is bAsync – is it an asynchronous request? It’s false above, meaning that the request is synchronous. It continues to leak. It would be more likely to leak asynchronously than synchronously, however changing that to true makes no difference.

So where is the leak being triggered? By process of line elimination we can reveal that the memory is committed into the svchost wrapper during xml.send(). Run it without .Send() as below and there is no growth in the scvhost process memory footprint no matter how many times you run it..

Option Explicit
Dim xmlset xml = CreateObject("Microsoft.XmlHttp")
xml.open "POST", "http://127.0.0.1", false
' COMMENTED OUT      xml.send "he=llo"
set xml = nothing

In the MSDN documentation for the .Send() method, it states

“If the input type is a BSTR, the response is always encoded as UTF-8. The caller must set a Content-Type header with the appropriate content type and include a charset parameter.”

So far we haven’t done that and we are sending a VBString – which is ultimately a BSTR in C++, so add in the necessary setRequestHeader beneath the .Open() method call in case it is a case of not following the documentation:

Option Explicit
Dim xmlset xml = CreateObject("Microsoft.XmlHttp")
xml.open "POST", "http://127.0.0.1", false
xml.setRequestHeader "Content-Type", "application/x-www-form-urlencoded; Charset=UTF-8"
xml.send "he=llo"
set xml = nothing

It isn’t. There is no change, it still results in an increase in process memory after cscript.exe has shutdown.

We have confirmed that there is a memory leak, where it is and what is triggering it. We can also be confident that given the extremely simple nature of the sample code printed above – and its match to the samples documentation – that it is being implemented correctly.

So the next step is to try and prove that there is an issue in the COM implementation between CreateObject and set nothing. This is achieved by running the allocate/deallocate (set/set nothing) in a loop as shown below

Option Explicit
Dim i
Dim xmlwscript.echo TypeName(xml)              ' This returns "empty" on this testfor i = 0 to 999

set xml = CreateObject("Microsoft.XmlHttp")
xml.open "POST", "http://127.0.0.1", false
xml.setRequestHeader "Content-Type", "application/x-www-form-urlencoded; Charset=UTF-8"
xml.send "he=llo"
wscript.echo xml.responsexml.xml    ' This returns nothing on this test
wscript.echo xml.statusText         ' This returns "OK" on this test
set xml = nothing

next

wscript.echo TypeName(xml)              ' This returns "nothing" on this test

At this point you would expect to see a large increase in the svchost.exe memory footprint.

It does not happen.

1000 iterations and instantiation of the IXMLHTTPRequest later and there is no obvious exponential increase in the memory footprint of svchost.exe. It simply increments once i.e. the additional memory consumption is no worse than running the script with only 1 call to CreateObject/set nothing despite the fact that .send() has been called 1000 times.

What does that mean? Well, it would seem to suggest that the fault isn’t actually in IXMLHTTPRequest (Microsoft.XMLHTTP), but actually in VBScript itself. As a speculative suggestion, I would suggest that VBScript is registering event callbacks with COM+’s Event Management System on the first call to .Send() which are not being cleaned up by the garbage collector when “set nothing” is called in the code. So either there is a bug in VBScript or there is a bug in the event handling interface for COM+ event registration through which IXMLHTTPRequest is registering its own actions.

Most people aren’t going to notice this problem, they are morelikely to iterate instance of Microsoft.XmlHttp inside VBScript than they are to repeatedly externally iterate accross it. It just so happens that I need to fire it externally to the script processor via the command shell. The chances are that if you are reading this, so do you.

The Fix

As of writing, I have not found a direct way to force VBScript to release the memory from scvhost, short of rebooting (or migrating to Windows Server 2012). Calling Microsoft.XmlHttp from WScript or CScript seems to be the problem and the fact that the web service scripts are using an external iterator to repeatedly call n new instances of CScript are exacerbating the situation. Simply put, the transaction load is the catalyst for spotting the leak. In most cases growth would be very subtle as would growth were the iteration internal to the CScript.exe script instance.

While not necessarily ideal, if you are in the position of being able to change provider, you can substitute Microsoft.XMLHTTP for MSXML2.ServerXmlHttp, which provides most of the functionality without making use if WinHTTP. This provider does not exhibit the memory growth issue as in its client counterpart, however its use requires MSXML 3 or 6 and you lose some functionality.

The fact that I could not reproduce the issue under Windows Server 2012 R2 suggests that the culprit has been fixed – either intentionally or inadvertantly. By default, Microsoft.XMLHTTP is a COM Class ID reference to msxml3.dll. Under Windows Server 2008 R2 the file version is SP11 at 8.110.7601.18334, under 2012 R2 the file version is simply 8.110.9600.16483. Yet oddly, with all systems fully patched, vbscript.dll under Windows Server 2008 R2 is version 5.8.9600.17041 (KB2929437) while its counterpart under Server 2012 R2 is 5.8.9600.17031.

What I can tell you is that these systems have been running this recursion script every 2 hours since the beginning of 2012 and the issue has only been observed in more recent months, therefore I suspect that Microsoft have a regression bug on their hands. Until it is fixed however, I have a load of (thankfully firewalled, private network) web service that have a DOS vulnerability. So do you.

How to extend Net-SNMP to retrieve system specification / hardware specifications from system_profiler under OS X

System Requirements:

  • OS X 10.2+
  • Net-SNMP

The Problem:

The thing that I have never understood with SNMP is why others, particularly OEM’s do not see it as a extremely useful and worth while (if badly implemented) technology.

If you have ever attempted to lookup system information on Windows, you will probably have used WBEM; it is eloquent, effective and Microsoft do a great job of keeping it fresh and current. While WBEM is available as OpenWBEM for Unix/Linux, it seems more like something designed to ward small children from a path along the dark side than anything usable by system administrators.

Unfortunately while WBEM on Windows is well maintained by Microsoft, the same is not said of SNMP. While WBEM has been enriched with access to new vectors on system analysis, most implementations of SNMP are nothing more than the most basic set of target variables available through a default Net-SNMP install – in short very little of any real use.

Cue Apple to make a bad situation worse – in a way only they know how.

OS X 10.6

Enabling SNMP on OS X 10.6, I noticed a fairly glaring problem

.1.3.6.1.2.1.2.2.1.6.4 = STRING: 00:93:e9:05:1c:67
.1.3.6.1.2.1.2.2.1.6.5 = STRING: 00:93:e9:05:1c:67

SNMP was returning the same MAC address for two separate adapters. What that should have said was:

.1.3.6.1.2.1.2.2.1.6.4 = STRING: 10:93:e9:05:1c:67
.1.3.6.1.2.1.2.2.1.6.5 = STRING: 02:93:e9:05:1c:67

Notice that the first hex pair in the actual output has been 00’d.

  1. So I remade the SNMP configuration files
  2. Reset the services
  3. Fully patched OS X 10.6 up to 10.6.8

No change! So off we go to the App store and buy 10.7…

OS X 10.7

After the install, same problem.

So I fully patched it up to 10.7.4… same problem.

Then I noticed something else…

Under 10.6.8 “system memory size in bytes” or in the vernacular of SNMP .1.3.6.1.2.1.25.2.2 or iso.identified-organization.dod.internet.mgmt.mib-2.host.hrStorage.hrMemorySize was returning “INTEGER: 2147483648”. Under 10.7.0 and 10.7.4 it now returned “”.

Yep, nothing, nada, zip.

So not only paying out for the OS upgrade failed to fix the problem, but it has made things worse!

In Summary

If you are determined to use SNMP to obtain system information on a Mac, what can you do given that Apple have crippled it? Read-on.

More Info

Here is an up front disclaimer for you

  1. This is not ideal
  2. It does not follow in the spirit for which SNMP was designed
  3. It does work
  4. I’ve only validated it under 10.7 but in theory it should work down to at least 10.2

The Fix

Net-SNMP (the SNMP service that ships with OS X) can of course be extended with Management Information Base (MIB) extensions to formally restore this missing functionality. You as a diligent systems administrator can of course write your own because Apple clearly don’t want to provide any.

It is however quite time consuming to write your own MIB and integrate it into SNMP. So what I propose is using the SNMP Enterprise Extension MIB’s to retrieve the information that you require on the local host and make it available through the snmp daemon.

To make use of the Enterprise Extensions:

  1. Open your preferred shell (Terminal)
  2. cd /private/etc/snmp
  3. sudo emacs OSX_Inventory.sh
  4. Provide your password and then type out the following (pay attention to where I have spaces, they are important):
    arg=$1
    if [[ “$arg” == “SerialNumber” ]]; then
    /usr/sbin/system_profiler SPHardwareDataType | grep “Serial Number” | cut -d”:” -f2- | cut -d” ” -f2-
    elif [[ “$arg” == “ModelNumber” ]]; then
    /usr/sbin/system_profiler SPHardwareDataType | grep “Model Identifier” | cut -d”:” -f2- | cut -d” ” -f2-
    fi
  5. Save the file (Ctrl + X, Ctrl + S) and then exit (Ctrl + X, Ctrl + C)
  6. Configure SNMP as you require:
    snmpconf -g basic_setup
  7. cd /private/etc/snmp (if you left the directory)
  8. sudo emacs snmpd.conf
  9. Provide your password if asked
  10. Scroll down until you get to the “Executables/scripts” section. Beneath “exec echotest /bin/echo hello world” add:
    exec info_SerialNumber       /bin/sh /private/etc/snmp/OSX_Inventory.sh SerialNumber
    exec info_ModelNumber       /bin/sh /private/etc/snmp/OSX_Inventory.sh ModelNumber
  11. Save the file (Ctrl + X, Ctrl + S) and then exit (Ctrl + X, Ctrl + C)
  12. Stop the SNMPd service:
    sudo killall snmpd
  13. Restart it (OS X will do this for you if you have set it to auto-start):
    snmpd
  14. Now issue a SNMPTable command against the User account / community string you specified in SNMP setup against this OID: .1.3.6.1.4.1.2021.8 e.g.
    snmptable -v 1 -c private localhost .1.3.6.1.4.1.2021.8

If you did everything correctly you should see something along the lines of:

extIndex extNames extCommand extResult extOutput extErrFix extErrFixCmd
INTEGER: 1 STRING: info_SerialNumber STRING: /bin/sh INTEGER: 0 STRING: <YOUR SERIAL NUMBER HERE>
INTEGER: 2 STRING: info_ModelNumber STRING: /bin/sh INTEGER: 0 STRING: MacBookAir3,2

If you extend the OSX_Inventory.sh script with additional field definitions and add them to the SNMPd.conf, you can retrieve any information you want from SNMP using SNMPGet, SMNPTable or SNMPWalk against the descendants of .1.3.6.1.4.1.2021.8.

Things to keep in mind:

  1. You can only return one (1) line via the Enterprise extensions section, anything after the first LF is dropped. If you need to return multiple lines, I suggest storing all lines in a variable in OSX_Inventory.sh and using tab separation by replacing LF (\n) with TAB (\t).
  2. Enterprise management and monitoring systems will not know about these extensions, clearly they are proprietary to you.
  3. You will need to distribute the modified snmpd.conf file and the OSX_Inventory.sh file to all clients and maintain them (you had to do it with snmpd.conf anyway)
  4. Don’t forget to start the SNMPd service at boot on your client system
  5. Don’t forget to restart the SNMPd service after making changes to your snmpd.conf file or you will not see the results of any changes

PEAR for PHP Error “No releases available for package “xxx” install failed” after running ‘pear install xxx’ on Windows Server 2008

System Requirements:

  • Windows Vista
  • Windows 7
  • Windows Server 2008, 2008 R2

The Problem:

You know that something a bit odd is going on when one of a batch of servers starts throwing errors that the others sailed past. In this case trying to configure PEAR for a new PHP install with Mail, Mail_Mime and Net_SMTP (pear.php.net/mail, pear.php.net/mail_mime and pear.php.net/net_smtp) should be fairly standard. The other servers took the install and even this server too Mail and Mail_Mime but would not accept Net_SMTP returning:

C:\Program Files (x86)\PHP>pear install net_smtp
No releases available for package “pear.php.net/net_smtp”
install failed

Leaving it overnight before rolling up my sleeves (in case it was just downtime at the package repository) the fix was fairly simple.

The Fix

If you are experiencing the same problem this server was having, running the following

pear remote-list

Will result in

SECURITY ERROR: Will not write to C:\Users\<user[8.3]>\AppData\Local\Temp\pear\cache\e9b88593398eb79a9aa91024351d646arest.cacheid as it is symlinked to C:\Users\<user>\AppData\Local\Temp\pear\cache\e9b88593398eb79a9aa91024351d646arest.cacheid – Possible symlink attack

If you get something akin to the above simply browse to:

C:\Users\<user>\AppData\Local\Temp\

and delete the pear folder

Visual Basic 6.0 Error: ‘Compile Error: Procedure declaration does not match description of event or procedure having the same name’

System Requirements:

  • Visual Basic 6.0

The Problem:

After copying a control placed on a form in the VB 6.0 IDE you receive the following error message when you attempt to run the program.

Compile error:

Procedure declaration does not match description of event of procedure having the same name

Compile Error Message

The program will not compile correctly even after the VB control that you copied has been deleted from the form.

More Info

This seems to be a bug in the VB 6.0 IDE. Remember this?

Create Control Array Message

It asked you if you wanted to place the control into an array so that multiple controls named ‘Command1’ in this case could be referenced from a common name i.e. Command1(0), Command1(1), Command1(2) etc.

Very occasionally the VB IDE gets this wrong, attempting to reference a control variable as an array variable. This in turn leads to non-compliable code and no clear way to get out of the problem.

The Fix

The fix is to prevent the VB IDE from trying to treat the original variable as an array by removing its index parameter.

  1. Delete or rename the duplicate named control
  2. Note the name of the control in the error message
  3. Save your project and exit VB
  4. Open the location of your .vpb (Visual Basic Project) file
  5. Locate the .frm file (form) that your control appears in amongst the list of files
  6. Open notepad and drag and drop the .frm file into its window so that you can see the underlying code-behind the form
  7. Search the file for the name of the control named in the error message
  8. Beneath it you should find a parameter list associated with the control. Locate and delete the line “Index” line as highlighted below
  9. Save and exit notepad and restart the IDE

Or, if the control exposes it, clear the value of ‘index’ on the properties explorer in the IDE.

The Fix in Notepad