Exploration of which request headers you need, should and cannot set when using MsXml2.XmlHttp, Microsoft.XmlHttp and MsXml2.ServerXmlHttp

System Requirements:

  • Windows NT 4.0 Server SP4+
  • Windows Server 2000
  • Windows Server 2003
  • Windows Server 2008, R2
  • Windows Server 2012, R2
  • Windows 2000 Professional
  • Windows XP
  • Windows Vista
  • Windows 7
  • Windows 8, 8.1
  • IIS 4.0, 5.0, 5.1, 6.0, 7.0, 7.5, 8.0
  • ASP 3.0 (Classic)
  • VBScript

The Problem:

I was browsing around the web earlier looking for inspiration on making PowerShell send asynchronous XmlHttp requests when I came across a block of VBScript examples. In them, I observed that the coder was religiously attempting to set the Content-Length request header, something that when using MsXml I don’t usually do – it’s a different story with PHP.

Equally, I observed that the programmer was using Len() to define the length of the content body for text, but ASP usually operates using an internal UTF-16 encoding to represent strings, thus does it follow that Len() will give the correct value?

So I thought that I would explore the issue to see for my own amusement, what was actually going on.

More Info

Let us take a standard code block to use for our tests and use the services of the good people at httpbin.org for a public message echo system.

Option Explicit
Dim x
Dim strRequest : strRequest = "a=1&b=2&c=3" ' <- 11 charactersset x = CreateObject("MsXml2.XmlHttp")
call x.open("POST", "http://httpbin.org/post", false)
call x.setRequestHeader("Content-Length", 11)
call x.setRequestHeader("Accept-Language", "EN-GB")
call x.setRequestHeader("Connection", "keep-alive")x.send(strRequest)

The above gives us a boiler plate for what we believe the correct header should looklike written in VBScript. When executed, HttpBin responds with:

{
"args": {},
"data": "a=1&b=2&c=3",
"files": {},
"form": {},
"headers": {
"Accept": "*/*",
"Accept-Encoding": "gzip, deflate",
"Accept-Language": "EN-GB",
"Cache-Control": "no-cache",
"Connection": "close",
"Content-Length": "11",
"Host": "httpbin.org",
"User-Agent": "<omitted>",
"X-Request-Id": "<omitted>"
},
"json": null,
"origin": "<ommitted>",
"url": "http://httpbin.org/post"
}

HttpBin provides us with an echo of the header and WireShark provides us with a wire capture of the transport packet being sent between the XmlHttpResponse provider and the HttpBin server. With these two tools we can analyse the header states.

Content-Length

If you take out the line call x.setRequestHeader(“Content-Length”, 11) and do not specify a content-length, both HttpBin and Wireshark’s trace of the connection report Content-Length: 11\r\n, which is correct. So logically, this has been inserted by the XmlHttp parser during the request to .send().

So what happens if we attempt to override it. If we inject call x.setRequestHeader(“Content-Length”, 15), which is 4 characters too long, again both WireShark and HttpBin report

Content-Length: 11\r\n

The same occurs if we under-report the content length, i.e. 10 characters. Therefore it is safe to conclude that MsXml2.XmlHttp’s send() operator handles this for you and any processing of the Len() of the message body before sending is just wasting CPU time, memory and lines of code as it is ultimately an action that will be duplicated.

To further test the theory to see if it is required in down-level versions of MsXml, I repeated the experiment using HttpBin and Wireshark with the following COM objects:

Microsoft.XmlHttp
MsXml2.XmlHttp            <-which is logically v3.0
MsXml2.XmlHttp.3.0
MsXml2.XmlHttp.6.0
MsXml2.ServerXmlHttp      <-which is logically v3.0
MsXml2.ServerXmlHttp.3.0
MsXml2.ServerXmlHttp.6.0

None of them could be coerced to include an incorrectly sized Content-Length and all of them appended the correct 11 character content length on their own when no attempt was made to manually provide it.

We can therefore categorically conclude that with MSXML 3.0 and higher, you do not need to waste time calculating the content length.

A an aside, when dealing with character data, the use of Len() appears to be correct for UTF-8 encoding. LenB() should be used when passing in binary data i.e. ADODB.Stream data, FileSystemObject data etc.

Content-Type

If you do not manually specify a Content-Type header, all MSXML providers appear to transmit POST encoded data as

Content-Type: text/xml; charset="utf-8"\r\n

As we were sending POST data (i.e. a HTML form), this is incorrect as the content-type should be “application/x-www-form-urlencoded; Charset=UTF-8”. Thus is is appropriate and necessary to include the Content-Type header manually as shown below.

call x.setRequestHeader("Content-Type", "application/x-www-form-urlencoded; Charset=UTF-8")

Accept-Language

Accept language was decidedly different depending on which provider you are using. You can safely override Accept-language by manually specifying it using setRequestHeader(“Accept-Language”, “EN-GB”).

Microsoft.XmlHttp         <- NOT SET
MsXml2.XmlHttp.3.0        <- NOT SET
MsXml2.XmlHttp.6.0        <- Derived it from OS i.e. Accept-Language: en-gb\r\n
MsXml2.ServerXmlHttp.3.0  <- Derived it from OS i.e. Accept-Language: en-gb\r\n
MsXml2.ServerXmlHttp.6.0  <- Derived it from OS i.e. Accept-Language: en-gb\r\n

Thus, if you are using the legacy Microsoft.XmlHttp or the client MsXml2.XmlHttp.3.0, it is necessary to manually enter the Accept-Language if of course it is needed.

Connection

All examples set the connection value to Connection: Keep-Alive\r\n when communicating with the HTTP/1.1 protocol. If you need to force the remote server to close the TCP connection immediately after the response has been received, you must manually set Connection: Close\r\n using setRequestHeader(“Connection”, “Close”).

Cache-Control

Cache-control is only set by the client providers and can be added manually to the server providers.

Microsoft.XmlHttp         <- Cache-Control: no-cache\r\n
MsXml2.XmlHttp.3.0        <- Cache-Control: no-cache\r\n
MsXml2.XmlHttp.6.0        <- Cache-Control: no-cache\r\n
MsXml2.ServerXmlHttp.3.0  <- NOT SET
MsXml2.ServerXmlHttp.6.0  <- NOT SET

Accept-Encoding

Accept-encoding is only set by the client providers and can be added manually to the server providers.

Microsoft.XmlHttp         <- Accept-Encoding: gzip, deflate\r\n
MsXml2.XmlHttp.3.0        <- Accept-Encoding: gzip, deflate\r\n
MsXml2.XmlHttp.6.0        <- Accept-Encoding: gzip, deflate\r\n
MsXml2.ServerXmlHttp.3.0  <- NOT SET
MsXml2.ServerXmlHttp.6.0  <- NOT SET

Accept

Accept is set to Accept: */*\r\n by all providers and can be overridden if required.

0x80090020 when attempting to load a .PFX Private Key Certificate into a CAPICOM_MEMORY_STORE using Store.Load() or Certificate.Load() using CAPICOM 2.1.0.2

System Requirements:

  • Windows Server 2008, R2
  • Windows Vista
  • Windows 7
  • Windows 8, 8.1
  • 7.0, 7.5, 8.0
  • ASP 3.0 (Classic)
  • CAPICOM 2.1.0.2

The Problem:

Ah encryption, that most noble of things. One thing that is sure to drive every developer close to the brink on the odd occasion. The one time where clear, concise API documentation should be considered mandatory – and the one place where good API documentation it seems is an obligation itself not to provide. Be it Microsoft, Java, BouncyCastle, PHP it would seem they are all blighted with the same issue.

Attempting to use legacy API on an unsupported platform should seem like an exercise in masochism, however, you know how much I like to avoid using .net whenever I can.

If you attempt to do this

Dim cert
set cert = Server.CreateObject("CAPICOM.Certificate")
call cert.load("c:\myPrivateKey.pfx", "test", CAPICOM_KEY_STORAGE_EXPORTABLE)

or this

const CAPICOM_MEMORY_STORE = 0
const CAPICOM_LOCAL_MACHINE_STORE = 1
const CAPICOM_STORE_OPEN_READ_WRITE = 1
const CAPICOM_KEY_STORAGE_EXPORTABLE = 1Dim store
set store = Server.CreateObject("CAPICOM.Store")
call store.Open(CAPICOM_MEMORY_STORE, "MemoryStore1", CAPICOM_STORE_OPEN_READ_WRITE)
call store.load("c:\myPrivateKey.pfx", "test", CAPICOM_KEY_STORAGE_EXPORTABLE)

you will get back

error '80090020'
/file.asp, line ###

If you send in a .cer file instead of a .pfx, it works without error but doesn’t allow you to access the Private Key.

More Info

Taking the two code samples in order

Dim cert
set cert = Server.CreateObject("CAPICOM.Certificate")
call cert.load("c:\myPrivateKey.pfx", "test", CAPICOM_KEY_STORAGE_EXPORTABLE)

Should you be getting a 0x80070056 error, your password is wrong. If the file doesn’t have a password, only send parameter 1 (which is about to cause you a problem). To resolve the 0x80090020 error while using a CAPICOM_MEMORY_STORE, you need to stop CAPICOM from attempting to insert the certificate as a resource for a user. If the IIS worker process that you are using doesn’t connect to a user account and has no permissions, the default parameter CAPICOM_CURRENT_USER_KEY or 0 will throw 0x80090020.

To change the scope, ensure that you use the fourth parameter and set the value to CAPICOM_LOCAL_MACHINE_KEY.

const CAPICOM_CURRENT_USER_KEY = 0
const CAPICOM_LOCAL_MACHINE_KEY = 1Dim cert
set cert = Server.CreateObject("CAPICOM.Certificate")
call cert.load("c:\myPrivateKey.pfx", "test", CAPICOM_KEY_STORAGE_EXPORTABLE, CAPICOM_LOCAL_MACHINE_KEY)

To resolve the second issue, modify the original code to make use of the now fixed certificate.load() call and import it vie the long route.

const CAPICOM_MEMORY_STORE = 0
const CAPICOM_LOCAL_MACHINE_STORE = 1
const CAPICOM_STORE_OPEN_READ_WRITE = 1
const CAPICOM_KEY_STORAGE_EXPORTABLE = 1Dim cert
Dim store
set store = Server.CreateObject("CAPICOM.Store")
call store.Open(CAPICOM_MEMORY_STORE, "MemoryStore1", CAPICOM_STORE_OPEN_READ_WRITE)set cert = Server.CreateObject("CAPICOM.Certificate")
call cert.load("c:\myPrivateKey.pfx", "test", CAPICOM_KEY_STORAGE_EXPORTABLE, CAPICOM_LOCAL_MACHINE_KEY)

call store.add(cert)

If you receive 0x80070005, you are either getting an Access Denied error to the MEMORY_STORE or you are attempting to import a certificate into the instantiated store which already exists. Similarly, if you receive 0x80070056, your password is wrong.

‘Bug’ in ASP 3.0 Application.Contents iterator causes undesired deletion patterns when Application.Contents.Remove() is called from within a for each / for loop

System Requirements:

  • Windows NT 4.0 Server SP4+
  • Windows Server 2000
  • Windows Server 2003
  • Windows Server 2008, R2
  • Windows Server 2012, R2
  • Windows Server 2016
  • Windows Server 2019
  • Windows 2000 Professional
  • Windows XP
  • Windows Vista
  • Windows 7
  • Windows 8, 8.1
  • windows 10
  • IIS 4.0, 5.0, 5.1, 6.0, 7.0, 7.5, 8.0, 10.0
  • ASP 3.0 (Classic)

The Problem:

I remember, long ago in approximately 2001 – in my less competent days – fighting to make something work and ultimately concluded that it was a hapless endeavour and ultimately went about it in a different way. The task was to clear down all but a small number of elements from the ASP 3.0 Application.Contents object.

What I concluded then, is something that I’ve only just re-remembered now after finally making a determined effort to hunt down a bug in a module on HPC:Factor – which is being used elsewhere – and in which a recent change brought the issue back to light.

There is an iterator issue come bug (depending on your point of view) in the ASP 3.0 Application object.

More Info

We’ll lead by example with this one. After spending an hour or so reacquainting myself with the problem while fixing (read making more robust) the HPC:Factoor class module, a fairly simple process can be used to demonstrate it. Whether or not you see this as a natural feature, or a sincere bug is something that I’ll leave to you. There are always ways around this sort of thing, so I guess that what counts is whether you think it should be fixed in the iterator or by the end user.

Take the following code

Option Explicit
Dim strKey
Application.Contents.RemoveAll()
Application.Contents("one") = "a"
Application.Contents("two") = "b"
Application.Contents("three") = "c"
Application.Contents("four") = "d"
for each strKey in Application.Contents
  Response.Write strKey & " == " & Application.Contents(strKey) & "<br />"
next

It’s obviously going to print out the following

one == a two == b three == c four == d

So what if we now do this:

for each strKey in Application.Contents
  Application.Contents.Remove(strKey)
next
for each strKey in Application.Contents
  Response.Write strKey & " == " & Application.Contents(strKey) & "<br />"
next

Clearly it should print

For dramatic effect, that’s “absolutely nothing being printed”. The application object should be completely empty.

Wrong! It prints:

two == b
four == d

What’s going on is quite simple. The for each iterator being called from the Application.Contents collection is indexed, in other words when items are added or removed they are given a numeric, integer based index in order to aid lookup.

This index becomes stateful as it initially exists at call time for the “for each” provider and its content is copied out to the iterator, By Value (ByVal). It should really be passed out By Reference (ByRef) i.e. via a Pointer.

What this means (using comments to explain the process) is that the following logic occurs:

Option Explicit
Dim strKey
Application.Contents.RemoveAll()            ' Delete all indexes, release pointers to all data
Application.Contents("one") = "a"           ' Create Index 1, Key:"One", Value:"a"
Application.Contents("two") = "b"           ' Create Index 2, Key:"two", Value:"b"
Application.Contents("three") = "c"         ' Create Index 3, Key:"three", Value:"c"
Application.Contents("four") = "d"          ' Create Index 4, Key:"four", Value:"d"
' Application.Contents.Count = 4

for each strKey in Application.Contents     ' Create an iterator of the index [1 - 4]
  ' Iterator Index i = 1
  Application.Contents.Remove(strKey)     ' Remove item at index 1
  ' Index 1 removed, compact index
  ' Index 1, Key:"two", Value:"b"
  ' Index 2, Key:"three", Value:"c"
  ' Index 3, Key:"four", Value:"d"
  ' Application.Contents.Count = 3
  ' Move to NEXT
  ' Iterator Index = 2 (i = (i + 1))
  Application.Contents.Remove(strKey)     ' Remove item at index 2
  ' Index 2 removed, compact index
  ' Index 1, Key:"two", Value:"b"
  ' Index 2, Key:"four", Value:"d"
  ' Application.Contents.Count = 2
  ' Move to NEXT
  ' Iterator Index = 3 (i = (i + 1))
  ' 3 is greater than 2 (the index is > count), exit

The problem is that the Index is being compacted on a successful call to .Remove(). The count of the number of items in Application.Contents is being updated to reflect the correct number of items, but the iterator isn’t being told i = (i – 1) after the successful completion of the Remove() method.

The same thing happens if you use “for” rather than “for each”:

Option Explicit
Dim i
Application.Contents.RemoveAll()
Application.Contents("one") = "a"
Application.Contents("two") = "b"
Application.Contents("three") = "c"
Application.Contents("four") = "d"
for i = 1 to Application.Contents.Count
  Application.Contents.Remove(i)
next

This also results in data still remaining inside the Application Object due to the same error, except here we are directly calling the iteration number ourselves via i so we also get 2 and 4 left in the collection as with calling .Remove() from the “for each”.

If the Application.Contents.Remove method supported a success/failure return type – for example a boolean true for item removed and boolean false for no such item in collection, then the fix would be simple:

for i = 1 to Application.Contents.Count
  if (Application.Contents.Remove(i)) then
    i = (i - i)
  end if
next

Sadly the method doesn’t support a return type.

The Fix

This bug means that there are only two ways to deal with it The first way would be to iterate across the collection, store the Keys in an array and then in a second pass remove all of the items that you want to delete by using an external array.

It does the job and allows you to continue to use keys, but why use two loops when you can use one? In the knowledge that the following is true:

  1. The index is compacting
  2. The iterator is not being reduced by 1 after a successful call to .Remove()

The second and simplest approach to solve the problem is to force the for loop to decrement it for you. In other words, reverse iterate instead of forward iterate through the collection.

Dim i

Application.Contents.RemoveAll()
Application.Contents("one") = 1
Application.Contents("two") = 2
Application.Contents("three") = 3
Application.Contents("four") = 4
for i = Application.Contents.Count to 1 step -1
  Application.Contents.Remove(i)
next

for i = 1 to Application.Contents.Count
  Response.Write Application.Contents.Key(i) & " == " & Application.Contents.Item(i) & "<br />"
next

By going backwards, the index is decremented and so is the external iterator, meaning that they keep in sync with each other.

To adapt this further, if you only want to remove certain items from the collection and want to delete based upon the key, use the following.

Dim i
Application.Contents.RemoveAll()
Application.Contents("one") = 1
Application.Contents("two") = 2
Application.Contents("three") = 3
Application.Contents("four") = 4
for i = Application.Contents.Count to 1 step -1
  if ((Application.Contents.Key(i) = "one") OR (Application.Contents.Key(i) = "three")) then
    Application.Contents.Remove(i)
  end if
next

for i = 1 to Application.Contents.Count
  Response.Write Application.Contents.Key(i) & " == " & Application.Contents.Item(i) & "<br />"
next

Using our example above, the output will correctly be:

two == b
four == d

“Cannot connect to Primary Server” from Mac OS X clients running Sophos Anti-Virus 8.x after the Enterprise Console has been updated to distribute Anti-Virus 8/9 for OS X via a HTTP connection served through IIS 6, 7, 7.5, 8

System Requirements:

  • Sophos Enterprise Console 5.2.1
  • Mac OS X 10.4+

The Problem:

If you have been deploying Sophos Enterprise Console 5 to update Mac OS X using a HTTP connection, end devices may start refusing to connect to the Primary server. The Sophos Anti-Virus 8 log will state that it was unable to connect to the server.

Windows clients can update successfully from the same server over HTTP.

More Info

In this particular case, the HTTP distribution point was backed from a public facing web server running IIS 7.5 under Windows Server 2008 R2. The server has an encrypted tunnel connection and read-only pass through authentication to a SMB share on an internal security server running Windows Server 2008 SP2 and Sophos Enterprise Console 5.2.1.

The SMB share is the same as the one exposed to the internal LAN for direct SMB updating and Windows clients using either HTTP or SMB will update successfully.

Mac OS X 10.7, 10.8 or 10.9 clients were however locked at Anti-Virus 8.0.5 and were only receiving definition updates from the secondary server at Sophos. The client will connect to the Primary HTTP share and begin to download, but terminate after a few kilobytes have been transferred.

The Fix

Somewhere between 8.0.5 and the current 9.0.5, there would appear to have been a change in the internal structure of the Sophos Anti-virus .mpkg.

Enable directory browsing and test that you can open the following file:

/SophosUpdate/CIDs/S000/ESCOSX/Sophos Anti-Virus.mpkg/Contents/_CodeSignature/CodeDirectory

Note that CodeDirectory is a file and not a folder. This is the root of the problem.

Under the default metabase configuration in IIS, IIS will only serve files with known extensions and registered MIME types. So what happens with file with no file extension? By default a 404 error.

Under IIS 6

  1. Right click your servicing web site in IIS manager and choose Properties
  2. Go to the HTTP Headers tab and choose MIME Types
  3. Click New…
  4. In Extension type:
    .*
  5. In MIME type type:
    application/octet-stream
  6. Commit all changes

Under IIS 7

  1. Select the servicing web site in IIS manager
  2. Choose MIME Types under the IIS heading
  3. Click Add… in the top right
  4. In File name extension type:
    .*
  5. In MIME type type:
    application/octet-stream

Under IIS 7.5 / 8

  1. Select the servicing web site in IIS manager
  2. Choose MIME Types under the IIS heading
  3. Click Add… in the top right
  4. In File name extension type:
    .
  5. In MIME type type:
    application/octet-stream

Under IIS 7, 7.5, 8 you may also need to add the following to your root Web.config

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<system.webServer>
<security>
<requestFiltering>
<fileExtensions allowUnlisted="true" />
</requestFiltering>
</security>
</system.webServer>
</configuration>