IIS Server Security - IOException whit large request using certificates on IIS 6.0

Author

IOException whit large request using certificates on IIS 6.0

Martijn van Schie

2007-06-25, 7:21 am

I'm working on a webservice that excepts a batchfile as a parameter.
When i do a post to this webserver using unsecure http connection, i works
fine.

Now when i configure IIS for the production enviroment i get an error when
the request is to large (We have not actually tested the threshold, but our
"large" file is about 800kb).
The configuration is as follows:

- SSL Enabled;
- Accept client certificate;
- Certificate mapping configured;
- ASP.NET configured for impersonation of the mapped account.

The exception we get is:

System.Net.WebException: The underlying connection was closed: An unexpected
error occurred on a send. ---> System.IO.IOException: Unable to write data
to the transport connection: An established connection was aborted by the
software in your host machine. ---> System.Net.Sockets.SocketException: An
established connection was aborted by the software in your host machine
at System.Net.Sockets.Socket.MultipleSend(BufferOffsetSize[] buffers,
SocketFlags socketFlags)
at System.Net.Sockets.NetworkStream.MultipleWrite(BufferOffsetSize[]
buffers)
--- End of inner exception stack trace ---

We also tested the same configuration with a smaller file and this work fine
(small meaning about 100kb).

Another thing is that the exception occured when the iis timeout is reached.
Changing this timeout to a higher value only extend the time before the
exception occured. We have raised it to 4 minutes, although the same
request, using the larger file takes about 10 second using HTTP.

Regards,
Martijn van Schie

David Wang

2007-06-25, 7:20 pm

On Jun 25, 4:18 am, "Martijn van Schie" <mvansc...@newsgroup.nospam>
wrote:
> I'm working on a webservice that excepts a batchfile as a parameter.
> When i do a post to this webserver using unsecure http connection, i works
> fine.
>
> Now when i configure IIS for the production enviroment i get an error when
> the request is to large (We have not actually tested the threshold, but our
> "large" file is about 800kb).
> The configuration is as follows:
>
> - SSL Enabled;
> - Accept client certificate;
> - Certificate mapping configured;
> - ASP.NET configured for impersonation of the mapped account.
>
> The exception we get is:
>
> System.Net.WebException: The underlying connection was closed: An unexpected
> error occurred on a send. ---> System.IO.IOException: Unable to write data
> to the transport connection: An established connection was aborted by the
> software in your host machine. ---> System.Net.Sockets.SocketException: An
> established connection was aborted by the software in your host machine
> at System.Net.Sockets.Socket.MultipleSend(BufferOffsetSize[] buffers,
> SocketFlags socketFlags)
> at System.Net.Sockets.NetworkStream.MultipleWrite(BufferOffsetSize[]
> buffers)
> --- End of inner exception stack trace ---
>
> We also tested the same configuration with a smaller file and this work fine
> (small meaning about 100kb).
>
> Another thing is that the exception occured when the iis timeout is reached.
> Changing this timeout to a higher value only extend the time before the
> exception occured. We have raised it to 4 minutes, although the same
> request, using the larger file takes about 10 second using HTTP.
>
> Regards,
> Martijn van Schie

Can you provide the corresponding log entry in IIS for this failed
request?

Sounds like a known limitation with SSL Client Certificate public
specification, but the IIS log entry will tell.

//David
http://w3-4u.blogspot.com
http://blogs.msdn.com/David.Wang
//

Martijn van Schie

2007-06-26, 7:17 am

David,

The result of the log gave me a clue and send me to this page:
http://technet2.microsoft.com/windo...3.mspx?mfr=true

I used it to set a new size (first the example value), as the request is
about 1MB (and can grow) i used 1.500.000 as a value.
The note below says that it not recommended for non client certificate
authentication, but although we are using this, am i using the correct fix
in the correct way.

This is the log from IIS for the specific call, before the value changed:

----
#Software: Microsoft Internet Information Services 6.0
#Version: 1.0
#Date: 2007-06-26 07:20:14
#Fields: date time s-sitename s-ip cs-method cs-uri-stem cs-uri-query s-port
cs-username c-ip cs(User-Agent) sc-status sc-substatus sc-win32-status
2007-06-26 07:26:24 W3SVC1159432196 129.227.40.132 POST
/sanws/sanservice.asmx - 81 - 192.39.234.162
Mozilla/4.0+(compatible;+MSIE+6.0;+MS+Web+Services+Client+Protocol+2.0.50727.42)
413 0 0
2007-06-26 07:27:53 W3SVC1159432196 129.227.40.132 POST
/sanws/sanservice.asmx - 81 - 192.39.234.162
Mozilla/4.0+(compatible;+MSIE+6.0;+MS+Web+Services+Client+Protocol+2.0.50727.42)
413 0 0
----

Ofcourse the 413 helped me out here.
I didn't expect this to be logged in IIS because during debugging of the
receiving webservice i never got a breakpoint to hit, and i didn't get a
SOAPException on the calling application.

So the main question is: Is this the correct fix?

"David Wang" <w3.4you@gmail.com> wrote in message
news:1182813801.647872.83380@x35g2000prf.googlegroups.com...
> On Jun 25, 4:18 am, "Martijn van Schie" <mvansc...@newsgroup.nospam>
> wrote:
>
>
>
> Can you provide the corresponding log entry in IIS for this failed
> request?
>
> Sounds like a known limitation with SSL Client Certificate public
> specification, but the IIS log entry will tell.
>
>
> //David
> http://w3-4u.blogspot.com
> http://blogs.msdn.com/David.Wang
> //
>

David Wang

2007-06-27, 1:21 am

Yes, that is the only available workaround for getting 413 with SSL
Client Certificates that I was alluding to earlier - increase the size
of UploadReadAheadSize. It is really a problem caused by the SSL
Specification.

Not getting a breakpoint in a managed code WebService hardly means the
request did not arrive at IIS. There's a bunch of native code, whose
execution is logged, which execute way before managed code is even
involved, WebService invoked, and breakpoints triggered... so always
check for log files. :-)

//David
http://w3-4u.blogspot.com
http://blogs.msdn.com/David.Wang
//

On Jun 26, 1:02 am, "Martijn van Schie" <mvansc...@newsgroup.nospam>
wrote:
> David,
>
> The result of the log gave me a clue and send me to this page:http://tech=
net2.microsoft.com/windowsserver/en/library/0ef55842-24d2-...
>
> I used it to set a new size (first the example value), as the request is
> about 1MB (and can grow) i used 1.500.000 as a value.
> The note below says that it not recommended for non client certificate
> authentication, but although we are using this, am i using the correct fix
> in the correct way.
>
> This is the log from IIS for the specific call, before the value changed:
>
> ----
> #Software: Microsoft Internet Information Services 6.0
> #Version: 1.0
> #Date: 2007-06-26 07:20:14
> #Fields: date time s-sitename s-ip cs-method cs-uri-stem cs-uri-query s-p=
ort
> cs-username c-ip cs(User-Agent) sc-status sc-substatus sc-win32-status
> 2007-06-26 07:26:24 W3SVC1159432196 129.227.40.132 POST
> /sanws/sanservice.asmx - 81 - 192.39.234.162
> Mozilla/4.0+(compatible;+MSIE+6.0;+MS+Web+Services+Client+Protocol+2.0.50=
72=AD7.42)
> 413 0 0
> 2007-06-26 07:27:53 W3SVC1159432196 129.227.40.132 POST
> /sanws/sanservice.asmx - 81 - 192.39.234.162
> Mozilla/4.0+(compatible;+MSIE+6.0;+MS+Web+Services+Client+Protocol+2.0.50=
72=AD7.42)
> 413 0 0
> ----
>
> Ofcourse the 413 helped me out here.
> I didn't expect this to be logged in IIS because during debugging of the
> receiving webservice i never got a breakpoint to hit, and i didn't get a
> SOAPException on the calling application.
>
> So the main question is: Is this the correct fix?
>
> "David Wang" <w3.4...@gmail.com> wrote in message
>
> news:1182813801.647872.83380@x35g2000prf.googlegroups.com...
>
>
>
>
>
>
>
the[vbcol=seagreen]
s,[vbcol=seagreen]
>
>
>
>
>
>
>
> - Show quoted text -

Martijn van Schie

2007-06-27, 7:22 am

Your right offcourse ... I should have known better ;).
Thank for the hint.

Only two questions remain:

1) Do i need actually need to set the UploadReadAheadSize higher then the
expected request size.
2) Is this issue fixed in IIS 7 (I'm have Longhorn beta3 installed on a
virtual machine at the moment, so i could test it).

regards,
martijn

"David Wang" <w3.4you@gmail.com> wrote in message
news:1182917436.520897.305980@n2g2000hse.googlegroups.com...
Yes, that is the only available workaround for getting 413 with SSL
Client Certificates that I was alluding to earlier - increase the size
of UploadReadAheadSize. It is really a problem caused by the SSL
Specification.

Not getting a breakpoint in a managed code WebService hardly means the
request did not arrive at IIS. There's a bunch of native code, whose
execution is logged, which execute way before managed code is even
involved, WebService invoked, and breakpoints triggered... so always
check for log files. :-)

//David
http://w3-4u.blogspot.com
http://blogs.msdn.com/David.Wang
//

On Jun 26, 1:02 am, "Martijn van Schie" <mvansc...@newsgroup.nospam>
wrote:
> David,
>
> The result of the log gave me a clue and send me to this
> page:http://technet2.microsoft.com/windo...f55842-24d2-...
>
> I used it to set a new size (first the example value), as the request is
> about 1MB (and can grow) i used 1.500.000 as a value.
> The note below says that it not recommended for non client certificate
> authentication, but although we are using this, am i using the correct fix
> in the correct way.
>
> This is the log from IIS for the specific call, before the value changed:
>
> ----
> #Software: Microsoft Internet Information Services 6.0
> #Version: 1.0
> #Date: 2007-06-26 07:20:14
> #Fields: date time s-sitename s-ip cs-method cs-uri-stem cs-uri-query
> s-port
> cs-username c-ip cs(User-Agent) sc-status sc-substatus sc-win32-status
> 2007-06-26 07:26:24 W3SVC1159432196 129.227.40.132 POST
> /sanws/sanservice.asmx - 81 - 192.39.234.162
> Mozilla/4.0+(compatible;+MSIE+6.0;+MS+Web+Services+Client+Protocol+2.0.5072_7.42)
> 413 0 0
> 2007-06-26 07:27:53 W3SVC1159432196 129.227.40.132 POST
> /sanws/sanservice.asmx - 81 - 192.39.234.162
> Mozilla/4.0+(compatible;+MSIE+6.0;+MS+Web+Services+Client+Protocol+2.0.5072_7.42)
> 413 0 0
> ----
>
> Ofcourse the 413 helped me out here.
> I didn't expect this to be logged in IIS because during debugging of the
> receiving webservice i never got a breakpoint to hit, and i didn't get a
> SOAPException on the calling application.
>
> So the main question is: Is this the correct fix?
>
> "David Wang" <w3.4...@gmail.com> wrote in message
>
> news:1182813801.647872.83380@x35g2000prf.googlegroups.com...
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> - Show quoted text -

David Wang

2007-06-27, 7:22 am

1=2E Yes
2=2E No

There's really nothing to fix.

The underly problem is that IIS has to read in enough of the SSL blob
to figure out the client certificate (it has to do the client cert
mapping, after all), and there is no guarantee that certificate is in
the first X bytes read by IIS, controlled by W3SVC/
UploadReadAheadSize. You can keep bumping X upward to increase the
likelihood that the certificate is completely buffered, but the same
buffering becomes a security risk since attacker can take advantage of
buffering to mount a DoS attack.

Similar behavior exists in normal HTTP traffic. Some authentication
protocols can have authorization tickets so large that they exceed the
limit of Y bytes that IIS reads to parse as request header. There is
no guarantee that the header fit completely in the first Y bytes read
by IIS, so while you can keep bumping Y upward to increase the
likelihood that the header containing the authorization ticket is
completely read in by IIS, the buffering becomes the same DoS security
risk as above.

IIS comes with secure defaults for those buffers that should work for
most people, but there are always users who need to tweak the setting,
and they need to do so while understanding the rationale and
consequences.

//David
http://w3-4u.blogspot.com
http://blogs.msdn.com/David.Wang
//

On Jun 26, 11:44 pm, "Martijn van Schie" <mvansc...@newsgroup.nospam>
wrote:
> Your right offcourse ... I should have known better ;).
> Thank for the hint.
>
> Only two questions remain:
>
> 1) Do i need actually need to set the UploadReadAheadSize higher then the
> expected request size.
> 2) Is this issue fixed in IIS 7 (I'm have Longhorn beta3 installed on a
> virtual machine at the moment, so i could test it).
>
> regards,
> martijn
>
> "David Wang" <w3.4...@gmail.com> wrote in message
>
> news:1182917436.520897.305980@n2g2000hse.googlegroups.com...
> Yes, that is the only available workaround for getting 413 with SSL
> Client Certificates that I was alluding to earlier - increase the size
> of UploadReadAheadSize. It is really a problem caused by the SSL
> Specification.
>
> Not getting a breakpoint in a managed code WebService hardly means the
> request did not arrive at IIS. There's a bunch of native code, whose
> execution is logged, which execute way before managed code is even
> involved, WebService invoked, and breakpoints triggered... so always
> check for log files. :-)
>
> //Davidhttp://w3-4u.blogspot.comhttp://blogs.msdn.com/David.Wang
> //
>
> On Jun 26, 1:02 am, "Martijn van Schie" <mvansc...@newsgroup.nospam>
> wrote:
>
>
>
>
d2-...[vbcol=seagreen]
>
fix[vbcol=seagreen]
>
d:[vbcol=seagreen]
>
5072=AD=AD7.42)[vbcol=seagreen]
5072=AD=AD7.42)[vbcol=seagreen]
>
>
>
>
>
>
but[vbcol=seagreen]
>
>
>
on:[vbcol=seagreen]
ine[vbcol=seagreen]
e[][vbcol=seagreen]
>
ork[vbcol=seagreen]
>
the[vbcol=seagreen]
>
>
>
>
>
>
> - Show quoted text -

Martijn van Schie

2007-06-29, 1:22 pm

David,

Thank you for clearing this out for me. It makes more sense to me now.
I'll have to tell to the customer that we have to change this setting, and i
can come with a clear explanation now.

regards,
Martijn

"David Wang" <w3.4you@gmail.com> wrote in message
news:1182940825.769448.62210@i13g2000prf.googlegroups.com...
1. Yes
2. No

There's really nothing to fix.

The underly problem is that IIS has to read in enough of the SSL blob
to figure out the client certificate (it has to do the client cert
mapping, after all), and there is no guarantee that certificate is in
the first X bytes read by IIS, controlled by W3SVC/
UploadReadAheadSize. You can keep bumping X upward to increase the
likelihood that the certificate is completely buffered, but the same
buffering becomes a security risk since attacker can take advantage of
buffering to mount a DoS attack.

Similar behavior exists in normal HTTP traffic. Some authentication
protocols can have authorization tickets so large that they exceed the
limit of Y bytes that IIS reads to parse as request header. There is
no guarantee that the header fit completely in the first Y bytes read
by IIS, so while you can keep bumping Y upward to increase the
likelihood that the header containing the authorization ticket is
completely read in by IIS, the buffering becomes the same DoS security
risk as above.

IIS comes with secure defaults for those buffers that should work for
most people, but there are always users who need to tweak the setting,
and they need to do so while understanding the rationale and
consequences.

//David
http://w3-4u.blogspot.com
http://blogs.msdn.com/David.Wang
//

On Jun 26, 11:44 pm, "Martijn van Schie" <mvansc...@newsgroup.nospam>
wrote:
> Your right offcourse ... I should have known better ;).
> Thank for the hint.
>
> Only two questions remain:
>
> 1) Do i need actually need to set the UploadReadAheadSize higher then the
> expected request size.
> 2) Is this issue fixed in IIS 7 (I'm have Longhorn beta3 installed on a
> virtual machine at the moment, so i could test it).
>
> regards,
> martijn
>
> "David Wang" <w3.4...@gmail.com> wrote in message
>
> news:1182917436.520897.305980@n2g2000hse.googlegroups.com...
> Yes, that is the only available workaround for getting 413 with SSL
> Client Certificates that I was alluding to earlier - increase the size
> of UploadReadAheadSize. It is really a problem caused by the SSL
> Specification.
>
> Not getting a breakpoint in a managed code WebService hardly means the
> request did not arrive at IIS. There's a bunch of native code, whose
> execution is logged, which execute way before managed code is even
> involved, WebService invoked, and breakpoints triggered... so always
> check for log files. :-)
>
> //Davidhttp://w3-4u.blogspot.comhttp://blogs.msdn.com/David.Wang
> //
>
> On Jun 26, 1:02 am, "Martijn van Schie" <mvansc...@newsgroup.nospam>
> wrote:
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> - Show quoted text -