Slow POST attacks through Forefront TMG 2010

What exactly is a slow POST?
Well, basically a POST(over a persistent HTTP connection) may look like:

POST / HTTP/1.1
Host: www.mytmgsite.com
User-Agent: Mozilla
Connection: keep-alive
Content-Length: 1000
Content-Type: application/x-www-form-urlencoded
<post data>

For example there are 1000 bytes of POST data(POST request body).

If we want a slow POST we can send the POST body 1 byte at a time with every byte being sent at a certain interval.

The result?
A possible Layer 7 DDoS if we open multiple such connections from many clients simultaneously.

This attack is very well explained in the Layer 7 DDoS OWASP AppSec DC 2010 presentation [1](PDF format, slides).

What Forefront TMG 2010 does?
It does not so good.
Basically I managed to DoS a TMG VM running on an ESXi server in a certain configuration quite easily with the OWASP HTTP Post Tool [2]; the VM could have allocated more hardware resources, but I used the minimum hardware system requirements in order to not increase the attack scale too.

First of all TMG has some flood mitigation settings in place, including for HTTP [3].
However none applies to such an attack.
While they limit the number of connections usable for slow POSTs from a single client, this helps little in a DDoS; for example we can easily open 100 of slow POSTs from a single client, if we have 20000 of such clients….

If you use a web publishing rule by default TMG does not buffer the POST request in that it will forward it as is to the backend server and not wait to receive it completely first; not quite desired in this situation.
Also the timeout of a connection for the respective web listener is 1800 seconds; so we can easily have a 100 seconds interval between the sent single bytes.
Plus we can have by default any payload length for POST request.
Combining these means we can send a very slow POST that will reach the backend HTTP server.

Furthermore even if the back server finishes the opened TCP connection used by the slow POST, the persistent HTTP connection between TMG and the client will still be available waiting for another request [4].

Modify TMG’s behavior from the MMC
We can modify the timeout of a connection from the web listener’s properties [5] but this is kind of useless as we just need to adapt the interval between the single bytes sent to have TMG not to disconnect it for inactive use.

We can limit the allowed payload length for POST requests using the HTTP filter [6].
This relates to what is a legit POST payload length for your application.
Doing so TMG will start buffering the POST requests, meaning the back server will not see the connection anymore until TMG receives the POST request completely(I did no try with very large POST requests); so at least we will have TMG take the hit.
However TMG does no kill the connection looking at the Content-Length value, it will wait for the entire POST to be delivered before denying the connection; so we can still DoS TMG itself.

In my lab(ESXi) with a stock TMG configuration of web server publishing rule(just modified the IPS flood mitigation settings to allow me to open multiple connections from a single host; don’t have an army of bots in my lab), at about 16000 active connections TMG stopped forwarding requests to the back server; interestingly the back server handled the situation and could still serve clients.
If I remember correctly the logs o TMG shown a ProxyVmemAlloc3pSiz related error(Event ID: 31212, web memory pool that handles HTTP connections is low) [7].

What we have wanted TMG to do
We wanted it to close the connection when the entire POST request wasn't received within a specified interval of time; similar with what F5 Big-IP can do [8].
As far as I know this is not configurable from the GUI.

References
[1] Layer 7 DDoS
https://www.owasp.org/images/4/43/Layer_7_DDOS.pdf
[2] OWASP HTTP Post Tool
https://www.owasp.org/index.php/OWASP_HTTP_Post_Tool
[3]Setting flood mitigation connection limits
http://technet.microsoft.com/en-us/library/dd441028.aspx
[4] Wikipedia Hypertext Transfer Protocol - Persistent_connections
http://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol#Persistent_connections
[5] Modifying network objects
http://technet.microsoft.com/en-us/library/cc995078.aspx
[6] Configuring HTTP filtering
http://technet.microsoft.com/en-us/library/cc995081.aspx
[7] Eventid 31212 : ProxyVmemAlloc3pSize registry value calculation
http://blogs.technet.com/b/sooraj-sec/archive/2011/01/10/eventid-31212-proxyvmemalloc3psize-registry-value-calculation.aspx
[8] Mitigating Slow HTTP Post DDoS Attacks With iRules
http://devcentral.f5.com/Tutorials/TechTips/tabid/63/articleType/ArticleView/articleId/1086402/Mitigating-Slow-HTTP-Post-DDoS-Attacks-With-iRules.aspx

Comments (2) -

  • The built-in HTTP filter in ISA/TMG is not very powerful, but the underlying web filter API is very capable.

    This kind of thing is precisely why we made IsaScript.  You could define a script that saves the time when a POST starts, and a content filter that runs whenever bytes are received.  It could check the time interval from start, and this gets above a configured value, you could error out the connection.
    • Thanks for the reply.

      I always thought that ISA/TMG has a lot of potential waiting to be unleashed; never understood why MS did not added more features into it.

      And yes, with IsaScripts one can do a lot. Smile
      Heard only good things about it.

      Cheers!
      Adrian
Comments are closed