Throttling Session Concurrency With Exchange 2003

As of late there has been a drastic increase among complaints about not receiving all the email sent to a particular Exchange 2003 server. Most of these complaints come from servers that do not have adequate protection on the perimeter. By now almost all businesses have some sort of a good perimeter protection be it in house, such as SPAM appliances or Microsoft ISA or appliance firewalls or from cloud security services such as ExchangeDefender that limit the number of concurrent sessions passed through to the end user.*

Many places that do not have any protection at all are noticing a huge increase in the amount of messages that are arriving either late or out of sequence or are just never arriving at all. How does this happen? Well, it’s called a Distributed Denial of Service (DDoS) attack and sometimes its planned by a hacker, sometimes its just accidental. At it’s core, Distributed Denial of Service attacks  are designed to launch a large amount of connections at a computer to exhaust its resources and make it impossible for that computer to fulfill legitimate requests. For our particular example, the target SMTP server (Exchange 2003) gets an abnormally high number of simultaneous connections so that legitimate connections have no way of getting through.

Now that you know the basics let’s s see if you’re affected. To find out your current connection count, that is: to find the number of computers currently connected to your Exchange 2003 server, click on Start, All Programs, Microsoft Exchange, Exchange System Manager.

You will see the screen below showcasing your Exchange 2003 system. Expand your Servers, Expand your server, Expand Protocols, SMTP, Default SMTP Virtual Server. Click on Current Sessions. Illustrated below:

The server described above is protected by ExchangeDefender so it only has a handful of connections that are actively delivering mail to it. If this screen shows pages and pages and pages of “Users” in a connected state you may be exhausting the resources of your server and jeopardizing your ability to allow legitimate mail senders to deliver messages to your server.

What is the precise limit?

The exact limit to the number of SMTP connections on a stock Exchange 2003 server (or Small Business Server 2003) is 500 connections. You can increase or decrease the number of connections but the default is 500 connections.

Another default to keep an eye on is the Connection time-out, set to 10 minutes by default (600 seconds). Although the RFC only calls for 5 minutes (300 seconds) Microsoft has chosen to give Exchange 2003 SMTP server some leeway.

What is the problem?

Lately people have started complaining about not receiving email, not receiving it on time or in correct sequence. To address that complaint we need to look at the recent cause for those issues. Recently a large spamming botnet malfunctioned. Instead of connecting and delivering a piece of SPAM, the botnet host would connect and hang after issuing the helo statement. This host would then wait until the connection timed out.

Let’s do some math. Exchange limits concurrent connections to 500 simultaneous SMTP connections. It times them out after 600 seconds. So if a botnet launched 50 connections a minute, all your resources would be exhausted by the 10th minute. Meaning, no valid mail could get through. Of course, those connections would eventually time out but the point is that the SPAM botnets would exhaust your connection limit fairly quickly and without affecting the resources on your server (trust me on this, hundreds of open connections that do nothing would not show up in your regular CPU/network graphs because nothing is being transfered or processed).

Your server would run out of connections before it got a chance to expire these “ghost” connections that are just hanging off your server.

* Note: This is where networks like Postini fail to protect your servers from DDoS because they open a direct connection to your server for every connection they accept on their end.

How do we fix it?

First, identify if this is really a problem for your network. Change the limit of connections from 500 to 1,000. Keep an eye on it. If after a few minutes you see anywhere close to 500 (or even over 500 connections) in an open/hung state you are definitely not in control of your resources.

There are two ways to approach this problem. One is a fix, the other is a cautious avoidance of the solution. Allow me to explain:

1. Option one involves placing a higher number of connections on your server and opening up more resources to the Internet so you can receive more mail. This of course means more bandwidth, more resources and more processor going to waste (assuming the botnet actually delivers the message and your IMF/antivirus has to deal with the message). To do so make sure the box next to Limit number of connections to: is checked and pick an arbitrary large number.
2. Option two involves throttling down the Connection time-out (minutes) parameter to 5 minutes (300 seconds, RFC requirement) or even lower (not advised if you receive a lot of large attachments that can take more than five minutes to transfer).

The ultimate solution of course is making all this someone elses problem, but if the budget does not allow for more advanced protection this will help you weather the storm. It is also a good practice to check the Enable Logging and keep an eye on any excessive connections and timeouts.