ExchangeDefender

Throttling Session Concurrency with SBS / Exchange 2003

I have really slacked off when it comes to my Technical Articles but I have to say that last two weeks of direct involvement with ExchangeDefender and fighting spammers has really reinvigorated my passion for helping people deal with this scum. So, what’s the problem?

Having any email troubles as of late? Email messages delayed? Coming in out of sequence or with a large delay? How about peers asking you why you haven’t responded to an email they sent you (and you explaining that you’ve just never seen it?)

If that sounds familiar, you have been a victim of a latest botnet, which whether intentionally or inconsequently, launched a large scale DDoS attack against many mail servers on the Internet.

This post shows you how to throttle down session concurrency with SBS / Exchange 2003.

I would like to thank my good friend Karl Palachuk for being a true leader and writing a series of blog posts on managed services that really encouraged me to start sharing my technical expertise with my audience. I hope you guys enjoy it.

Note: I wrote the post completely and totally drunk. I hope that is enough to excuse any typographic errors.

Friends on Rough Days

Today was a rough day. Even though the trouble lasted only about 40 minutes and affected just a handful of people its yet another smack in the face of the perfection that we strive for. Bah. I was out having lunch with Jen and bailed out early (thanks for picking up the check, you rock!) to realize I am not Google yet. Anyhow, I had this in the corner of my third screen all day long, it kept on looking at me while I was taking calls during the storm, updating the blog, checking on status reports, on the solution, looking what caused it, troubleshooted it with partners and so on. This was an IM I got at about 5 AM:

Morning V, before I got swamped again with work today I wanted to let you know that your last four posts rock - in particular the one about being mean and beloved and the one about Karl. I don't care if you think you can't write, you can, and you do it well. The best thing is that you write like you speak and so even though you are far away, I always have a little piece of Vlad on my desktop. :)

It’s good to have friends on days like today.. Now, off to Montana to breed sheep.

Changing Face of SPAM Filtering

August was a hard month for ExchangeDefender with a lot of long planned changes being moved up the timeline to address what I believe will be the end of unmanaged mail servers. Allow me to explain. Assuming you can deal with a mild annoyance and have an extremely powerful server you can pretty much set up your choice of a mail server software and leave it alone. As in Windows NT/2000 leave it alone – configuration OK, system up, job done.

Over the past few years the above case has only shifted slightly. Those that cared for productivity and efficiency of their resources bought SPAM filtering software, outsourced SPAM filtering, hired MSPs and ASPs to handle their mail but for the most part there was no big “threat” just a huge “annoyance” impacting productivity and server efficiency. For example, you can install an SBS box and leave it up and running on Exchange 2003 SP1 or SP2 without a care. Sure, it may take a minute or two to open up the server manager, you might be eating up the storage quota on the server by storing all sorts of junk, you could even have an infected system from time to time.. but the life goes on, IMF may help along a little bit, for the most part there is no need to watch over the server 24/7/365 and given a good backup and some failure tollerance you’re good to go.

I believe the good times are over.

I started ExchangeDefender years ago in order to protect our Exchange and Sendmail servers. Not really to make money but to save money from expensive Exchange AV solutions that  also ended up trashing our resources. Over the years I’ve seen the product evolve beyond the wall into a managed piece of network intelligence, business continuity with LiveArchive, compliance and regulatory tool and a heck of a lot more. But in the end, it will always be a central network that we use to isolate the threats before they get to the pieces that we cannot manage – users and servers.

Last month saw an interesting change in the way problems escalate out of proportion, almost one after another. First, we continue to see the increases in the amount of SPAM being relayed through. Nothing new there but the volume is important because it tasks the resources much harder making fewer cycles available for the more detailed policy enforcement, such as that of filtering image spam. You see, where most SPAM used to come in little tiny text messages that could easilly be filtered out with the lowest power appliance, the new SPAM comes in as a PDF, a zip file, an image file, etc. We are now not dealing with 1–2KB files, we’re now dealing with documents and images that are several hundred times on the order of magnitude. And as we lose more and more cycles to the garbage that seems to only be annoying us, we are letting more and more really dangerous stuff slip through.

Namely, over 25% of the computers on the Internet are pwn3d in a number of ways. They may have a rootkit, virus, trojan or even a part of a botnet. That is a huge part of the network that is really going to waste and it’s only going to get worse. For example, the issue with the latest botnet code that was likely inadvertently messed up. Instead of delivering SPAM it would open the connection and leave it hanging after issuing RCPT TO: command. The result is that thousands upon thousands of connections were left open, bringing the server down to its knees. Imagine 25% of the Internet doing something “weird” for a day or two.

It is easy to see how little issues like this can take down companies quite easilly. I believe the days of set it and forget it are now firmly over. It, in part, is why we pushed up so many security features because we don’t want to have to solve the “problems” that haven’t appeared yet when it becomes a little too late.

SPAM Hoover: BBB Trojan

So here is what we’re fighting with today, Better Business Bureau fraud/trojan SPAM:

Complaint Case Number: C465B4
Complaint Made by Consumer Mrs. Marcia E. Worthington
Complaint Registered Against: Elizabeth MeHaffey of TIMCO Aviation Services, Inc.
Date: 05/14/2007/

Instructions on how to resolve this complaint as well as a copy of the original complaint can be obtained using the link below:

CLICK HERE TO DOWNLOAD AND VIEW DOCUMENTS FOR CASE #C465B4 <>

Disputes involving consumer products and/or services may be arbitrated. Unless they directly relate to the contract that is the basis of this dispute, the following claims will be considered for arbitration only if all parties agree in writing that the arbitrator may consider them:

*       Claims based on product liability;

*       Claims for personal injuries;

*       Claims that have been resolved by a previous court action, arbitration, or written agreement between the parties.


The decision as to whether your dispute or any part of it can be arbitrated rests solely with the BBB.

The BBB offers its members a binding arbitration service for disputes involving marketplace transactions. Arbitration is a convenient, civilized way to settle disputes quickly and fairly, without the costs associated with other legal options.

2007 Council of Better Business Bureaus, Inc. All Rights Reserved.

'You'll be openin' without him, then,' Parkins said, thinking that if the prices he had seen in the window were any indication, Straker wouldn't exactly be swamped with customers. 'What's Mr Barlow's first name, by the way?' He went out the back door and around the house to where he kept the big yellow bus parked. He felt tough and coldly competent. This was infiltration, just like the Army. 'Yes.' 'Well, there's Marie Boddin. I could walk-' 5 Pat Middler picked at a callus on his left palm with great interest.

In case you’re wondering, the link leads to an executable attachment.

All this, yet not a day goes by without someone begging us to let .exe’s through ExchangeDefender. This is why we don’t, it’s been out for over a day yet none of the 10 or so AV engines we use are currently detecting the attachment as anything dangerous. And if you’ve got more than one employee I guarantee there is at least one dumb enough to click on the link. Into the SPAM hoover with you.

Huge SPAM / DDoS-like Explosion

If you’re not so fortunate to have ExchangeDefender protecting your mail server you may have noticed your mail server is slowly choking in the junk that seems not only to be dumping massive amounts of data off but also significantly slowing everything else down..

Why?

Wel, there is a huge Trojan outbreak of Agent-GBX at the moment and it seems that the saturation is pretty remarcable. The throttling control on this beast is out of wack too, it seems to be opening so many connections that the connections hang and just sit in an open state until the server terminates them. So, throttle down your timeout values while you can still get to your ESM

ExchangeDefender v3.1 sneak peak

Ironically enough, this blog gets more traffic than our corporate web site so I’m very happy to share with you another great accomplishment: ExchangeDefender v3.1. Why happy? Because I feel this release pretty much crushes all of our competition in the SMB market when it comes to business continuity concerns. I’ve had the pleasure of being the DR headquarters for every hurricane, mud slide, terrorist attack, act of Bell South incompetence and then some – and the first thing that people want to get to when their stuff falls apart is email. Really, once we get our ducks in a row we think about our business, our customers, and how to contact them. Email, even so beyond web, is the most critical piece of communications infrastructure you’ve got.

So for years I went around and asked people what would help, what we could do, what would fit the budget. And well, a few years later, dirt cheap bandwidth, inexpensive servers and some ingenuity… ExchangeDefender now delivers live archive functionality that lets you read, respond, send and manage past 7 days of your email at any point. Server went down? Hit the live archive box at ExchangeDefender via web/ssl. Mailbox corrupt and Outlook or OWA won’t start for that user Chad? Hit the live archive box at ExchangeDefender via web/ssl. Entire office trailer fell off the cliff? You’ve guessed it, hit the live archive box at ExchangeDefender via web/ssl.

Whats nice about the implementation is how gentle it is. While the original message is still in stream (or in queue if the server is down) a copy of it is spooled to our live archive box. It works constantly, in background, in the cloud, and requires no action on the users part at all. If they can figure out Hotmail, they can figure out this. The second nice thing about this is that the archive is live – meaning it will keep on receiving messages even if your server is down. That means you can use the archive mail server to send/receive/respond to mail for days without anyone knowing what happened. Third thing is auditing – auth trail, timestamps, etc – full control over whats going on. No plain text password or username access… EVER. Email is in the ExchangeDefender cloud so its secure, encrypted..

Oh, I forgot a small detail. It’s friggin FREEEEEEEEEEEEEEEEEEEEE!

Download guide: ExchangeDefender.pdf (543 KB)

More about Microsoft PAL's

Now if the unofficial live announcement from WWPC and the VladCast #9 didn’t answer your questions here is the official (as in, “you can take that to the bank and tell them Ligman sent you”) hint on just who the PALs are, where they are along with the contact info.

The Microsoft PAL initiative recognizes and thanks outstanding Microsoft Small Business Specialist partners for their community participation and willingness to help other Microsoft partners who serve entrepreneurial and midsized customers. The PAL designation is given to exceptional Microsoft’s Small Business Specialists who lead and foster the involvement of other Microsoft partners in their respective communities around implementing best business practices and supporting customers. The PAL designation recognizes the most active Small Business Specialists from around the world who provide invaluable online and offline expertise that enriches the Microsoft partner community focused on delivering Microsoft’s small and midsized business products.

Click here to read more from mssmallbiz.com…

On behalf of the ExchangeDefender team I’d like to thank Eric for helping spam those folks to death by publishing their email address. Only four on the list are current customers/partners and every bit of junk mail helps bring people on. Thanks dude!  

The Joy of SPAM

So I’m sitting around this morning finishing up some bugfixes for ExchangeDefender v3.1 and I took a moment to look at yesterdays stats. So far so good for a Monday. Attachment filtering breakdown. Whooa, Microsoft Excel (.xls) in #2 spot? What the…

“Oh yeah. We’ve been seeing a huge spike in the number of forged Excel files doing the same stock pattern spam that we were dealing with a few weeks ago with pdf’s and greeting cards.”

Fun. Another attachment that can’t be trusted anymore.

Interesting weekend ahead: Big Infrastructure

Every now and then poor planning and an incredible growth curve meet to make something special. We’re having a special maintenance cycle this weekend at OwnWebNow and rather than to test our luck I’ve extended the maintenance cycle by 5 hours to make sure everything thats being brought online is done in a manner that won’t cause me to have to apologize for a week straight.

Absolutely every single core service is being scaled up, along with both physical and topological change in the infrastructure: new DNS server core, three more data centers.

Tip for newbies: never be a smartass in an engineering meeting. For example, if you hear someone say the above to you and you are compelled to say “No biggie, whats the worst thing that could happen” prepare for everyone else to pause and look at you, visually painting “fault point” on your forehead. On the flip side, its a hell of a way to get a Saturday off

Why can't I email zipped executables like all the worms and viruses do?

The #1 source of frustration and anger with ExchangeDefender is directly related to the way it handles executable content – with a shotgun. Through the years, we’ve seen executable email content damage so many computers and networks that we outright ban that content at the gate, nothing executable comes in or goes out. We go a few steps further too – You can’t zip it, you can’t rename it, you can’t rename it and zip it. We eval the file not just for the filename pattern but for the filetype MIME match as well. Then we check if it has multiple extensions trying to hide the .exe at the end. Then we check the length of the filename, then… Well, let’s just say we’re thorough.

Yet every Monday morning there are the unfortunate few customers that will have an urgent, critical business need to receive an executable file. The reason? You’ve guessed it:

My vendor distributes the software updates through email.

While ordinary ExchangeDefender users have a right and even an excuse to be ignorant of this, software vendors do not. Let me explain the ginormity (new word of the day) of this stupidity in somewhat plainer terms. Your software vendor is trying to behave like 98.3% of all worms and viruses out there. And after years and years of “Here is your Microsoft update, attached.” exploits there are still people under the rock about this. Here is how we like to translate your business need to patch applications that only receive updates through email:

My pharmaceutical distributor only sells this drug in a dark alley between 2 and 6 am.

My butcher will only sell that piece of meat after hours, and always locks the door after I come in.

My electronics dealer has the cheapest and newest stuff around, but only seems to have one piece of each model. He also only takes cash payments and asks me if I’ve seen any cops around on my way in.

Please, end this stupidity. There is no circumstance, whatsoever, under which we will ever allow executable content that we cannot validate through our network. Even if you’ve spent last 260 years in network security, have written books on it, have a pending Nobel prize in World Peace category for helping save starving African villages through your ability to keep orphanages running with software that only distributes patches via executable email attachments… even then we will block your ability to receive refuse to hand you a loaded gun to play Russian roulette with.

In closing, if this software vendor absolutely cannot deliver their software through anything but email…. Go get a gmail, yahoo, hotmail, etc account and have your patches sent there. Then download those directly onto the box running the said software, but ask yourself this:

If this software vendor has put so little thought into the distribution of their patch management and cannot even figure out how to put up an authenticated web page for their file distribution, which is a job of a jr system administrator…. what other holes have they left in their application if they can’t grasp these simple concepts?

The preceeding Monday morning rant is brought to you curtesy of Own Web Now Corp support.