Vlad Mazek – Vladville Blog

With the first 72 hours running against delinquent accounts, the new routing engine performed remarkably well. In about an hour we’ll do a wide deployment across the entire network. It will take approximately 3 hours for the upgrades to be performed.

The good news:

1. Performance was off the charts. Always good when rolling out new software.
2. We were never the bottleneck. This was a significant area of doubt, even though we had all the numbers in the world, we never had a proven model over time. Here is essentially how it works: messages are flushed to a queue and delivered sequentially from each processing node. As the mail load increases throughout the day, is it due to multiple ExchangeDefender connections or is it due to the saturation of the link? Good news is, it’s not us. When tested over 300+ IP’s worldwide, when certain links showed slowdown, others went through just fine.
3. Network congestion or server overload? This is something we are generally not alerted to and something VARs rarely either know how to access or have permissions to view. Exchange 2007 does issue performance based errors but your weakling consumer firewalls do not – they just defer the connection or drop it outright.
4. DNS issues? This one was fun.. we pretty much DDoS’ed people 🙂 We found hosts who took forever to issue a banner – so we flooded them with SMTP connections. Then we started transfering 256K attachments, then 1 Mb attachments. Guess what? They flew!!! We are narrowing this down to two effects: 1) Problems with DNS servers. 2) Excessive RDNS or RBL lookups.

Problems with DNS servers are more difficult to isolate because they may be sporadic depending on their load. As most sites are not likely to run their own name servers or their own caching name servers, external lookups may take longer. Sites that ALWAYS had terrible initial greeting are very likely just using dead RBLs or way too many antispam measures – ALL of which need to be shut off.

So far we’re looking at the healthiest week on the network, despite DDoS and attacks as ususal. Let’s see what we can pull up when the entire network is actively managing connectivity to target servers.

I guess we will never learn if the support team could kick developer teams ass.

Earlier today we rolled out the new ExchangeDefender code to a select* group of users. This new software is intended to adjust connection timeouts during peak hours and log timeouts so it can be reported.

One of the more frustrating things we have is that a lot of people simply do not have a network with enough capacity to run a mail server. Without limits in place we find sites that at random times get congested and only accept the smallest of messages. Bigger pieces, particularly those with attachments, can get delayed from a few minutes to a few hours. This is a rare issue but it creates an insane amount of support requests.

Recently we started extending connection timeouts on per-IP basis but I’d rather throw alerts and advise clients to upgrade their connections (or suffer timeouts and delays).

Now if we get over 30 seconds of inactivity the message is flushed to a queue with a 90 second timeout. If it’s more than that, it will be switched to a queue with life of 180 second timeout.

Logging of this activity will help us plot the timeouts on per-host basis so we can have a model for each client that reports an error. It takes far too much time to go through each queue to inspect each message, now we can have full host reputation reporting to justify the recommendation they get more bandwidth.

Some experimental code ought to improve things. We’ll know tomorrow.

* “Hey wait a minute, I am not paying you to run experimental software on my business email.” – I know and I have great news, we’re only using this on the delinquent accounts for people that are more than 30 days out. 🙂 So if you complain, expect to talk to the billing person first 🙂

The official announcement is on our official blog. Please subscribe to that feed and join the other 8 people reading it on an annual basis. That’s the official story, however, in the interest of honesty to our partners I have to optimistically apologize for things taking to long to get to this point.

You see, as ExchangeDefender (and I) grew up, the product matured as well. I’ve always embraced the development cycle that primarily served 90% of our users, with lower priorities assigned to the needs of the admins, MSPs, IT royalty and power users. Why? Well, the ExchangeDefender service is first and foremost a protection mechanism for people that are not INFOSEC hobbyists. That’s the customer that spends the money on the service, that’s the person I need to secure and so long as I please them, we get to keep the client.

Always (always, always) take care of your customer first.

This is why ExchangeDefender has some of the highest retention rates around.

I think we failed to stay true to that mission in our objectives. Basically, we dictated some features for the sake of simplicity and cornered ourselves into a position where users have become so dependant on the logically broken process because they have conditioned to it, and will not consider something better.

The most hated (and most beloved, and most pleaded not to be removed) feature in ExchangeDefender are the daily reports. People loved this stuff in 2002! It provided archival data for the SPAM past 7 days, daily reminders and stats, ability to release SPAM if it ever gets in there, etc. Fast forward a few years and the report is stacked with some of the most vile stuff anyone has ever written. Laced with stuff that would trigger even the worst of antispam software. But users are conditioned to expect the report. They bitch when they don’t get it. Most don’t read it – it goes straight to trash.

It is in our (OWN) best interest for our users to better interact with the service because it can learn to protect them better. I think I said something in one of the interviews today regarding the value of client installed software for a pure off premise solution. We aren’t just fighting with external threats, we are fighting with our users stupidity and naivete. When stuff breaks the cost of the product skyrockets if everyone in the technology chain has to be involved in fishing or tracking the email through the system. We can do much better than that, but we need to offer something that lets the user take a chance on the new solutions.

So after months of hard work, ExchangeDefender Client Suite is out. We’ll add file sharing, web filtering and a few other things throughout 2009 because we are no longer able to protect the user with just the SMTP filtering alone. Much like we can’t offer them business continuity with only 5 days of retries – where LiveArchive thrives in the enterprise contracts and SMB shops alike.

I hope you enjoy it. And to the slimy ones in the audience, I hope you can make some profit off it. Differentiation demands premiums, pimp on!