The Worst Week of My Life

Last week was the worst week of my life. Much like the most depressing lines out of the movie Office Space: “Each day is worse than the previous one. So every day you see me, it is the worst day of my life.” I’ve blogged here about failure (extensively) in the past and how one should not be afraid of it – but the disheartening thing is that no bad news comes alone – it usually turns into an ugly streak. When this happens my only prerogative is to keep on working on finding a solution instead of giving up. Hard head is better than no head.

So what went wrong this week? How about a 40% increase in the network load combined with the 600% increase in sales? Biggest week on the books (ever) coming in during the week when you experience 40% increase in traffic is a blessing and a curse. On one hand, mo money. On the other hand, you’re stretched to the limits not just in terms of infrastructure and NOC upkeep, but you’re also spending time with service provisioning and sales management (for our purposes, service provisioning including capacity planning is a part of a sales process). So on the week in which I had planned to put in 20 hours or so to wrap up Shockey Monkey 2, I spent 40+ hours on ExchangeDefender, OWN’s infrastructure shipments to Australia and Europe, meetings and more contracts signatures than a mortgage process requires.

The Fun

This week we experienced 40% increase in the network activity. Why? Spammers got a new trick. They are bouncing mail off recipients with OOFs enabled. They also started actively abusing the loophole in rfc2821 which requires every MTA to accept null sender messages. Originally, this process was envisioned as a way to communicate transmission errors and delivery problems without putting any additional checks on the senders email address. For example, non-delivery and delivery receipts use the mail from: <> process. Try it, you can deliver any message to any server to any recipient using the mail from: <> trick.

Now, consider that you can’t block mail from: <> and also consider that you cannot build an RBL context against a server that is being attacked by a third party, and consider that there is nothing in the message to filter the junk contextually. Long story short, we implemented watermarking in ExchangeDefender that uses an encryption key inserted on the outbound server and validated on the inbound server. Basically, valid NDR and rejections will have the header value that can be validated if it passed through our servers. If it didn’t… poof.

But whatever affects ExchangeDefender, affects the Internet in general. And anything widespread enough can start taking servers down all over the place. This was the case that brought in the new customers this week. I personally spoke to several people that were referred to us by Microsoft CSS because their servers were completely DoS’ed off the network and they had no way to deal with it.

“We spent the past three days on phone with Microsoft and did everything from renaming the server, changing the MX record, changing the IP addresses and server names.. and we are still dying.”

Now, I hear that every now and then. I rarely take a dozen calls a day and have them sound the same. I also never hear the words: “Microsoft recommended you.” – I’d send you folks a gift package or something but I’m guessing your superiors wouldn’t be happy knowing you’re not selling EHS Foreskin or whatever it’s called this week. Thank you nonetheless, and in case you didn’t know, we give free ExchangeDefender to blue badges for their home servers 🙂

In Other News

You can almost taste the fear in IT hinting the imminent downturn of technology spending and hiring.

How do I know? My sales figures. They are through the roof. Why? My best guess is that everyone out there is so scared of the 2008 outlook and the looming recession in United States that they are hitting the pavement hard and trying to sell and close deals. The pace is almost unbelievable. Partners that used to add a customer a week or a month are adding multiple customers a week. They are also growing in size and in scope (though the last one is partially biased by us since we’ve stepped up the game of offering a whole bunch of services and making ourselves one-step, one-contact, one-support solution so our customers are now trusting us with more solutions instead of nickel and diming around other solutions and holding the support bag)

So I guess nothing motivates like fear. Or maybe with the SPFs and riffraff out of the way and out hunting tax certificates and mortgage foreclosures, the customer base is turning to the more reliable IT shops after getting burned. Or maybe the very same customers are turning more strategic in the face of fear and focusing on beefing up their own revenues instead of trying to DIY and patch things along.

 Whatever the reason, adversity either brings out the best or worst in us. Competition too. I was lucky enough to grow up in Europe (which has real racing, not hicks spinning around an oval) watching F1’s arguably greatest and most competitive racer Ayrton Senna. One thing that usually ended our mornings watching the RAI broadcast would be the rain on the circuit – when it started raining you knew Senna was going to take the race. He always pushed the limits, the car, and himself.. and in one of the interviews I will never forget, that hits me to this day, was the question the reporter asked Senna: “Why are you so good when it starts raining?”

Because where others hit the brakes, I floor it.

Remember, most people don’t start a business and most that do end up failing. So you’re up against the best, you’re either in this to compete and win or you should just quit now. Nobody wants to hear your troubles, though they make for great blog posts.

(Senna’s story of aggressive competitive nature and courage would be a little more impressive/relevant if he didn’t kill himself by slamming into a wall at over 200mph. Ditto with Earnheardt. But if you love what you do, and you get a thrill from being the best its hard to take #2)