The Joy of Mailbombs

One of the coolest things you can do once you’ve built a really, really large network is to look at the network patterns that emerge, hour to hour, day to day.

For example, every Monday at about 11 AM EST we deal with what has affectionately been themed “the royal mail server flush” – between 10 am and about noon, every Monday without fail, our network capacity drops by at least 30% – and for the longest time we thought it was just because most people got really busy on Monday mornings. But then we looked at the SPAM trends and something ridiculous like 99.3% of the messages relayed during this hour were SPAM messages. Now, when you correlate the IP reputations of the sending IP addresses with the volume of messages relayed over the past 24 hours from the same address and it becomes very clear what is going on. Corporate networks have so many internal systems that have been compromised that are sending dozens of messages (quite little) that over the weekend clog up these tiny servers. So, when the Internet connection or SMTP service or the fish appliance or whatever is in the way of this avalanche of SPAM gets repaired on the Monday morning…. the royal SPAM flush happens.

The other cool thing is, you are no longer succeptible to the ISP bull “Nothing out off the ordinary is going on” when you approach them with “Hi. We have 2 TB of email waiting to be delivered to your network and you’re unreachable from 22 of the largest networks. When do you expect things to be normal?” Fun, fun, fun.

Network ops… gotta love it.