OwnWebNow

Vlad Megapresentation Tonight - You're invited

I was invited to present content at Sacramento SBS User Group and Kansas City SBS User Group a while ago. I didn’t add either to my calendar, so when both Karl and Jason emailed me yesterday to see if I was ready I knew it was the time to test what’s know as the MVP race condition: “Can an MVP talk for more than 5 hours straight without sucking all the air out of the room and losing consciousness in the process?” We’ll find out tonight. 

Tonight’s presentation is free, live over the web and starts at 7:30 PM EST. It ends at approximately midnight EST. Here is the preliminary schedule.

7:30 – 8:30 EST: Exchange 2007 Overview (Level 100)

8:30 – 9:00 EST: ExchangeDefender, Shockey Monkey (Level 0)

9:30 – Midnight: Virtualized Services (Level 200), Shockey Monkey, ExchangeDefender, and other tools we make (and why) that IT solution providers are using to make more money. These will be product demo’s, not sales pitches.

Is there anything else you’d like to hear? Since we’re pretty much inviting SBSers worldwide, is there something you’d like to see? See you tonight, livemeeting info below:

Meeting URL: Livemeeting Address

Meeting ID: D7QS3H

Meeting Key: bn6'%~P

 

On Microsoft

Just wanted to clear the air a little because I got an email from a Microsoft PM today that said, paraphrasing “We want to keep you as an MVP and a fan of the product”. So for him and all the other Microsoft people who seem to have their heads chopped off on this blog on the daily basis, here was my response:

I'm still a HUGE fan. It's what I run on my desktops, laptops, phones, game console. The only things in house not running Microsoft are the router and the file server. So there is no love lost here.

Business-wise, Microsoft is killing us. The bigger we get, the more we spend on process and change control of Microsoft products yet our SLA and satisfaction ratings are in the toilet because we have run out of excuses to explain to people why we can't get by one week without a problem. Not just that, but with every issue there is a "cascade fault effect" by which we are blamed for every remotely related issue to the problem and have to prove it's not our fault by fixing it. Then just when everything is back in order we get blindsided again.

OWN is in a very interesting edge of the server-based services. Because everyone and everything is remote we don’t have the benefit of setting our own risk tollerance that Susan talks about all the time. Our risk is the day of, or in some cases like the latest DNS warning, weeks before Microsoft produces a hotfix/patch. We patch immediately because we are a large enough target, with a large enough pipe and buffet-style storage that make us a target for anyone field testing their latest exploit. You can’t set your own risk level by blackholing /21 sized subnets access to port 80, it’s a different scope of service.

My biggest beef with Microsoft on the business side is the lack of business owner level type of threat assessment that is as readily available as their marketing. When a business owner is down due to a Microsoft patch, hotfix, etc I need Microsoft to corroborate my story and act like my partner. I need my customers to be able to see the problem, confirm with Microsoft, and let me go about fixing it without having to sit in with the management of each affected company and try to explain technical jargon to a customer without a way to back it all up through doing my own forensics, technet articles, newsgroup posts, etc.

But since I know that Microsoft’s upper management reads my blog and knows just how hardheaded I am… How would you like it if your partners got together and created a public site to collect and share issues related to latest hotfixes and patches in the open so that customers themselves can see the the wide death-toll counts?

ExchangeDefender v3 - Take a look!

It gives me great pleasure to announce that ExchangeDefender v3 is up and running and we’re taking a few days off  Seriously, this has been a long development cycle and a big source of pride for us is that the interface guidance was given by actual ExchangeDefender users who wanted to be more efficient as they go through the mail. We’ve managed to simplify the interface even further to really make security management as seamless as possible.

Don’t believe me? Check out this video: ExchangeDefender v3 Video Tour (5 minutes)

Thanks to all that made this release possible, in particular three folks outside OWN that worked very hard to bring you this thing: Rich Walkup, Judy Schmidt and Pablo Averbuj. It is absolutely amazing when your own customers take the time to improve the product because they like what it does and want it to work better: and to that end thanks to all the customers for all the feedback, guidance and testing. Thank you for helping us get to this point.

What’s next? Well, MSP stuff this week, agents hit next week, Live Archive feature (which is absolutely revolutionize how you think about SMB messaging continuity) and more all coming online over the course of the next month or so. The goal behind ExchangeDefender v3 is ambitious – we aim to be the most feature-packed easy-to-use, over-hyphenated mail security service out there!

By the way, for any suggestions/bugs/complaints/etc please use the Own Web Now official forums where things can be seen out in the open and we can have a more open discussion that one-on-one. You do have to be registered to post, so contact Support to get an account. 

Big Pimpin'

This is a business post, as you can tell by the category, but I didn’t think that the title was appropriate for the Own Web Now Corp blog. So here goes the pitch, I’m interested in your feedback if you’re giving us money.

For years our business plan has been to offer high-quality SMB commodity services with a high degree of actual service. So instead of driving the cost of the service straight into the ground and making you pay through the nose for support or through lost business when we embarras you like my competitors do, we kept our pricing at levels where we can all make a profit but you get the benefit from a true partnership. Makes sense, right?

Since the startup and maybe until a year ago or so most resellers we dealt with were just happy to either get a check or pass the discount on to their customers and just move on. Not quite the case anymore. Today resellers are firmly embedded in the clients IT infrastructure, support, etc. They more you promise to those clients in terms of flat support costs and services the less margin you have to use to deal with services that will fail you – the services everyone needs like offsite backups, web hosting, email, virtual servers, dedicated servers, etc. OWNs are more reliable, thus cheaper overall. In the end, we grow much faster together.

But…

But times change.

I have been one of the most prominent pimps of the work Karl Palachuk has been doing. The books aren’t cheap – but I’ve yet to meet a person that wouldn’t have paid 10x as much after having read it.

Last weekend I was in LA and Erick took me to check out what Intelligent Enterprise is doing. I got to watch one of their MSPU bootcamps and.. well, wow. If you thought the book was something you’ve got to check that thing out. Erick actually let me tape one of the sections of the bootcamp and show me their entire practice.. I was kind of skeptical of the SMBer that would pay $3K for that training but having seen it live… again, wow, they got their s… together in a really big way, talk about an investment in getting to the next level.

Amy Luby also has an organization that spawned out of the SMB Managed Services yahoo group, that takes an online community approach to helping VARs scale up. I haven’t seen it yet but I keep on hearing good things about it.

So here is the point..

All of these things cost money, in some cases a lot of money. And they are well worth it. It is in Own Web Now Corp’s best interest to support these organizations because they produce content, training and support services to help smaller IT shops make it big, take things to the next level, get organized, etc. For example, we’re sponsoring SBS Migration’s ITPRO conference because its a pure SMB community event where people can come and exchange ideas without being harrased by vendors posing as presenters and forcing people to eat out in a parking lot. It just makes sense for my partner base.

The more embedded my partners become in the SMB they support, the better off OWN is because we get more business, more support, more feedback and in turn provide better services that make us all more money…

Get to it already..

…. so I’m really happy to pimp what others in the community are doing because it makes sense for everyone involved. Thus Vladfire, SBS Show, Vladville and the thousands and thousands of people that have partnered with us. I can tell you to go buy Karl’s book, but what if I actually gave you a cash incentive?

So here is the pitch I want your feedback on: Any discount offered on ExchangeDefender service would just go on to pay for something else. Maybe power this month, rent the next, etc. Most of you that are in the partner program and took the time to join our conference calls last night already know just what we’re up to in terms of the new support services and so on… but what if I gave you a true incentive to sign up for some of these SMB services by giving you a discount if you’re doing business with me AND you belong to their programs, books, camps, etc?

I want to give you a financial incentive to get training and stay involved in the community as you grow your business.

Look at it as my investment in your business by taking a part of the revenues you bring in and reinvesting them in education and training that would scale you to the point where you’d need even more of my services. Not that I don’t believe in that very thing today, but lets face it, money talks bull…. walks.

Good idea? Bad idea? You know where to let me know whats on your mind…

Form SPAM out of control

Over at OwnWebNow and ExchangeDefender we have forms that allow our customers to request a callback.. so instead of calling us and leaving a message or sending appointment requests they can just tell us when to call them.

Sounds good, right?

Well, over the past few months the amount of form SPAM has absolutely gone wild. I've initially hacked in the WordPress mechanism for blacklisting IP addresses.. then I tied in CAPTCHA.. then I tied in the 'prove you are not a computer' deal where I threw in a form input element and using CSS set it invisible - pretty neat, worked for about a month. But now they have broken through that as well..

My last step is to keep on rotating my hidden elements and timing the sessions to make sure they are not direct hits.

What I have found in common is that they all are just junk using a freemail address like yahoo, aol or hotmail. I have been junking them pretty rapidly but I suppose the reason I am writing this email is that I see a business model in the online form management. I talk to people every day that are having issues with ExchangeDefender and forms that end up in quarantines (later they apologize when they see that the form had a great stock tip in it instead of that part order) but there is a lot of demand for online data management and customers do not want to pay for a Sharepoint CAL... so, where is the middle?

Day in a life of a spam killer: pre-greeting traffic

Yesterday we implemented some new magic against tar-pitting smashers and man does it make a difference: Since noon EST we have rejected an average of 46,128 messages per-node. We then ran a report – 100% of the IP addresses that had their email rejected had mail delivered to the SureSPAM Quarantine the day before.

The significance? System performance, employee efficiency.

System Performance – Any bit helps. By rejecting the messages at the perimeter (yes, even ExchangeDefender as a perimeter service has a perimeter network) we save network resources because we have less messages to process through bayesian filters, less statistics to calculate and ultimately less spam that we’ll have to store.

Employee Efficiency – Two fold: First and less important, the remote likelyhood that the mail produced by these spamming hosts does not get properly identified as SPAM and ends up in the users inbox. Second and most important: less mail for the employees to review. Since I can guarantee that no legitimate mail was sent by these systems I can save my customers time from having to browse through meaningless quarantine reports and let them look at the stuff that is actually likely to be SPAM.

That last part is where I and some of my partners have very vocal arguments:

My mission is to cut down and eliminate spammers and their impact to the customer, not create pretty graphs.

My partners need the graphs and the data to prove to their customers that the SPAM is really bad enough to justify ExchangeDefender.

I think pretty much anyone who would spend a day on IMF/Outlook-only protection would scream their lungs off and rip their hair out over the amount and volume of junk thats floating out there. And while I can understand that sometimes people need to be shown what they are paying for I am not a fan of selling the fear. People either have a problem with their email or they don’t. And I’ll tell you this much – we do not lose customers. Nobody that has ever become a paying ExchangeDefender customer has ever left (short of AUP violations and going out of business) so I’m doing the right thing here, I really don’t think that at the end of the day the customer is going to be concerned about the value if they see 0 junk in their mailbox - they all have multiple accounts and contacts that are being flooded, they know the impact.

The second and more techical reason for what I do — By rejecting a message on connection basis I am saving back-scatter. When I reject the message with an SMTP error code the remote mail server is left holding the bag. When I accept the message and then have to send it back due to the prohibited content or system policy I am doubling the amount of junk that is being sent over the Internet. “But Vlad, why don’t you just not send it at all, just junk it?” – supose the message was legit but did not meet the policy requirements, or had questionable content, or did not meet the security requirement (some customers only accept encrypted messages or only specific senders), or had a virus, or had an attachment that customer would not accept, or, or, or.. My whole technical point here is that there are many good reasons for sending back an NDR and we (ISPs) need to be good Internet citizens and not flood the network with the meaningless ones.

Adjustments to Tarpitting for ExchangeDefender

Earlier today we have made some significant changes to the tar-pitting mechanism under ExchangeDefender. The new mechanism is designed to reject messages from hosts that do not follow the proper RFC SMTP dialog and attempt to smash tar-pitting. More on the basic concept of tar-pitting is described here.

The Problem

While tar-pitting is great for throttling remote mail servers and reducing their ability to efficiently deliver a lot of messages, the concept only applies against botnet servers that are attempting to deliver mail in bulk. Anotherwords, tar-pitting is only effective against servers that are concerned about getting the message out as fast as possible. By delaying the SMTP greeting banner, in theory, the remote mail server would have to wait a pre-determined amount of time before starting to send mail. Many open connections at once would overload a single node.

However, spammers no longer exclusively use single nodes in a full force attack. They use the botnet concept by load balancing their broadcasts through multiple servers. As such, those servers connect every few minutes and only relay a single message. By doing so its hard to blacklist them immediately because their overall reputation does not have enough data to be determined. These botnets are designed to bypass tar-pitting by opening a connection and sending data as soon as the connection is opened.

The conversation looks somewhat like this:

Trying xx.xx.xx.xx.exchangedefender.com.

Escape character is '^]'.
ehlo spamming-idiot.org
mail from: spammer@spammer.org
rcpt to: vlad@ownwebnow.com
data
Subject: Get a college diploma.
Ohio State University may be a loser but they'll give you a Ph.D in nuclear physics based on your life experience.
.

Now the (target) tar-pitting mail server has accepted the connection but it has never sent the SMTP greeting. However, it will process the message as soon as its tar-pitting interval passes, thus in part bypassing the tar-pitting and delivering the message. Not good.

Notice that the client above did not wait for the 200 greeting banner, did not wait for the 250 Hello, did not wait for the server to acknowledge the recipient and the sender. They just wrote to the socket and waited. Now even though this does diminish the spammers performance a little (by taking 5 seconds to deliver the message) the message still gets delivered. That’s a problem.

The Solution

The solution is fairly simple: Drop connections with mail servers that are not adhering to RFC. The second the mail server issues a command before the 2.2.0 hostname greeting banner it will get dropped, logged and its tar-pitting interval extended.

Instead of a tar-pitting process that delays the connection a few seconds, this process allows for a connection immediately but delays the SMTP greeting banner a few seconds. As such, it can eliminate server load caused by spammers that think they have found a way around tar-pitting.

We ran this in testing on our production systems and have found 0 false positives over the course of one week. All hosts that were rejected were also on multiple RBLs. The implementation is transparent to the user and administrator and introduces a random (less than 5) second pause on all connections that do not have a reputation rating with ExchangeDefender. Less spam, less stuff to review, less bandwidth and less stress for you.