ExchangeDefender

State of the MSP

What a fantastic few days it has been playing a host to the cloud of MSP providers the ConnectWise and HTG conferences have brought to Orlando. I have three quick thoughts I have to share with you all, but given the extent of NDAs and confidentiality agreements I have to be a bit vague:

1. Thank you all for coming to Orlando. Having spent most of the time in hotels and on the road (vs. my new house) I am truly glad that I can drive ten minutes back to Windermere and just sink into my own bed.

2. It’s tough out there for a pimp. Most of the owners and managers I’ve been talking to are not doing well. As a matter of fact, a few percentage points of growth seem to be good news. Sales of hardware are down. Adoption of managed services is down. Overall climate seems to be decidedly down. People are making money but they want more. That’s capitalism.

3. Our new pitch of “cloud direct” is going well. The “ExchangeDefender 75 Cent Cut Throat Special” has been amazingly received by everyone I’ve talked to and the market share we will gain on the back of this is amazing. We are really going all out here because even a blind man sees the end of this tunnel: everything shall soon be free. That’s not good news for people that had hoped to make money selling software but the true definition of a business success is in the amount of the profits it generates, not how it goes about doing so (within law and reason of course).

The conversations I’ve had today, even with the rabid fans of other products, makes it clear that business people make business decisions. They don’t buy it on personal intangibles over how much they like the guy, we’re not Hannah Montana fans running around with glitter in our hair – bottom line is.. well.. what contributes to that bottom line. Profits.

And as I said in the previous post – game on folks. This isn’t the IT sector of the 90’s or 2000’s, fighting for every dollar and every penny is going to get harder. It’s your call whether you’re going to do it, or watch from the sidelines as others take it all over. Not sure what you’re going to be doing but I’m not stopping. Google isn’t stopping. Microsoft isn’t stopping. Yahoo isn’t stopping. Opportunity is yours for the taking.

One thing I heard the most this week: “Well, I’d love to work with you on that” when talking about the direct model. Looks like all the partner feedback built something good, eh? We’ll be at CW and HTG all week, track me or Shannon down and find out what it means for you.

Community Challenge (Important)

Over the next two months we will be unveiling our grand strategy for the deployment and management of cloud services in SMB. We have been wildly successful with this model for years and it’s what’s attributed to the phenomenal growth of Own Web Now.

Note: This is being piloted through HTG partners first and it’s the only way to get it right now. It is not an exclusive arrangement by any means, it’s just that a lot of companies in HTG reviewed, guided and helped design this model for us. Later this year it will be available for everyone and through our alliances.

Ok, now that we got the messy part let’s get straight to the point:

We are going direct!!!

If you haven’t had a heart attack or a brief moment of blurred vision, congratulations. You’ve worked with me before, you know what I’m all about. No, Own Web Now Corp is not going direct. No way, no how.

But we have built a brandable (your label) portal, workflows, support and accounting processes to enable you to go direct and not incur the costs of supporting our own products. Here is what we’ve got:

  • Free branded portal secured with SSL (https://yourportal.yourcompany.com)
  • Free backend integration to ConnectWise and Autotask for service orders, service tickets, account configurations, inventory and statistics.
  • Free support services for all our products. Your users can open support requests either through this portal, or through your own ConnectWise or Autotask and automatically get delegated to us for resolution.
  • Free updates to inventory and configurations, when something changes on our end it also changes in your own home portal so you can always bill properly.

The big picture here is that you can now offer cloud based services backed by a SAS 70 – Type II audited company, with a full SLA, redundancy and history to back it all up.

So without the cost of supporting, or accounting or actually maintaining and managing the service… because hint: Users can make their own orders and changes for things like Exchange hosting… what are you going to do with the markup?

I’m buying a Ferrari or two. But that’s just me.

Most of you have seen this path a long time ago – there is no ignoring the consumer that wants a cheap cloud solution. But as so many of the HTG members said over and over again.. if we can’t make a margin, we can’t do it. Well, Own Web Now has always been behind the channel and now we’re doing it again.

Oh… and one more thing:

There has been a little company out of Cali that has for year been a giant pain in my butt. It’s the only one we lose to competitively. Why? Price. Always comes down to price. Well, we’ve optimized ExchangeDefender to the point that the new release (ExchangeDefender 5) runs in 1/5th the infrastructure the v4 did.

So for the first quarter of ExchangeDefender 5, all new accounts will come with a new price. $0.75. No minimums. One user at a time. Full feature set. Everyone you sign up at that price level will stay at that price level.

Oh… and just one more thing: Instead of launching the product and providing education, marketing and training after you’ve already grown frustrated with the product: We’re spending all of November holding webcasts, training and producing your direct marketing collateral. Bigger partners also get MDFs.

So when December 1st, 2009 strikes… I will have the biggest month in the company history. Vladdy needs a new Ferrari and a bigger market share.

Game on. I’ll be in Orlando pimping this stuff all of next week.

What’s left of the cloud after it’s done raining?

A pwn3dclose personal and professional friend of mine and OWN’s apparently made a terrible business partnership decision. Naturally, I promptly offered a swift kick in the balls because your friends should always be there to kick you when you’re down so you never let it happen again.

For the past two days, a small portion of our friends client base had been seeing the screen to the right. It sucks. But as these things go, the CEO apologized, refunds have been issued and hopefully the business goes on.

My heart goes out to them on this, disasters suck... but..

The one thing…

The one thing I have learned in the past 12 years of managing Exchange is that Microsoft will never, EVER, make one that is capable of being fault tollerant. Never gonna happen. We’re getting as much unpredictable crap with 2010 as we’ve gotten with 5.5 and every version in between. Such is the nature of the mail servers, throw enough crap at them and they will go down on their knees.

This in no small part is why ExchangeDefender exists.

One thing I know as a cloud provider is that my stuff will eventually crash. It has in the past, it will in the future.

One of key things behind ExchangeDefender is LiveArchive, a realtime business continuity solution based on Exchange 2007. Not on Bubba’s Pretty Good ASP Webmail.

As we process inbound and outbound mail through ExchangeDefender, we create a copy. One goes to the recipient, one gets copied to your ExchangeDefender LiveArchive enterprise network in our data centers.

When, not if, your Exchange server goes down you just open up our OWA and continue working in realtime, with your same identity and email address. Even same login and password. Everything works – calendars, notes, reminders, and you have a FREE year worth of your email that was sent and received – not just a few messages since you’ve crashed.

This, in no small part, is why even people that hate the cloud work with Own Web Now and use ExchangeDefender. Certain CPA’s. We even provide LiveArchive with our Exchange hosting. Yes, the same clustered Exchange 2007 we offer for resale comes with ExchangeDefender and gets you a full year of free business continuity. Why? Because if we have ever leaned anything in our experience is that stuff WILL go down.

And we do it for free. We’ve covered people during migrations, when servers were down for a few days, even our partners non-profit who had their server go down and couldn’t afford a new one – they worked through LiveArchive for over 2 months.

This is the KEY aspect of providing stuff in the cloud – who has your back? Can you trust that the specs are what they are? Maybe.. But can you trust the odds that two separate server infrastructures won’t go down?

When it does, you can continue working. All you gotta do is drop me an email. If you want to check it out, vlad@ownwebnow.com is where you can get it from me for free.

Open letter to OWN/ExchangeDefender and Autotask / Connectwise community

Every now and then we come to the difficult decision, or a set of decisions, that might impact the way our partners work. This is my personal attempt, as the guy who is paying to make all this happen, to explain our situation and answer the questions before you’ve had a chance to form your opinion based on the first few keywords that upset you.

Later this month we (Own Web Now Corp) will be announcing our next generation integrations with Autotask, Connectwise and yes, even Shockey Monkey. This is a significant effort, a significant gesture – because it is in no way a part of our products functionality – to help you think about OWN a lot less and make a lot more money. I will go a step ahead and even say that what we are bringing to the integration picture (and beyond, in custom software for the platforms) is unlike anything available on the market… and if you’re an MSP you owe to give us a look.

On to the imaginary questions:

Q: How are we going to implement the integration? Can I have the white paper? Can you walk me through it?

There are two possible ways to get Own Web Now integrated with your PSA deployment:

Supported – Under this scenario we will offer you the whitepapers, create the integration workflows, queues, service boards, dialin numbers, offer best practices and support the integration when it breaks. This service will come at a reasonable one-time cost and will include training and ongoing best practices from others that use it.

Unsupported – Under this scenario we will send you the whitepapers and wish you the best of luck. This will be available free of charge and will include all the features of our integration suite. No support is available. If you choose this route you will not be able to upgrade to the supported product in the future. I know, I know, while it may make sense for you to try it on your own, fail, and then blame my staff into doing it for you for free while also training you to use your PSA, it is really not a fair scenario and not one I expect to see – you know exactly how much time goes into supporting, managing and billing the solution, so if there is even an ounce of question of whether our service is worth $500 then you have more serious questions to address.

The integration is available between us and Connectwise, Autotask and Shockey Monkey.

Q: We use ____ PSA? Will you integrate with that? When will you integrate with it?

No, we only integrate with Connectwise, Autotask and Shockey Monkey. We never (ever, ever) expect to integrate with anything else. If your product doesn’t contain Connectwise, Autotask or Shockey Monkey we will not integrate with it.

Q: Will the integration be different between different platforms?

While we have tried to bring the identical level of integration to all platforms, there are many factors prohibiting us from offering the same feature set across all platforms. Things contributing to differences between platforms can be attributed to the maturity of the API we are using to integrate with your platform, similarity of data types for services / contracts / products, vendors willingness to assist with problems, vendors restrictions of features across product lines.

Most importantly, our technical inability to create an integration point because of our unwillingness, incompetence and BSTDWOT (better “stuff” to do with our time). It’s not that we are bad people, it’s that your feature request will never generate enough money for us to justify writing the software in the first place.

Q: But..but..but… Vlad! If you can make this work with Autotask, why can’t I have it in Connectwise? Why does it work on the dedicated installation of Connectwise and not hosted? I have this feature in ConnectWise but I’m moving to Autotask Go to save money, I can just assume it will work the same right?

I don’t want to oversimplify this but here it goes: If you see a feature that you don’t have in Connectwise, it’s Arnie Bellini’s fault. If you see a feature you don’t have in Autotask, it’s Bob Godgart’s fault. If it’s a feature that you don’t have in Shockey Monkey then it’s coming in the next release.

Joke aside, see the above answer. The platforms are too different for us to offer the same feature set. If you see a feature in Connectwise that you need in Autotask or your business will implode, please switch to Connectwise. Or vice versa. Again, we are not bad people, we just need to explain that it’s not really apples to apples out there:

All OWN workflow and order management is designed on top of Shockey Monkey. That work is then ported to an extent to two different sync APIs. Some stuff will work with your PSA, some will not. That’s just the reality of software development.

Q: Is this just an elaborate ploy to sell more Shockey Monkey?

No. As a matter of fact, we will not take competitive orders for people looking to switch from Autotask or Connectwise. Truth is, we develop software for our platform and then extend it, within reason, to our partners platforms. We do not have any more advanced access to either platform than any other vendor, we are bound by the same legal and licensing as every other vendor and it’s a leveled playing field.

Q: When will you do ____?

Official answer will always be no. I cannot stress this enough, we are talking about software and features that save you money and make you more productive – I do not want to put more pressure on my staff or shift their focus from our tasks. We don’t write our software in India or China for $2/hr so the feature you are looking for will be available in the next release (maybe not the very next one)

This may sound harsh, and I wanted to give you an idea of the possible conflicts I see in this initiative so you can know where our line is drawn. We will do everything in the power to help out the people that see the value in this.

By all means we’re going to do everything in our power to give our partners an advantage with their PSA choice. All of the above will be made very pretty contract language with blog posts and screenshots to follow.

Partner Up

If you work with me, you will be getting an email from me over the next 24-48 hours from now. It’s pretty important (to me, to you) so I hope you take a moment to go through it and see if you can benefit from it at all.

To sum it up: Things are going really well at OWN, we are taking advantage of the economy and we’re about to embark on an insane release schedule to expand our solution portfolio which is why we’ve been relatively silent on a number of fronts since last summer. Generally, there is a lag between us announcing something, partner signing customers up and then realizing there are missing pieces. I’m offering some incentives to be more involved in the OWN design process.

This is the extension of the “V…” newsletter, only applied more closely with what I actually work on. While I gave the open community one a shot, I am tired of trying to figure out where the line between the personal and business stuff happens to be – if it doesn’t apply to sleep, it doesn’t apply to newsletters :)

Bye Bye DSBL - Check your mail servers!

At one point last year the DSBL blacklist, quite popular in it's hayday, lost it's entire database in a RAID system crash. The name servers kept on answering requests in the meantime. Yesterday, this activity stopped.

This means that if you still use the DSBL, you might be delaying or bouncing your inbound mail. Check your servers and make sure the DSBL is not listed. In short, everything with *.dsbl.org must go.

Note: We have not / do not use DSBL in ExchangeDefender so if you are with us, you were not affected.

New ExchangeDefender engine goes all in

With the first 72 hours running against delinquent accounts, the new routing engine performed remarkably well. In about an hour we'll do a wide deployment across the entire network. It will take approximately 3 hours for the upgrades to be performed.

The good news:

  1. Performance was off the charts. Always good when rolling out new software.
  2. We were never the bottleneck. This was a significant area of doubt, even though we had all the numbers in the world, we never had a proven model over time. Here is essentially how it works: messages are flushed to a queue and delivered sequentially from each processing node. As the mail load increases throughout the day, is it due to multiple ExchangeDefender connections or is it due to the saturation of the link? Good news is, it's not us. When tested over 300+ IP's worldwide, when certain links showed slowdown, others went through just fine.
  3. Network congestion or server overload? This is something we are generally not alerted to and something VARs rarely either know how to access or have permissions to view. Exchange 2007 does issue performance based errors but your weakling consumer firewalls do not - they just defer the connection or drop it outright.
  4. DNS issues? This one was fun.. we pretty much DDoS'ed people :) We found hosts who took forever to issue a banner - so we flooded them with SMTP connections. Then we started transfering 256K attachments, then 1 Mb attachments. Guess what? They flew!!! We are narrowing this down to two effects: 1) Problems with DNS servers. 2) Excessive RDNS or RBL lookups.

Problems with DNS servers are more difficult to isolate because they may be sporadic depending on their load. As most sites are not likely to run their own name servers or their own caching name servers, external lookups may take longer. Sites that ALWAYS had terrible initial greeting are very likely just using dead RBLs or way too many antispam measures - ALL of which need to be shut off.

So far we're looking at the healthiest week on the network, despite DDoS and attacks as ususal. Let's see what we can pull up when the entire network is actively managing connectivity to target servers.

New ExchangeDefender Router Code

I guess we will never learn if the support team could kick developer teams ass.

Earlier today we rolled out the new ExchangeDefender code to a select* group of users. This new software is intended to adjust connection timeouts during peak hours and log timeouts so it can be reported.

One of the more frustrating things we have is that a lot of people simply do not have a network with enough capacity to run a mail server. Without limits in place we find sites that at random times get congested and only accept the smallest of messages. Bigger pieces, particularly those with attachments, can get delayed from a few minutes to a few hours. This is a rare issue but it creates an insane amount of support requests.

Recently we started extending connection timeouts on per-IP basis but I'd rather throw alerts and advise clients to upgrade their connections (or suffer timeouts and delays).

Now if we get over 30 seconds of inactivity the message is flushed to a queue with a 90 second timeout. If it's more than that, it will be switched to a queue with life of 180 second timeout.

Logging of this activity will help us plot the timeouts on per-host basis so we can have a model for each client that reports an error. It takes far too much time to go through each queue to inspect each message, now we can have full host reputation reporting to justify the recommendation they get more bandwidth.

Some experimental code ought to improve things. We'll know tomorrow.

* "Hey wait a minute, I am not paying you to run experimental software on my business email." - I know and I have great news, we're only using this on the delinquent accounts for people that are more than 30 days out. :) So if you complain, expect to talk to the billing person first :)

ExchangeDefender rolls out Client Software Suite

The official announcement is on our official blog. Please subscribe to that feed and join the other 8 people reading it on an annual basis. That's the official story, however, in the interest of honesty to our partners I have to optimistically apologize for things taking to long to get to this point.

You see, as ExchangeDefender (and I) grew up, the product matured as well. I've always embraced the development cycle that primarily served 90% of our users, with lower priorities assigned to the needs of the admins, MSPs, IT royalty and power users. Why? Well, the ExchangeDefender service is first and foremost a protection mechanism for people that are not INFOSEC hobbyists. That's the customer that spends the money on the service, that's the person I need to secure and so long as I please them, we get to keep the client.

Always (always, always) take care of your customer first.

This is why ExchangeDefender has some of the highest retention rates around.

I think we failed to stay true to that mission in our objectives. Basically, we dictated some features for the sake of simplicity and cornered ourselves into a position where users have become so dependant on the logically broken process because they have conditioned to it, and will not consider something better.

The most hated (and most beloved, and most pleaded not to be removed) feature in ExchangeDefender are the daily reports. People loved this stuff in 2002! It provided archival data for the SPAM past 7 days, daily reminders and stats, ability to release SPAM if it ever gets in there, etc. Fast forward a few years and the report is stacked with some of the most vile stuff anyone has ever written. Laced with stuff that would trigger even the worst of antispam software. But users are conditioned to expect the report. They bitch when they don't get it. Most don't read it - it goes straight to trash.

It is in our (OWN) best interest for our users to better interact with the service because it can learn to protect them better. I think I said something in one of the interviews today regarding the value of client installed software for a pure off premise solution. We aren't just fighting with external threats, we are fighting with our users stupidity and naivete. When stuff breaks the cost of the product skyrockets if everyone in the technology chain has to be involved in fishing or tracking the email through the system. We can do much better than that, but we need to offer something that lets the user take a chance on the new solutions.

So after months of hard work, ExchangeDefender Client Suite is out. We'll add file sharing, web filtering and a few other things throughout 2009 because we are no longer able to protect the user with just the SMTP filtering alone. Much like we can't offer them business continuity with only 5 days of retries - where LiveArchive thrives in the enterprise contracts and SMB shops alike.

I hope you enjoy it. And to the slimy ones in the audience, I hope you can make some profit off it. Differentiation demands premiums, pimp on!

Grumble Grumble Grumble

This was originally intended for OWN Blog, but I could not quite manage to make the post respectful or free of profanities, so Vladville it is.

I want to let you in on something I'm thinking about.

The most exciting product I have is ExchangeDefender. There are days when I just sit there, logged into one of the clusters and watch the lines pop up a few times a second: Detected 112 SPAM, 5920 SureSPAM [Quarantined].

Each time a line pops up on my screen, a little dream of a spammer dies.

It's the best job you can have on the Internet while still keeping your clothes on.

When I started writing this software in the 90's I never thought we'd make it to year 2000 doing this, much less 2009. But here we are in 2009. Now we have new enemies, it's not just Vlad vs. Spammers.

Our users are getting progressively more retarded. (#2)

Our partners are getting dumber by the minute. (#1)

Our battle is no longer against the Spammers. (#3)

The reality of the situation is that we've reached the level of technology use in business where education is not enough to keep stupidity away.

#1 Lot's of people made lot's of money with ExchangeDefender. It seems they invested it in hiring people dumber engineers because not a day goes by that I don't hand myself a full faceplant reading some of the feedback and support requests. It's as if people actively try to shoot themselves in the foot, screw their customer, and ask us to find a more lenient way for them to cover it all up. This leads to #2:

#2 I have lost the number of counts I have personally had to explain to people that the UPS message that we blocked as SPAM was actually a forged SPAM and not really from UPS. I am not a violent person, but I wish I could jump through the phone and choke the stupid fuck, who had the above explained to him, that still wanted the attachment just in case.

#3 The most dangerous people happen to be semi-technical ExchangeDefender users that think they are smarter than the average bear. You know, the CPA network managers (happy new year dear!) who think they have this invisible shield around their network that guards their users from running around the room with scissors and impaling themselves twice a week.

Our threat portfolio has now grown beyond the realm of email.

My personal problem with the above is that it's not our job to protect users from their stupidity. Web filtering is clearly a realm of local area network policy management, not cloud service provider who takes the brunt of the attack.

Here is a way to simplify the situation: When you are being attacked by thousands of Russians, Chinese, organized criminals and mobsters you do want to have a 500 lb gorilla in front of you. When your dumbass employee wants to download TV shows at work, you beat them with spare office furniture.

That's just my opinion. Check the local laws, it may not be legal. God bless Texas.

Nobody gives a flying f... what I think:

So we're extending our ExchangeDefender portfolio to include web filtering.

The ExchangeDefender Desktop Agent (www.exchangedefender.com) will have a new beta available mid-January with a web filtering addon that will explicitly protect users from malware. We're going to try our best to make this free.

Depending on the reception, we will extend it to include policy management and other categories, though that product will come at an additional cost (approximately double what ExchangeDefender currently costs). Almost the entire fee will go to pay for the definition files because, believe it or not, people that browse around pr0n sites to categorize them as pr0n charge for a list of those web sites. A lot.

It has been my firm belief that no infosec company should ever behave like a mob and deliver "as much protection as you can afford" so it comes as a bitter change of course for us to offer an expensive addon to the service.

Anyhow, just thought I'd let you know what's on my mind. If anyone on my staff would like to take a crack at this blog post and make it sound better than "You all suck, so we have to change our approach to protecting you" please let me know :)