Disturbance in the Microsoft Patching Force

Looks like there was more fun overnight than the Central Florida hurricanes, check out what OWN was up to:

For the past 10 hours or so we have been handling an 820% surge in reboot requests for hung Microsoft servers after applying the latest security patches. Our managed network of Windows 2003 servers has not been affected but a huge portion of our network apparently has, please be advised.

If your Windows Server becomes inaccessible as a result of the latest patches please open a ticket request and mark it as urgent. You will not be charged for the support request and your reboot will be handled with the highest priority. We have an additional shift on hand in all data centers to help you through this network event.

Ouch. I dig the quantified percentage surge breakdown, and they bitch about Shockey Monkey reporting 🙂

To be honest, I am quite happy with Microsoft patch quality as of late. For months now you could reliably install a patch and not think about it twice about not rebooting properly, hanging the system, bluescreening or worse. It has been pretty much as rock solid as patching gets.. Do I wish that was the case before a multimillion dollar investment to provide every server on the network with a remote reboot switch? Yeah, little bitter about that, but it has saved more money in SLA refunds that we were cutting because we stood behind the reliability of Microsoft software. I’m pretty content with Microsoft, even given the events.

I did a little bit of digging. The freezes/hangs were not associated with the systems that were on a regular patch schedule – so the systems that were managed by windows update server or regularly scheduled windows update were not the problem. The systems where people did their own patches and manually did Microsoft Update in the browser… poof. I guess that will make Susan feel a little better because her experience has been exactly opposite in the past few months – manual updates fine, managed updates causing problems.

The big picture…

I spent the better part of 2000’s bitching and moaning about Microsoft patches, cutting checks when they failed, escalating them through Microsoft and feeling like a complete douchebag when I asked my PSS friends a support question at 1 AM. Nobody likes being picked on for something that is not their problem. And to give Microsoft credit, the patch situation has greatly improved. 

Over the last few days there has been a little conversation going around the general displeasure with Microsoft, you know, things just not working up to the expectations. I shared a few details on the thread about how OWN faced the same issues and how we overcame them and why we were able to overcome them. The message was largely ignored, pushed forward by other complaints and issues until multiple people basically leveled that they don’t have the time to sit around and complain but a business to run.

Therein lies the problem. There are companies that are service focused and driven by the customer feedback. Then there are companies that are product focused and believe that the resolution to issues is an upgrade. I believe that Microsoft and I sit on the polar opposite ends of that world, one does everything it can to reassure the customer base that the fix is on its way, the other tries to fix the problem right away and hopes the customer doesn’t bash them as ignorant.

Customers see and recognize this as well. When everyone is complaining about an issue out loud and nothing is done about it, people stop complaining. For the most part, the peer system is very much a collaborative sanity check for an IT person – ok, now that we all agree its broken let’s just wait for a fix – but what happens when the problem is in licensing? Or product implementation? Or partner program benefits?

Well, you get quiet. You sit on your hands, you look around, you cough, and then you realize you got all this shit that needs to get done and you focus on your own problems and try to minimize the impact that Microsoft has on you. Whether you stop patching completely or invest millions of dollars in reboot switches, you find your solution and you move on.

MVPs want people to keep on complaining, to keep on escalating issues, to keep calling PSS, to keep flooding the newsgroups… because they care about the product and want to see it improve. They need the ammo to say that a product sucks and ask for changes. And more power to the MVPs.

Business owners have better shit to do, after multiple complaints and no response they get the message, they believe that if Microsoft was truly concerned about the problems they would work on fixing them, that it’s Microsoft’s job to fix the problems not theirs to continue to complain and that if Microsoft was truly interested they would do something about all their previous issues.

Although I’m an MVP, I am on the business owners side of the argument – If a company is truly concerned about their problems they would be more proactive in addressing, admitting and processing them, they wouldn’t be reactionary to the “filtered up” complaints of people who haven’t become completely disaffected yet.

[ __________ insert link where Susan beats me up over what I just said ]