Security

Microsoft, Dell: Please stand closer together (OWN Alert)

Dear partners, please stand closer together so I can get you all in the same shot. No, thats not the camera I'm holding.

Just wanted to alert you to a few things that have been going on at Casa de Vlad in our 11 data centers. We're having issues over a large portion of our Dell server base that has SCSI/PERC4 controllers installed and/or USB 2.0 drives installed. The behavior is as follows:

  1. System powers on and allows OS selection. Windows 2003 Server splash screen comes up, few seconds later black screen or reboot.
  2. When booted up using Safe Mode the system hangs on loading acpitabl.dat

There are dozens of solutions of which we have identified the following so far:

  1. Unplug the USB 2.0 drive. This worked on just a few systems and allowed the boot to proceed. On reboot it powered on as well. Will test again later.
  2. Clear SCSI controller card configuration and rescan the bus. This worked on a few as well.
  3. Power system down and perform a full fs check. Windows PE is your friend.
  4. At this moment we have over 200 systems that are going through a manual repair using Windows Recovery Console. Hope you set one up. The process of doing so is to replace update.sys file with the one from Windows 2003 RTM media. Generally located in %systemroot%\system32\drivers 

We are seeing the above on Dell PowerEdge 800, 1800, 1425 systems only running the Standard version of Microsoft Windows 2003 SP1. Identical systems running Windows 2003 Server Enterprise Edition, Web Edition, Small Business Server and R2 editions of Standard & Enterprise do not seem to be affected. Will update as we go along. Working closely with Microsoft and Dell to recover, sorry for the inconvenience.

Comments closed, pings allowed. Believe me, I know what you're going to say "Micro$oft security blows, switch to Linux, switch to Mac, switch to Amiga, Ballmer is evil, this wouldn't have happened on an HP..." There, I saved you 20 minutes.

Disable, uninstall neuter & understand WGA

Have your friends been calling you every few days to check if you found a way around WGA or if you got something other than Devilsown ISO key? If you’re getting your product key out of FILE_ID.DIZ or devilsown.nfo then activating new systems has been a pain in the ass lately. Though not quite as painful as the one thats coming after the feds raid your office and you’re stuck clinching to the cell bars while Bubba works his magic from behind you . But I digress…

There aren’t many happy WGA stories out there. Some consultants are excited because they can make a $50 margin on selling OEM software on after-market PC’s (you’re a national treasure boys) but it seems like everyone I’ve talked to has legitimate Windows XP copies throwing warnings, crashing and really showing many signs of a coder that had to ship the code by the deadline but spent too much time on YouTube downloading behind the scenes of These Boots Are Made For Walking. Sorry folks, it shows.

First of all, there is the Microsoft way to get rid of WGA. It’s not pretty. There is an easier way, too. But if you really want to know how WGA works (down to the TCP dump) here is a site for you:

http://www.firewallleaktester.com/news.htm#60

Awe, but Vlad that page is all words! Don’t despair, these guys also publish a tool to remove WGA in three different ways. Original name too, RemoveWGA. Git’r’gone.

Exchange IMF v2 Operations Guide

IMF v2 operations guide is out and packed with 33 pages worth of goodness for you cheap bastards that won’t pay for ExchangeDefender  Joke aside, very decent entry level effort, and if you’ve got nothing to beat away spam, don’t be lazy, configure this free spam filtering that comes with the Exchange 2003 SP2 for free anyhow. What do you have to lose, you already know you qualify for a Ph.D in Nuclear Physics based on your life experience.

At the very least check out pages 25 – 28, they show you the most common errors and ways to get some monitoring and reporting back from IMF.

To patch a patch

Microsoft releases an update to WSUS, Windows Software Update Services.

Honey, did you remember to patch the patch machine? Well, the SP is out, RTFM then plow away. As for what is in it, here is an overview from the readme:

  • Windows Vista client support: Computers running Windows Vista can be updated by WSUS SP1 Server.
  • More client language support: Support for all Office and Windows Vista languages.
  • New version of WMSDE: The WMSDE instance will be upgraded to WMSDE SP4 by WSUS SP1 (WSUS RTM uses WMSDE SP3).
  • Performance improvements: WSUS SP1 includes various performance improvements to accelerate user interface response times.
  • All hotfixes: WSUS SP1 includes all changes and hotfixes that have been released since WSUS RTM.
  • Support for SQL Server 2005.

For the newbies, RTM means Released to Manufacturing. Same as going “gold” (burned on a CD) and so on and so forth. So go patch yourself.

OneCare Live Ships

Microsoft ships Windows Live OneCare and you might have a free copy waiting for you.

OnecareliveSometimes beta testing, or pretending to, pays off. Such is the case of Windows Live OneCare, the all-in-one security and safety service from Microsoft integrating the antivirus, antispyware, maintenance (defrag) and backup functionality. This is Microsoft’s first significant push into the retail security space with a consumer product, according to the blog, available in dozens of US retail outlets and at http://onecare.live.com site. 

Site seems to be broken so you can’t quite buy it or try it, but suffiice to say this is a preview of Microsoft’s software-as-a-service strategy as this service will run you $49.95 a year. Intended for the home users of course, this is different from the Antigen product I’ve recently been showing you folks at Florida Technet events.

The Joy of Patching

Sometimes I find myself living in the future while the problems of the present still flood my desk. Case and point, last nights patch cycle. I have a lot of servers I'm responsible for pretty much everywhere from continents to data centers down to regional offices we manage. The patching process is, in short, a fun way to test your ironman instincts as you test, plan, deploy and clean up.

I tend to be optimistic about Microsoft and patching software, IE being a huge exception to that since it does not belong on the Internet. It's convenient to just push a few buttons and get things taken care of - when they work. Last night was a not so fun experience with Windows/Microsoft Update and WSUS. Our corporate policy is to test and patch, ask questions later. Here are some of the fun that happened last night:

  1. Sysem just wound not patch. Automatic download + apply + reboot simply did not happen. Is the process running? Yup. Did it download an apply patch according to schedule? Nope.
  2. Windows Update crashing with no really identifiable cause. windowsupdate.log doesn't show anything out of the ordinary.
  3. Why can't SUS client successfully terminate the notepad.exe process? Its as if its immortal. It shuts down everything except notepad as if it is holding the task scheduler in it :)
  4. Windows Update and Microsoft Update just spinning arund with the progress bar completing endlessly with no real progress. No entry in the logs either.
  5. My absolute favorite. Something broke, please reboot and try again. And again. And again.

Now this is nothing new, I've encountered these on previous occasions but they have been very limited. Overnight though, ouch. We've really been giving our WSUS a workout and shifted a lot of the stuff that was not centrally managed through WSUS back into it. Talk about a bad experience encouraging product adoption.

If you've run into these problems please join me today in the Microsoft Technet Webcast covering May 9th Security Bulletins. Starts at 2 PM EST.

Microsoft Patchday: Exchange Troubles

Second Tuesday of the month and yup, you've got patches to deploy. It's a pretty nasty month in patchville with real products getting exploited instead of the usual IE problems. This month it's Exchange on the chopping block:

Microsoft Exchange Server does not properly handle the vCal and iCal properties of email messages. Exploitation of this vulnerability may allow a remote, unauthenticated attacker to execute arbitrary code on an Exchange Server. (CVE-2006-0027)

Unauthenticated attacker to execute arbitrary code. Ouch. Check out the Microsoft Security Bulletin Summary for May 2006 and get to patching.

Roadshow: No, WSUS can't do that

As mentioned earlier, I am on the road presenting a part of the security content for Microsoft Technet seminars and meeting partners and professionals all over the state. It is quite exciting and most of all - insightful. As a service provider I make my bacon fixing problems out there and sometimes you don't hear all the problems through your own feedback channels, you have to see people face to face.

The speach of the day: "WSUS doesn't fix stupid. I guarantee that nowhere in WSUS categories will you find a patch class for stupid user keeps on clicking on things. You can't fix stupid, but you can eliminate the amount of things they can be stupid with."

To put it into some context I opened my portion of the Microsoft event by saying that at least half of all the security problems with the Microsoft software are Microsofts fault. Not in terms of them writing insecure operating systems, but in terms of system administrators not applying service packs, patches and hotfixes in a timely manner. I held a quick poll asking people if they waited a few days to deploy critical patches. A surprising number of hands went up. Well folks, this is why your networks get pwned. If you don't have a clear schedule every second Tuesday of the month and plan to spend an hour or two at the Wednesday Technet webcast covering the patchday... well, update your resume. There is this notion that nobody wants to be the first to blow things up. Ok, fair enough - thats what testing is for folks. You test the service pack, you roll, you call PSS if you must to clean things up, you define a process. You don't stick your head in the sand, trust that Microsoft published every single thing that patch fixed and put your security in the hands of 16 year olds that got nothing to do between TRL and American Idol. Cleaning up a box with a broken patch is a hell of a lot easier than cleaning up a rootkit, if you even notice you've been pwned to begin with.

The other half of the security blame falls squarely on the user. You can patch your servers. You flash your firewalls. You can define strict firewall rules. You can turn up logging to find potential problems. You can backup. You can patch your workstations 15 different ways. But you can't fix stupid. If your junior admin got his MCSE from a paper-mill and his solution to NTFS permission problem is elevating end user security roles you've got a case of stupid going on. Give it up, you won't be able to train your accountant. Or your marketing guy. Or your boss (unless I'm your boss, in which case you can suck it monkeys - thats why I'm on a separate server than the rest of you serfs). They are untrainable. But you can train your staff. You can define a plan. You can set a schedule. You control your stupid exposure.

Security is a process. Do you have a clear one?

More Java Problems for APC

It seems like problems with APC are going to come back to the masses yet again. Remember last year? Well, it looks like strange things are afoot at the Java ranch again. Patch away. This kind of goes a long way toward showing you that appliances are not bullet proof. Sure they may not suffer from the same problems Microsoft operating systems do but they are still written by underpaid programmers and they still need to be managed. Patch away :) APC Security Advisory for PowerChute Business Edition 7.x & PowerChute Network Shutdown 2.2.x Java Runtime Environment Unsigned Applet Privilege EscalationA problem exists with multiple versions of Sun's Java Runtime Environment (JRE) that may allow an unsigned applet to escalate its privileges. PowerChute Business Edition and PowerChute Network Shutdown may install a vulnerable JRE. For PowerChute Business Edition 7.x Users: Download and apply the JRE update patch to all machines running the PCBEagent or server. Official Announcement