Angry Vlad

People often wonder what causes things like Vladfire and other acts where I pour more gasoline on the fire. I tend not to share what goes on in my professional life because, honestly, putting this type of stuff in writing hurts. Nobody likes to document their abhorent failures or reinforce that no matter how far we get in documenting and perfecting the process things tend to go horribly wrong.

Such is the case that happened yesterday. Let’s play a little game here of “Guess how this goes wrong”; Here is the scenario, one of the customers servers needs to be reprovisioned as they are ready to move to MySQL 5 and start implementing their old spaghetti php-mysql code as stored sql procedures.

Vlad: I would like to have one of the servers reimaged with CentOS 4.3, minimal install. The box in question is a small Dell SC420 server in … rack, it is the only one of the kind. It is labeled as … as far as I recall

Make the system one big ext3 partition filling up all available space (- swap of course)

Username: root
Password: …

Disable firewall
Disable SeLinux
Grub Boot Manager

IPADDR=…
NETMASK=255.255.255.128
GATEWAY=…

Connected to the … switch.

Please update the ticket when you unplug the server before you proceed - I want to make sure that the correct server is being reimaged and not a production one.

Now if you’re in IT you can probably see where this one is heading. So here is Marlon’s response:

Marlon: We will begain the install now and update you once the work is complete.

Thanks!

Ok, sounds like he didn’t read the last part of the ticket, doesn’t it? Maybe I’m misinterpreting what “update the ticket when you unplug the server before you proceed” – perhaps I’m also misinterpreting what “once the work is complete” means too, but I’m smart enough to clear it up. Here is my response:

Vlad: I just want to make sure the correct box is pulled off the rack - as far as I can tell this system is still online.

Now certainly Marlon will read this part and respond with something along the lines of  “Not a problem, will update the ticket when I unplug it and before the imaging starts” but instead I get one of these:

Marlon: Currently, we do not have CentOS 4.3 on hand. WOuld you like me to proceed with CentOS 4.2?

Am I talking to the wall here?

Vlad: Absolutely -- but the server I wanted you to reimage is still online. Have you pulled it off the rack yet? I have to make sure that the correct one is pulled off because I no longer remember the label that was placed on it.

Now certainly at this point he understands what needs to be done here, case closed. Right? Right?

Kevin: Vlad,
The server should now be down. OS installation is in progress.

Umm. Not quite.

[root@… ~]# uptime
 14:29:11 up 20 days, 23:39,  0 users,  load average: 0.00, 0.00, 0.00–

Yup, They reimaged a production server. What more can you possibly say?

23 comments on “Angry Vlad”

  1. Allen says:

    I always wondered why Vlad lives in Orlando and his data centers are nowhere near him. Now I know.

    Whats really sad is that Marlon and Kevin don't sound like dots. I'd expect that from India but Americans????

  2. Terry says:

    Can we have Marlon and Kevin's email addresses (are they Marlon@supergeek.com and Kevin@hotmail.com?) so that we can send abuse to them "by proxy" for you?

    One x DOS (denial of sanity) attack coming up!

    Don't let them charge you double when they do it for real...

  3. Aaron says:

    Obviously the person you're talking to is selectively reading, so by you putting "absolutely" as the first word in your response, he probably took that and ran.

  4. cscriber says:

    Ouch that hurts. I do appreciate your honesty though which is why we do so much business with OWN. It's hard to find someone that is so up front about what happens and so responsive. If this happened to any other company you would NEVER hear about it unless the customer flamed on the newsgroups.

    For an owner of the company to come out in public and talk about something that you never would have noticed.. much respect.

  5. DonMurphy says:

    Um what is a dot?

    Vlad don't forget that everyone learns in their own way. Kind of like the school of hard knocks when it comes to disaster recovery and backups.

    I bet you he will be more carefull next time and it has hopefully made him a better tech for it.

  6. Errol says:

    Fuck that, I hope he fired him. I don't want people like that anywhere near my servers and I know how much care Vlad puts in them to let someone like that touch stuff. It's almost hard to believe.

  7. Tony says:

    OK, IF they still work for you, then I suggest the next time you have them call you before you give them ANY details. The conversation should go something like this:

    Is this head Slacker #1? It's Vlad. I'm about to open a ticket for you to reimage a server, I want you to trundle on over to the rack and find the one I want you to work on. It's a Dell SC420, the only one there and I think it's labeled blah blah blah. Put me on hold and go check...

    Ok, did you find it? What's it labeled? {search memory banks to see if it sounds right}. Ok, now go unplug it and come right back {wait 30 seconds and commence ping to check if numbnuts found the right one}.

    OK Skippy, you got the right one, now go and check your ticket console and I'll give you the details.

    ===========

    Three words....Lowest Common Denominator. Your problem is you have to think like an idiot. BTW, are you hiring? :-)

  8. CharlesM says:

    Its posts like this that finally make me realize what language Vlad speaks. His term

    "choking distance"

    really takes on a new meaning as to why he doesn't work near his DCs.

    ROFL. The book of Vladisms.

  9. ScottInFlorida says:

    An amateur might snicker at that story and say, "that's funny". I guess I don't share the humor.

    The cascading mess of crap that comes out of a screw up like that takes a toll on everyone in the IT food chain. The IT guy, (Vlad), the customer that now has no server or application front end, and the end user now has nothing to do but endure another World Cup tie, (what's up with ties in soccer??).

    By the time this is over Vlad will have a 100 hours in this.

    Good luck buddy...

    S>

  10. Ken Edwards says:

    Well, if Scott is going to comment...

    I expect to get yelled at, but...

    I was told as a young field tech, that everyone gets on major screwup. I blew out the entire register system at a busy restaurant at lunch by shifting a power transformer into the edge of a motherboard. That register arched and the rest fell like dominoes. I was not injured, at least physically. They had to call out every field tech at the service center to get the system up quickly. Looked like lots of circus clowns getting out of tiny cars. Either I had a very understanding boss, or he saw something in me worth keeping on the payroll.

    The jest of the story is that I feel sorry of the guys that screwed up as it only takes a moment of inattention to really screw up... I feel every sorrier for Vlad as he has to face the client...

    -Ken

  11. DDD2 says:

    This is certainly a familiar scenario so I can sympathize. Fortunately in my case it's isn't a reload, it's just an ill timed reboot.

    There is no excuse for this kind of blatant carelessness. Is this a learning experience for the Jr. techie? Certainly. Will that customer think twice the next time Vlad says "Sure, I'll take care of that server reload, no problem"? I bet they will.

    Let's just hope the customer is forgiving.
    -ddd2

  12. vlad says:

    I'm glad I can make some of you laugh.

    Thankfully, the system that was blown up was one of our redundant systems so the downtime was nothing that would have been noticed by any of our customers. It was up and running within an hour. If this were one of our customers servers, even worse, if this were a customers colo server that was NOT backed up, we would have a lawsuit on our hands.

    There is such a thing as a learning experience, it is for those of us that solve problems. There is also such a thing as gross negligence and termination, for those of us who CAUSE problems. As you can tell by the trouble ticket log, this wasn't a single ommission, or a tiny error, or a clerical mistake - it was complete disregard for three ticket updates. I could have excused one, or perhaps two.. but at some point I have to face my customers and tell them that we do the absolutely best we possibly can.

    Ignoring ticket updates is NOT acceptable, no matter how fast the environment we are in. I've always prided myself and my company on doing the job right, even turning down sales orders to keep the existing customer base happy. This cannot and will not be compromised as long as I run this company.

    -Vlad

  13. vlad says:

    Reading through the comments...

    Dot = derrogative for Indian tech support.

    The phone call comment from Tony, although a good suggestion, we handle over 1,000 requests a day and I go through something like 2,000 - 3,000 messages on average. Spending time on the phone to double-validate each ticket is simply not possible. Also, there are data centers where there is no phone / cell service available.

    Tim, the server was delivered with 4.3 but they botched the paritioning and rolled it out as sever install instead of minimal that was requested.

    Ken, you can't compare your moment of carelessness to the three incidents of NOT reading the request at all. By comparison, you would have to physically damage each of the cash registers while the people around you kept on prodding "Are you sure you won't damage the cash register" as you went forward and broke one after another.

    When things like this happen you document, analyze and you move on.

    -Vlad

  14. vlad says:

    No customers were harmed in the imaging of the server. Unfortunately, the original customers imaging did take 6 hours longer than it should have but all in all, I didn't have to apologize to anyone.

    -Vlad

  15. Tony says:

    Oi! I can't imagine 300 messages, much less than 3K.

    You're right, no way you can call on that many, even if every center had a phone.

  16. Christopher L says:

    The first comment got it right --

    "You're fired."

    If you want to be creative bring both of them in the room then turn to the one that actually blew it up and say:

    "You are the weakest link. Goodbye."

  17. Albert says:

    I face people not reading all day long. When you take the time to document a request, it should be paid attention to especially when it comes from competent senior staff.

    I deal with so called experts and they too make mistakes and require teaching. The human factor must always be calculated into everything. Having the right recovery and contingency plan is a must.

    I run a datacenter for financial transaction at the billion dollar mark. If plans were not in place failure would be eminent. Chastising people for failure or mistakes would leave the workforce as skeleton. Patience is a virtue and error recover is what separates men from boys.

    My hat is off to you Vlad. Keep up the good work!

  18. Ken Edwards says:

    I understand, Vlad.

    THREE missed references is gross negligence.

    I still feel sorry for the screw-er-upper. They thought they knew what you wanted and did not read any deeper. While not in IT matters, I am guilty of this in other areas, on occasion, sometimes, when half asleep, or in hurry to enjoy a new toy, yada yada yada...

    -Ken

  19. ScottInFlorida says:

    Sorry to post so late in the day but I find this thread fascinating.

    Ken Edwards, yes, I too have fubar'ed a job, and more than once.

    Just one example, I was working on a brand new computer in a hotel in Portland, Maine.

    I was in a hurry and inserted an ISA modem card into a brand new Dell Pentium computer, (years and years ago). Did I mention the computer was booted up and running?

    Okay, so I stick the card in and instantly a streak of flame and blue smoke comes cascading out of the computer. The manager of the hotel says, "I hope the smoke alarm doesn't go off.", then less than two seconds later the fire alarm goes off. It was just after lunch so there weren't many guests, but one guy comes out of his room, in his underwear, and in a panic pulls the red fire alarm thing.

    You got it, water everywhere but only on the first floor. (Hotel was next to the Maine Mall, next to the airport.)

    Yes, I go fired. Insurance that my employer had covered the damage, but I learned a pretty big lesson.

    So yes Ken, techies do learn from mistakes and maybe they learn more when negative consequences happen.

    In closing, I played golf with my boss that fired me that day just about a month ago down in Venice, Florida. We laugh about it now but that particular day was a bad one for all of us.

    S>

  20. Hormel says:

    Vlad, your circus monkeys need a 3-ring binder with detailed steps. Expecting them to decypher cryptic notes in in the trouble ticket is only going to lead to more problems.

    BTW, usually the first step of a procedure is placed at the beginning of a document, not the end. LOL

Historical discussion. New comments are closed for this post.