I’ve written before about rock stars and IT super heroes. We all know or have worked with someone like this in the past. Perhaps we still do have someone in the organization that fits the description. But have you ever stopped to consider how it could be our culture that breeds the very people we don’t want around?

Keeping The Lights On

When’s the last time you got recognition for the network operating smoothly? Unless it was in response to a huge traffic spike or an attack that tried to knock you offline, the answer is probably never or rarely. Despite the fact that networks are hard to build and even harder to operate, we rarely get recognized for keeping the lights on day after day.

It’s not all that uncommon. The accounting department doesn’t get recognized when the books are balanced. The janitorial staff doesn’t get an exceptional call out when the floors are mopped. And the electric company doesn’t get a gold star because they really did keep the lights on. All of these things are examples of expected operation. When we plug something into a power socket, we expect it to work. When we plug a router in, we expect it to work as well. It may take more configuration to get the router working than the electrical outlet, but that’s just because the hard work of the wiring has already been done.

The only time we start to notice things is when they’re outside our expectation. When the accounting department’s books are wrong. When the floors are dirty. When the lights aren’t on. We’re very quick to notice failure. And, often, we have to work very hard to minimize the culture that lays blame for failure. I’ve already talked a lot about things like blameless post-mortems and other ways to attack problems instead of people. Companies are embracing the idea that we need to fix issues with our systems and not shame our people into submission for things they might not have had complete control over.

Put On The Cape

Have you ever thought about what happens in the other direction, though? I can speak from experience because I spent a lot of time in that role. As a senior engineer from a VAR, I was often called upon to ride in and save the day. Maybe it was after some other company had tried to install something and failed. Or perhaps it was after one of my own technicians had created an issue that needed to be resolved. I was ready on my white horse to ride in and save the day.

And it felt nice to be recognized for doing it! Everyone feels a bit of pride when you are the person to fix an issue or get a site back up and running after an outage. Adulation is a much better feeling than shame without a doubt. But it also beat apathy too. People don’t get those warm fuzzy feelings from just keeping the lights on, after all.

The culture we create that worships those that resolve issues with superhuman skill reinforces the idea that those traits are desirable in engineers. Think about which person you’d rather have working on your network:

• Engineer A takes two months to plan the cutover and wants to make sure everything goes smoothly before making it happen.
• Engineer B cuts over with very little planning and then spends three hours of the maintenance window getting all the systems back online after a bug causes an outage. Everything is back up and running before the end of the window.

Almost everyone will say they want Engineer A working for them, right? Planning and methodical reasoning beats a YOLO attitude any day of the week. But who do we recognize as the rockstar with special skills? Probably Engineer B. Whether or not they created their own issue they are the one that went above and beyond to fix it.

We don’t reward people for producing great Disaster Recovery documentation. We laud them for pulling a 36-hour shift to rebuild everything because there wasn’t a document in the first place. We don’t recognize people that spend an extra day during a wireless site survey to make sure they didn’t miss anything in a warehouse. But we really love the people that come in after-the-fact and spend countless hours fixing it.

Acknowledging Averages

Should we stop thanking people for all their hard work in solving problems? No. Because failure to appreciate true skills in a technical resource will sour them on the job quickly. But, if we truly want to stop the hero worshipping behavior that grows from IT, we have to start acknowledging the people that put in hard work day after day to stay invisible.

We need to give a pat on the back to an engineer that built a good script to upgrade switches. Or to someone that spent a little extra time making sure the site survey report covered everything in detail. We need to help people understand that it’s okay to get your job done and not make a scene. And we have to make sure that we average out the good and the bad when trying to ascertain root cause in outages.

Instead of lauding rock stars for spending 18 hours fixing a routing issue, let’s examine why the issue occurred in the first place. This kind of analysis often happens when it’s a consultant that has to fix the issue since a cost is associated with the fix, but it rarely happens in IT departments in-house. We have to start thinking of the cost of this rock star or white knight behavior as being something akin to money or capital in the environment.

Tom’s Take

Rock star culture and hero worship in IT isn’t going to stop tomorrow. It’s because we want to recognize the people that do the work. We want to hold those that go above and beyond up to those that we want to emulate them. But we should also be asking hard questions about why it was necessary for there to need to be a hero in the first place. And we have to be willing to share some of the adulation with those that keep the lights on between disasters that need heroes.