Everyone in IT has been there before. The core switches are melting down. The servers are formatting themselves. Packets are being shuffled off to their doom. Human sacrifce, dogs and cats living together, mass hysteria. You know, the usual. What happens next?
Strangely enough, how IT people react to stressful situations such as these has become a rather interesting study habit of mine. I know how I react when these kinds of things start happening. I go into my own “panic mode”. It’s interesting to think about what changes happen when the stress levels get turned up and problems start mounting. I start becoming short with people. Not yelling or screaming, per se. I start using short declarative sentences at an elevated tone of voice to get my point across. I being looking for solutions to problems, however inelegant they may be. Quick fixes rule over complicated designs. I’ve trained myself to eliminate the source of stress or the cause of the problem. I tend to tune out any other distractions until the issues at hand are sorted out. Should I find myself in a situation where I can effect a solution to the problem, or if I’m waiting on someone or something to happen outside my directly control, that is the time when the stress starts mounting. To those that share my “can do” attitude, this makes me look efficient and helpful in times of crisis. To others, I look like a complete jerk.
I’ve also found that there are others in IT (and elsewhere) that have an entirely different method of dealing with stress: they shut down. My observations have shown that these people become overwhelmed with the pressure of the situation almost immediately and begin finding ways to cope through indirect action. Some begin blaming the problem on someone or something else. Rather than search out the source of the trouble, they try to pin it on someone other than them, maybe in the hopes they won’t have to deal with it. These people begin to withdraw into their own world. They sit down and stare off into space. They become quiet. Some of them even break down and start to cry (yes, I’ve seen that happen before). Until the initial shock of the situation has passed, they find themselves incapable of rendering any kind of assistance.
How do we as IT professionals deal with these two disparate types of panic modes? You need to work out how to do that now so that you don’t have to come up with things on the fly when the core switches are dropping packets and the CxOs are screaming for heads, which is funny that the second category of blamers and inaction people always seem to be in management.
For people like me, the “doers”, we need to be doing something that can impact the problem. No busy work, no research. We need to be attacking things head-on. Any second we aren’t in attack mode compounds the stress we’re under. Even if we try a hundred things and ninety nine of them fail, we have to try to keep from going crazy. Think of these “doers” like a wind-up toy: get us working on something and let us go. You might not want to be around us while we’re working, lest you want some curt answers followed by looks of distaste when we have to stop and explain what we’re doing. We’ll share…when we’re done.
For the other type of people, those that have a stress-induced Blue Screen of Death (BSoD), I’ve found that you have to do something to get them out of their initial funk. Sometimes, this involves busy work. Have them research the problem. Have them go get coffee. In most cases, have them do something other than be around you while you’re troubleshooting. Once you can get them past the blame/sulk/cry state, they can become a useful resource for whatever needs to happen to get the problem solved. Usually, they come back to me later and thank me for letting them help. Of course, they also usually tell me I was a bit of an ass and should really be nicer when I’m in panic mode. Oh well…
Tom’s Take
I don’t count on anyone in a stressful situation that isn’t me. Most often, I don’t have the luxury of time to figure out how a person is going to react. If you can help me I’ll get you doing something useful. If not, I’m going to ignore or marginalize you until the problem is fixed. Over the last couple of years, though, I’ve found that I really need to start working with every different group to ensure that communications are kept alive during stressful situations and no one’s feelings get hurt (even though I don’t normally care). By consciously realizing that people generally fall into the “doer” or “BSoD” category, I can better plan for ways to utilize them when the time comes and make sure that the only thing going CRUNCH at crunch time is the problem. And not someone’s head.
The Networking Nerd 
” I start becoming short with people. Not yelling or screaming, per se. I start using short declarative sentences at an elevated tone of voice to get my point across…To those that share my “can do” attitude, this makes me look efficient and helpful in times of crisis. To others, I look like a complete jerk.”
That’s me all the way. Gotten in (mild) trouble multiple times because of my “tone” during a crisis. I’m not screamy or over-dramatic. Just very to the point, because there’s a problem to be fixed. Let’s dispense with the pleasantries, and tell me what I need to know so that the problem can get resolved as quickly as possible. If you don’t have the answer, kindly move aside and let me get it sorted. This is not the time for a conference room chat, whiteboard pontification, or a group hug.
I bet surgeons don’t get criticized when the patient is dying on the table and they get curt in an effort to save a life. Did I just say network engineers are like surgeons? Yeah…I think I did. 🙂
If network engineers are surgeons, then most of them must specialize in plastic surgery based on some of the networks we have probably all seen. I say that as someone who has made several bad design choices in the past. However, I would like to think that I no longer leave my patients(networks) looking like Joan Rivers.
To your point though, I too tend to get curt with others during a crisis. I start out civil, but after a certain time period, I lose patience with others and want to start breaking things.
@ecbanks: “This is not the time for […] a group hug”
I honestly hope there’s never a time for group hugs at work.
I love the quote btw.. “What he means is Old Testament, Mr. Mayor, real wrath of God type stuff.
Fire and brimstone coming down from the skies! Rivers and seas boiling!
Forty years of darkness! Earthquakes, volcanoes…
The dead rising from the grave!
Human sacrifice, dogs and cats living together… mass hysteria! ”
But seriously I have to agree with Ethan here, people honestly need to understand that this is an “emergency” and that you really don’t have the time to be civil and PC. If you don’t have business being around, then leave. I don’t need 10 people looking over my shoulders as I type, as I’m sure a surgeon doesn’t want a someone breathing down his neck in the ER.
I had a similar experience a few years ago at a small,growing into a mid-size company. The company decided to in source email from a cloud provider to in-house Exchange. The migration ran into a few glitches, our ‘consultant’ had several “Oh and I forgot to tell you” moments and upper management went
monkey droppingsbananas :). I got into my crisis-management mode–short but polite answers to the emails and calls every 10 minutes from my C level boss, and put my head down and pushed through the crisis for 36 hours without a break. A lot of people realized that what I was doing was more important to the company than fixing their browser issues, but a few (including my boss) did not. He didn’t understand why giving him “status updates” every 10 minutes slowed down the process, and he wanted to demonstrate to his boss how hard he was working to put out the fire. I told him the issue would be fixed faster with hourly updates, but that didn’t go over too well! “The foreign office can’t access BES server”, he said. My response was short and sweet: “Sir, I am trying to fix email for the entire company, including the CEO, BES will have to be on the back burner for now!”Manish Chacko
Technical Marketing Manage, Solarwinds
Great subject, Tom.
The absolute worst is when folks panic and just start pulling cables or yanking pieces of the configuration out with no regard to the cause… I call these folks “dart board panickers” or “deconstructors”. I have had 6 months of changes removed by these types. Usually, these are genuinly nice folks, and it’s the worst when you have to throw them under the bus during your outage debriefing.
As for attitude during an outage event, I try to keep things as light as possible- smile, drop a joke, whatever conveys a sense that the problem is under control. The second stress is conveyed, the deconstructors go to work because they think you are as lost as them. I try to get the deconstructors off to do a menial task like packet captures or something. Preferably in an IDF in the basement or somewhere where they can do as little damage as possible.
It also helps to have a tech manager or lead that can act as a communication buffer between you and the CxO’s, the help desk and the users standing outside your door with torches and clubs. Nothing elevates stress more than having each of your 6 bosses coming in every 5 minutes for a detailed status report.
Loved the article. I myself experienced a BSoD moment the first time we had an “emergency” and almost was sucked into the vortex. I now have learnt how easy others can bring you down. Those that aren’t of use get out of the way. Niceties can be resumed in a post-outage analysis.
Pingback: BPDU the next incrementation | Router Jockey