We’re all in IT. We’ve done our time in the trenches. We’ve…seen things, as Roy Batty might say. Things you wouldn’t believe. But in the end we all know the pain of trying to get support for something that we’re working on. And we know how painful that whole process can be. Yet, how is it that support is universally “bad” in our eyes?

One Of Us

Before we launch into this discussion, I’ll give you a bit of background on me. I did inbound tech support for Gateway Computers for about six months at the start of my career. So I wasn’t supporting enterprises to start with but I’ve been about as far down in the trenches as you can go. And that taught me a lot about the landscape of support.

The first thing you have to realize is that most Tier 1 support people are, in fact, not IT nerds. They don’t have a degree in troubleshooting OSPF or are signatories to the fibre channel standards. They are generally regular people. They get a week or two of training and off they go. In general the people on the other end of the support phone number are better phone people than they are technical people. The trainers at my old job told me, “We can teach someone to be technical. But it’s hard to find someone who is pleasant on the phone.”

I don’t necessarily agree with that statement but it seems to be the way that most companies staff their support lines. Why? Ask yourself a quick question: How many times has Tier 1 support solved your issue? For most of us the answer is probably “never” or “extremely rarely”. That’s because we, as IT professionals, have spent a large amount of our time and energy doing the job of a Tier 1 troubleshooter already. We pull device logs and look at errors and Google every message we can find until we hit a roadblock. Then it’s time to call. However, you’re already well past what Tier 1 can offer.

Look at it from the perspective of the company offering the support. If the majority of people calling me are already past the point of where my Tier 1 people can help them, why should I invest in those people? If all they’re going to do is take information down and relay the call to Tier 2 support how much knowledge do they really need? Why hire a rock star for a level of support that will never solve a problem?

In fact, this is basically the job of Tier 1 support. If it’s not a known issue, like an outage or a massive bug that is plastered up everywhere, they’re going to collect the basic information and get ready to pass the call to the next tier of support. Yes, it sucks to have to confirm serial numbers and warranty status with someone that knows less about VLANs than you do. But if you were in the shoes of the Tier 2 technician would you want to waste 10 minutes of the call verifying all that info yourself? Or would you rather put your talent to use to get the problem solved?

Think about all the channel partners and certification benefits that let you bypass Tier 1 support. They’re all focused on the idea that you know enough about the product to get to the next level to help you troubleshoot. But they’re also implying that you know the customer’s device is in warranty and that you need to pull log files and other data before you open a ticket so there’s no wasted time. But you still need to take the time to get the information to the right people, right? Would you want to start troubleshooting a problem with no log files and only a very basic description of the issue? And consider that half the people out there just put “Doesn’t Work” or “Acting Weird” for the problem instead of something more specific.

Sight Beyond Sight

This whole mess of gathering info ahead of time is one of the reasons why I’m excited about the coming of network telemetry and network automation. That’s because you now have access to all the data you need to send along to support and you don’t need to remember to collect it. If you’ve ever logged into a switch and run the show tech-support command you probably recall with horror the console spam that was spit out for minutes upon minutes. Worse yet, you probably remember that you forgot to redirect that spam to a file on the system to capture it all to send along with your ticket request.

Commands like show tech-support are the kinds of things that network telemetry is going to solve. They’re all-in-one monsters that provide way too much data to Tier 2 support techs. If the data they need is in there somewhere it’s better to capture it all and spend hours poring through a text file than talk to the customer about the specific issue, right?

Now, imagine the alternative. A system that notices a ticket has been created and pulls the appropriate log files and telemetry. The system adds the important data to the ticket and gets it ready to send to the vendor. Now, instead of needing to provide all the data to the Tier 1 folks you can instead open a ticket at the right level to get someone working on the problem. Mean Time to Resolution goes down and everyone is happy. Yes, that does mean that there are going to be fewer Tier 1 techs taking calls from people that don’t have a way to collect the data ahead of time but that’s the issue with automating a task away from someone. If the truth be told they would be able to find a different job doing the same thing in a different industry. Or they would see the value of learning how to troubleshoot and move up the ladder to Tier 2.

However, having telemetry gathered automatically still isn’t enough for some. The future is removing the people from the chain completely. If the system can gather telemetry and open tickets with the vendor what’s stopping it from doing the same proactively? We already have systems designed to do things like mail hard disk drives to customers when the storage array is degraded and a drive needs to be replaced. Why can’t we extend things like that to other parts of IT? What if Tier 2 support called you for a change and said, “Hey, we noticed your routing protocol has a big convergence issue in the Wyoming office. We think we have a fix so let’s remote in and fix it together.” Imagine that!

Tom’s Take

The future isn’t about making jobs go away. It’s about making everything more efficient. I don’t want to see Tier 1 support people lose their jobs but I also know that most people consider their jobs pointless already. No one wants to deal with the info takers and call routers. So we need to find a way to get the right information to the people that need it with a minimum of effort. That’s how you make support work for the IT people again. And when that day comes and technology has caught up to the way that we use support, perhaps we won’t need to defend them anymore.