Keynote Hate – Celebrity Edition

We all know by now that I’m not a huge fan of keynotes. While I’ve pulled back in recent years from the all out snark during industry keynotes, it’s nice to see that friends like Justin Warren (@JPWarren) and Corey Quinn (@QuinnyPig) have stepped up their game. Instead, I try to pull nuggets of importance from a speech designed to rally investors instead of the users. However, there is one thing I really have to stand my ground against.

Celebrity Keynotes.

We’ve seen these a hundred times at dozens of events. After the cheers and adulation of the CEO giving a big speech and again after the technical stuff happens with the CTO or product teams, it’s time to talk about…nothing.

Celebrity keynotes break down into two distinct categories. The first is when your celebrity is actually well-spoken and can write a speech that enthralls the audience. This means they get the stage to talk about whatever they want, like their accomplishments in their career or the charity work their pushing this week. I don’t mind these as much because they feel like a real talk that I might want to attend. Generally the celebrity talking about charity or about other things knows how to keep the conversation moving and peppers the speech with anecdotes or amusing tales that keep the audience riveted. These aren’t my favorite speeches but they are miles ahead of the second kind of celebrity keynote.

The Interview.

These. Are. The. Worst. Nothing like a sports star or an actor getting on stage for 45 minutes of forced banter with an interviewer. Often, the person on stage is a C-level person that has a personal relationship with the celebrity and called in a favor to get them to the event. Maybe it’s a chance to talk about their favorite charity or some of the humanitarian work they’re doing. Or maybe the celebrity donated a bunch of their earnings to the interviewer’s pet project.

No matter the reasons, most of these are always the same. A highlight reel of the celebrity in case someone in the audience forget they played sports ball or invented time travel. Discussion of what they’ve been doing recently or what their life was like after they retired. A quirky game where the celebrity tries to guess what they company does or tries to show they know something about IT. Then the plug for the charity or the fund they’re wanting to really talk about. Then a posed picture on stage with lots of smiles as the rank and file shuffle out of the room to the next session.

Why is this so irritating? Well, for one, no one cares what a quarterback thinks about enterprise storage. Most people rarely care about what the CEO thinks about enterprise storage as long as they aren’t going to shut down the product line or sell it to a competitor. Being an actor in a movie doesn’t qualify you to talk about hacking things on the Internet any more than being in Top Gun qualifies you to fly a fighter jet. Forcing “regular” people to talk about things outside their knowledge base is painful at best.

So why do it then? Well, prestige is one thing. Notice how the C-level execs flock to the stage to get pics with the celebrity after the speech? More posters for the power wall in their office. As if having a posed pic with a celebrity you paid to come to your conference makes you friends. Or perhaps its a chance to get some extra star power later on for a big launch. I can’t tell you the number of times that I’ve made a big IT purchasing decision based on how my favorite actor feels about using merchant silicon over custom ASICs. Wait, I can tell you. Hint, it’s a number that rhymes with “Nero”.

Getting Your Fix

As I’ve said many times, “Complaining without a solution is whining.” So, how do we fix the celebrity keynote conundrum?

Well, my first suggestion would be to get someone that we actually want to listen to. Instead of a quarterback or an actor, how about instead we get someone that can teach us something we need to learn? A self-help expert or a writer that’s done research into some kind of skill that everyone can find important? Motivational speakers are always a good touch too. Anyone that can make us feel better about ourselves or let us walk away from the presentation with a new skill or idea are especially welcome.

Going back to the earlier storyteller keynotes, these are also a big draw for people. Why? Because people want to be entertained. And who better to entertain than someone that does it for their job? Instead of letting some C-level exec spend another keynote dominating half the conversation, why not let your guest do that instead? Of course, it’s not easy to find these celebrities. And more often than not they cost more in terms of speaker fees or donations. And they may not highlight all the themes of your conference like you’d want with someone guiding them. But I promise your audience will walk away happier and better off.


Tom’s Take

Keynotes are a necessary evil of conferences. You can’t have a big event without some direction from the higher-ups. You need to have some kind of rally where everyone can share their wins and talk about strategy. But you can totally do away with the celebrity keynotes. Instead of grandstanding about how you know someone famous you should do your attendees a favor and help them out somehow. Let them hear a great story or give them some new ideas or skill to help them out. They’ll remember that long after the end of an actor’s career or a sports star’s run in the big leagues.

I Was A 10x Engineer. And I’m Sorry.

You probably saw the big discussion this past weekend on Twitter about 10x Engineers. It all started with a tweet about how to recognize a 10x Engineer, followed by tons of responses about how useless they were and how people that had encountered them were happy to be rid of them. All that discussion made me think back to my old days as a Senior Network Rock Star. As I reminisced I realized that I was, in fact, a 10x Engineer. And I was miserable.

Pour Some Work On Me

I wasn’t always the epitome of engineering hatred. I used to be a wide-eyed technician with a hunger to learn things. I worked on a variety of systems all over the place. In fact, I was rising through the ranks of my company as a Novell Engineer in an environment with plenty of coverage. I was just learning the ropes and getting ready to take my place in a group of interchangeable people.

Then I started getting into networking. I spent more time learning about routers and switches and even firewalls. That meant that my skill set was changing from servers to appliances. It also meant that I was spending more and more time working on devices that no one else could work on. I had special knowledge that made me much more valuable to the organization. Soon, I found myself spending less and less time working on the Novell command line and more time working on the Cisco CLI.

That was the first extra “x” on my resume. Because I had special skills it meant that I was being relied upon more to do work that no one else could do. Suddenly I wasn’t just a replaceable cog in the machine. Instead, I was a critical part of the infrastructure that needed to be on-site for certain jobs and deployments. I knew that I needed to have someone else to help me out or I was going to quickly find myself overwhelmed with work. But networking wasn’t the thing that ended up pushing me all the way to 10x territory.

A Voice In The Wilderness

In order to truly become an insufferable 10x Engineering talent, I had to pivot into voice deployments. That’s because my skill set went from “important but we’re training others” to “so complicated no one else can understand this”. Thanks to my knowledge of networking, I was asked to pick up the voice banner and run with it. And I ran really, really far.

I was the only person in the office working on Cisco voice deployments. I had my own method of doing things. My own flow for deployments. My notes were contained in a OneNote file that only I had access to. You can probably see all the issues already. But I couldn’t. To me, this was my jam! I had all the tools and talent to make this happen. I could type MAC addresses faster than filling them into a BAT spreadsheet. I could configure crazy hacks to get around limitations thanks to all the extra research I was doing. I was invincible!

I was also gumming up the works. Voice deployments had to be constantly rescheduled if I was out of town. Vacations were a distant memory at best. I wasn’t just the most important cog in the machine – I was the machine. Nothing went forward unless I was doing it. And that’s not scalable at all. Even when my boss realized that I couldn’t scale any more he also knew that getting someone up to speed on my deployment methods and knowledge was a very daunting task at best.

This is the classic setup for a 10x Engineer. All the talent in the world and all of the hubris. Large portions of the line-of-business relying on their knowledge and process. No documentation. No way to get things done other than to go through me. If you’ve ever read The Phoenix Project by Gene Kim you realize that I was Brent through and through. In fact, when I finally did get around to reading it I self-identified with him before I’d even made it 100 pages in.

Pride Goeth Before

Ultimately, it wasn’t failure that caused my 10x Engineering career to come to an end. Instead, it was success. I left my old job six years ago to come to Tech Field Day. I knew I wouldn’t be fixing routers or voicemail systems any longer. I’d be in for an entirely new and different kind of work.

But I couldn’t see how my old job would be able to work without me. I secretly confided to some friends that I thought their business would fall apart without me. How could it survive? I was responsible for so much! I was the only one that knew all of these things! Even a lengthy two-week info dump wouldn’t be enough.

The pride and hubris I displayed is still shocking to me all these years later. To think that a company that had been in business for almost 20 years before I got there would go out of business months after I left because they couldn’t do what I was doing. They did stay in business. They changed, for sure. They moved away from my specialized knowledge and found new ways to utilize their talent. They were able to keep my systems up and running with my notes and when the time came to replace them they used new systems that didn’t need my help to manage and install. They survived because they realized that what I represented was special and couldn’t be replicated.

Me? I’m happy they did. I didn’t want my 10x Engineering efforts to be an anchor around either of our necks. I couldn’t go back to fixing my old systems with my new workload. And my old company couldn’t count on me being available to fix things when I was gone. They made the right choices to put themselves into a position to keep going. And just like most positive 10x Engineer stories they found a happy ending.


Tom’s Take

I’m not proud of my engineering roots when it comes to how bad I was at times. It wasn’t always intentional. It was a product of where I was and the work I was doing. But I totally couldn’t see the forest for the trees. I realize now that I should have taken more people under my wing and helped them understand what I was doing. I should have documented my work and used repeatable methods to build processes that could be done by anyone. My institutional knowledge should have been a resource, not a crutch. And I should have had the humility to understand that companies can live and grow past a single engineer. Knowing all that today makes me realize that I may have looked like a 10x Engineer but everyone is much better off now that I’m just a simple ex-engineer.

Tale From The Trenches: The Debug Of Damocles

My good friend and colleague Rich Stroffolino (@MrAnthropology) is collecting Tales from the Trenches about times when we did things that we didn’t expect to cause problems. I wanted to share one of my own here about the time I knocked a school offline with a debug command.

I Got Your Number

The setup for this is pretty simple. I was deploying a CallManager setup for a multi-site school system deployment. I was using local gateways at every site to hook up fax lines and fire alarms with FXS/FXO ports for those systems to dial out. Everything else got backhauled to a voice gateway at the high school with a PRI running MGCP.

I was trying to figure out why the station IDs that were being send by the sites weren’t going out over caller ID. Everything was showing up as the high school number. I needed to figure out what was being sent. I was at the middle school location across town and trying to debug via telnet. I logged into the router and figured I would make a change, dial my cell phone from the VoIP phone next to me, and see what happened. Simple troubleshooting, right?

I did just that. My cell phone got the wrong caller ID. And my debug command didn’t show me anything. So I figured I must be debugging the wrong thing. Not debug isdn q931 after all. Maybe it’s a problem with MGCP. I’ll check that. But I just need to make sure I’m getting everything. I’ll just debug it all.

debug mgcp packet detail

Can You Hear Me Now?

Veterans of voice are probably screaming at me right now. For those who don’t know, debug anything detail generates a ton of messages. And I’m not consoled into the router. I’m remote. And I didn’t realize how big of a problem that was until my console started scrolling at 100 miles an hour. And then froze.

Turns out, when you overwhelm a router CPU with debug messages, it shuts off the telnet window. It also shuts off the console as well, but I wouldn’t have known that because I was way far way from that port. But I did starting hearing people down the hall saying, “Hello? Hey, are you still there? Weird, it just went dead.”

Guess what else a router isn’t doing when it’s processing a tidal wave of debug messages? It’s not processing calls. At all. For five school sites. I looked down at my watch. It was 2:00pm. That’s bad. Elementary schools get a ton of phone calls within the last hour of being in session. Parents calling to tell kids to wait in a pickup line or ride a certain bus home. Parents wanting to check kids out early. All kinds of things. That need phones.

I raced out of my back room. I acknowledged the receptionists comment about the phones not working. I jumped in my car and raced across town to the high school. I managed not to break any speed limits, but I also didn’t loiter one bit. I jumped out of my car and raced into the building. The look on my face must have warded off any comments about phone system issues because no one stopped me before I got to the physical location of the voice gateway.

I knew things were bad. I didn’t have time to console in and remove the debug command. I did what ever good CCIE has been taught since the beginning of time when they need to remove a bad configuration that broke their entire lab.

I pulled the power cable and cycled the whole thing.

I was already neck deep in it. It would have taken me at least five minutes to get my laptop ready and consoled in. In hindsight, that would have been five wasted minutes since the MGCP debugger would have locked out the console anyway. As the router was coming back up, I nervously looked at the terminal screen for a login prompt. Now that the debugger wasn’t running, everything looked normal. I waiting impatiently for the MGCP process to register with CallManager once more.

I kept repeating the same status CLI command while I refreshed the gateway page in CallManager over and over. After a few more tense minutes, everything was back to normal. I picked up a phone next to the rack and dialed my cell phone. It rang. I was happy. I walked back to the main high school office and told them that everything was back to normal.

Tom’s Take

My post-mortem was simple. I did dumb things. I shouldn’t have debugged remotely. I shouldn’t have used the detail keyword for something so simple. In fact, watching my screen fill up with five sites worth of phone calls in a fraction of a second told me there was too much going on behind the scenes for me to comprehend anyway.

That was the last time I ever debugged anything in detail. I made sure from that point forward to start out small and then build from there to find my answers. I also made sure that I did all my debugging from the console and not a remote access window. And the next couple of times I did it were always outside of production hours with a hand ready to yank the power cable just in case.

I didn’t save the day. At best, all I did was cover for my mistake. If it had been a support call center or a hospital I probably would have been fired for it. I made a bad decision and managed to get back to operational without costing money or safety.

Remember when you’re doing your job that you need to keep an eye on how your job will affect everything else. We don’t troubleshoot in a vacuum after all.

Predictions As A Service

It’s getting close to the end of the year and it’s time once again for the yearly December flood of posts that will be predicting what’s coming in 2018. Long time readers of my blog know that I don’t do these kinds of posts. My New Year’s Day posts are almost always introspective in nature and forward looking from my own personal perspective. But I also get asked quite a bit to contribute to other posts about the future. And I wanted to tell you why I think the prediction business is a house of cards built on quicksand.

The Layups

It’s far too tempting in the prediction business to play it safe. Absent a ton of research, it’s just easier to play it safe with some not-so-bold predictions. For instance, here’s what I could say about 2018 right now:

  • Whitebox switching will grow in revenue.
  • Software will continue to transform networking.
  • Cisco is going to buy companies.

Those are 100% true. Even without having spent one day in 2018. They’re also things I didn’t need to tell you at all. You already knew them. They’re almost common sense at this point. If I needed to point out that Cisco is going to buy at least two companies next year, you are either very new to networking or you’ve been studying for your CCIE lab and haven’t seen the sun in eight months.

Safe predictions have a great success rate. But they say nothing. However, they are used quite a bit for the lovely marketing fodder we see everywhere. In three months, you could see presentation from an SD-WAN vendor that says, “Industry analyst Tom Hollingsworth predicts that 2018 is going to be a big year for software networking.” It’s absolutely true. But I didn’t say SD-WAN. I didn’t name any specific vendors. So that prediction could be used by anyone for any purpose and I’d still be able to say in December 2018 that I was 100% right.

Playing it safe is the most useless kind of prediction there is. Because all you’re doing is reinforcing the status quo and offering up soundbites to people that like it that way.

Out On A Limb

The other kind of prediction that people love to get into is the crazy, far out bold prediction. These are the ones that get people gasping and following your every word to see if it pays off. But these predictions are prone to failure and distraction.

Let’s run another example. Here are four bold sample predictions for 2018:

  • Cisco will buy Arista.
  • VMware will cease to be a separate brand inside Dell.
  • Hackers will release a tool to compromise iPhones remotely.
  • HPE will go out of business.

Those predictions are more exciting! They name companies like Cisco and VMware and Apple. They have very bold statements like huge purchases or going out of business. But guess what? They’re completely made up. I have no insight or research that tells me anything even close to those being true or not.

However, those bold predictions just sit out there and fester. People point to them and say, “Tom thinks Cisco will buy Arista in 2018!” And no one will every call me on the carpet if I’m wrong. If Cisco does end up buying Arista in 2020 or later, people will just say I was ahead of my time. If it never comes to pass, people will forget and just focus on my next bold prediction of VMware buying Cisco. It’s a hype train with no end.

And on the off chance that I do nail a big one, people are going to think I have the inside track. My little predictions will be more important. And if I hit half of my bold ones, I would probably start getting job offers from analyst firms and such. These people are the prediction masters extraordinaire. If they aren’t telling you something you already know, they’re pitching you something that have no idea about.

Apple has a cottage industry built around crazy predictions. Just look back to August to see how many crazy ideas were out there about the iPhone X. Fingerprint sensor under the glass? 3D rear camera? Even crazier stuff? All reported on as pseudo-fact and eaten up by the readers of “news” sites. Even the people who do a great job of prediction based on solid research missed a few key details in the final launch. it just goes to show that no one is 100% accurate in bold predictions.


Tom’s Take

I still do predictions for other people. Sometimes I try to make tongue-in-cheek ones for fun. Other times I try to be serious and do a little research. But I also think that figuring out what’s coming 5 years from now is a waste of my time. I’d rather try to figure out how to use what I have today and build that toward the future. I’d rather be a happy iPhone user than the people that predicted that Apple’s move into the mobile market would fail miserably. Because that’s a headline you’ll never live down.

I’d like to thank my friends at Network Collective for inspiring this post. Make sure you check out their video podcast!

Scotty Isn’t DevOps

I was listening to the most recent episode of our Gestalt IT On-Presmise IT Roundtable where Stephen Foskett mentioned one of our first episodes where we discussed whether or not DevOps was a disaster, or as I put it a “dumpster fire”. Take a listen here:

Around 13 minutes in, I have an exchange with Nigel Poulton where I mention that the ultimate operations guy is Chief Engineer Montgomery Scott of the USS Enterprise. Nigel countered that Scotty was the epitome of the DevOps mentality because his crazy ideas are what kept the Enterprise going. In this post, I hope to show that not only was Scott not a DevOps person, he should be considered the antithesis of DevOps.

Engineering As Operations

In the fictional biography of Mr. Scott, all he ever wanted to do was be an engineer. He begrudging took promotions but found ways to get back to the engine room on the Enterprise. He liked working starships. He hated building them. His time working on the transwarp drive of the USS Excelsior proved that in the third Star Trek film.

Scotty wasn’t developing new ideas to implement on the Enterprise. He didn’t spend his time figuring out how to make the warp engines run at increased efficiency. He didn’t experiment with the shields or the phasers. Most of his “miraculous” moments didn’t come from deploying new features to the Enterprise. Instead, they were the fruits of his ability to streamline operations to combat unforeseen circumstances.

In The Apple, Scott was forced to figure out a way to get the antimatter system back online after it was drained by an unseen force. Everything he did in the episode was focused on restoring functions to the Enterprise. This wasn’t the result of a failed upgrade or a continuous deployment scenario. The operation of his ship was impacted. In Is There No Truth In Beauty, Mr. Scott even challenges the designer of the Enterprise’s engines that he can’t handle them as well as Scotty. Mr. Scott was boasting that he was better at operations than a developer. Plain and simple.

In the first Star Trek movie, Admiral Kirk is pushing Scotty to get the Enterprise ready to depart in hours after an eighteen month refit. Scotty keeps pushing back that they need more time to work out the new systems and go on a shakedown cruise. Does that sound like a person that wants to do CI/CD to a starship? Or does it sound more like the caution of an operations person wanting to make sure patches are deployed in a controlled way? Every time someone in the series or movies suggested doing major upgrades or redesigns to the Enteprise, Scotty always warned against doing it in the field unless absolutely necessary.

Montgomery Scott isn’t the King of DevOps. He’s a poster child for simple operations. Keep the systems running. Deal with problems as they arise. Make changes only if necessary. And don’t monkey with the systems! These are the tried-and-true refrains of a person that knows that his expertise isn’t in building things but in making them run.

Engineering as DevOps

That’s not to say that Star Trek doesn’t have DevOps engineers. The Enterprise-D had two of the best examples of DevOps that I’ve ever seen – Geordi LaForge and Data. These two operations officers spent most of their time trying new things with the Enterprise. And more than a few crises arose because of their development aspirations.

LaForge and Data were constantly experimenting on the Enterprise in an attempt to make it run better. Given that the mission of the Enterprise-D did not have the same five-year limit as the original, they were expected to keep the technology on the Enterprise more current in space. However, their experiments often led to problems. Destabilizing the warp core, causing shield harmonics failures, and even infecting the Enterprise’s computer with viruses were somewhat commonplace during Geordi’s tenure as Chief Engineer.

Commander Data was also rather fond of finding out about new technology that was being developed and trying to integrate it into the Enterprise’s systems. Many times, he mentioned finding out about something being developed the the Daystrom Institute and wanting to see if it would work for them. Which leads me to think that the Daystrom Institute is the Star Trek version of Stack Overflow – copy some things you think will make everything better and hope it doesn’t blow up because you didn’t understand it.

Even if it was a plot convenience device, it felt like the Enterprise was often caught in the middle of applying a patch or an upgrade right when the action started. An exploding star or an enemy vessel always waited until just the right moment to put the Enterprise in harm’s way. Even Starfleet seemed to decide the Enterprise was the only vessel that could help after the DevOps team took the warp core offline to make it run 0.1% faster.

Perhaps instead of pushing forward with an aggressive DevOps mentality for the flagship of the Federation, Geordi and Data would have done better to take lessons from Mr. Scott and wait for appropriate windows to make changes and upgrades and quite tinkering with their ship so often that it felt like it was being held together by duct tape and hope.


Tom’s Take

Despite being fictional characters, Scotty, Geordi, and Data all represent different aspects of the technology we look at today. Scotty is the tried-and-true operations person. Geordi and Data are leading the charge to keep the technology fresh. Each of them has their strong points, but it’s hard to overlook Scotty as being a bastion of simple operations mentalities. Even when they all met together in Relics, Scotty was thinking more about making things work and less on making them fast or pretty or efficient. I think the push to the DevOps mentality would do well to take a seat and listen to the venerable chief engineer of the original Enterprise.

More Accurate IT Acronyms

IT is flooded with acronyms. It takes a third of our working life to figure out what they all mean. Protocols aren’t any easier to figure out if it’s just a string of three or four letters that look vaguely like a word. Which, by the way, you should never pronounce.

But what if the acronyms of our favorite protocols didn’t describe what the designers wanted but instead described what they actually do?

  • Sporadic Network Mangling Protocol

  • Obscurity Sends Packets Flying

  • Expensive Invention Gets Routers Puzzled

  • Vexing Router Firmware

  • Really Intensive Protocol

  • Someone Doesn’t Worry About Networking

  • Somewhat Quixotic Language

  • Blame It oN DNS

  • Cisco’s Universal Call Misdirector

  • Some Mail’s Thrown Places

  • Mangles Packets, Looks Silly

  • Amazingly Convoluted Lists

  • ImProperly SECured

  • May Push Lingering Sanity To Expire

Are there any other ones you can think of? Leave it in the comments.

Is It Really Always The Network?

Keep Calm and Blame the Network

Image from Thomas LaRock

I had a great time over the last month writing a series of posts with my friend John Herbert (@MrTugs) over on the SolarWinds Geek Speak Blog. You can find the first post here. John and I explored the idea that people are always blaming the network for a variety of issues that are often completely unrelated to the actual operation of the network. It was fun writing some narrative prose for once, and the feedback we got was actually pretty awesome. But I wanted to take some time to explain the rationale behind my madness. Why is it that we are always blaming the network?!?

Visibility Is Vital

Think about all the times you’ve been working on an application and things start slowing down. What’s the first thing you think of? If it’s a standalone app, it’s probably some kind of processing lag or memory issues. But if that app connects to any other thing, whether it be a local network or a remote network via the Internet, the first culprit is the connection between systems.

It’s not a large logical leap to make. We have to start by assuming the the people that made the application knew what they were doing. If hundreds of other people aren’t having this problem, it must not be with the application, right? We’ve already started eliminating the application as the source of the issues even before we start figuring out what went wrong.

People will blame the most visible part of the system for issues. If that’s a standalone system sealed off from the rest of the world, it obviously must be the application. However, we don’t usually build these kinds of walled-off systems any longer. Almost every application in existence today requires a network connection of some kind. Whether it’s to get updates or to interact with other data or people, the application needs to talk to someone or maybe even everyone.

I’ve talked before about the need to make the network more of a “utility”. Part of the reason for this is that it lowers the visibility of the network to the rest of the IT organization. Lower visibility means fewer issues being incorrectly blamed on the network. It also means that the network is going to be doing more to move packets and less to fix broken application issues.

Blame What You Know

If your network isn’t stable to begin with, it will soon become the source of all issues in IT even if the network has nothing to do with the app. That’s because people tend to identify problem sources based on their own experience. If you are unsure of that, work on a consumer system helpdesk sometime and try and keep track of the number of calls that you get that were caused by “viruses” even if there’s no indication that this is a virus related issue. It’s staggering.

The same thing happens in networking and other enterprise IT. People only start troubleshooting problems from areas of known expertise. This usually breaks down by people shouting out solutions like, “I saw this once so it must be that! I mean, the symptoms aren’t similar and the output is totally different, but it must be that because I know how to fix it!”

People get uncomfortable when they are faced with troubleshooting something unknown to them. That’s why they fall back on familiar things. And if they constantly hear how the network is the source of all issues, guess what the first thing to get blamed is going to be?

Network admins and engineers have to fight a constant battle to disprove the network as the source of issues. And for every win they get it can come crashing down when the network is actually the problem. Validating the fears of the users is the fastest way to be seen as the issue every time.

Mean Time To Innocence

As John and I wrote the pieces for SolarWinds, what we wanted to show is that a variety of issues can look like network things up front but have vastly different causes behind the scenes. What I felt was very important for the piece was the distinguish the the main character, Amanda, go beyond the infamous Mean Time To Innocence (MTTI) metric. In networking, we all too often find ourselves going so far as to prove that it’s not the network and then leave it there. As soon as we’re innocent, it’s done.

Cross-function teams and other DevOps organizations don’t believe in that kind of boundary. Lines between silos blur or are totally absent. That means that instead of giving up once you prove it’s not your problem, you need to work toward fixing what’s at issue. Fix the problem, not the blame. If you concentrate on fixing the problems, it will soon become noticeable that networking team may not always be the problem. Even if the network is at fault, the network team will work to fix it and any other issues that you see.


Tom’s Take

I love the feedback that John and I have gotten so far on the series we wrote. Some said it feels like a situation they’ve been in before. Others have said that they applaud the way things were handled. I know that the narrative allows us to bypass some of the unsavory things that often happen, like argument and political posturing inside an organization to make other department heads look bad when a problem strikes. But what we really wanted to show is that the network is usually the first to get blamed and the last to keep its innocence in a situation like this.

We wanted to show that it’s not always the network. And the best way for you to prove that in your own organization is to make sure the network isn’t just innocent, but helpful in solving as many problems as possible.