Fast Friday- Perry Mason Moments

It’s the Thanksgiving holiday weekend in the US which means lots of people discussing things with their relatives. And, as is often the case, lots of arguments. It’s the nature of people to have a point of view and then to want to defend it. And it’s not just politics or other divisive topics. We see it all the time in networking too.

EIGRP vs OSPF. Cisco vs Juniper. ACI vs NSX. You name it and we’ve argued about it. Every viewpoint has a corresponding counterpart. Yes, there are good points for using one versus the other. But there are also times when every piece of factual information doesn’t matter because we “know” the right answer.

It’s those times when we run into what I call the “Perry Mason Problem”. It’s a reminder of the old Perry Mason TV show when the lawyer in the title would win a case with a carefully crafted statement that just ends any arguments. It’s often called a Wham Line or an Armor-Piercing Question. Basically, Mr. Mason would ask a question or make a statement that let all the air out of the argument. And often it would result in him winning the case without any further discussion.

Most people that argue keep searching for that magic Perry Mason moment. They want to win and they want to do it decisively. They want to leave their opponent speechless. They want to drop a microphone and walk away victorious.

And yet, that almost never happens in real life. No one is swayed by a single statement. No one changes their mind because of a question no matter how carefully crafted. Sure, it can add to them deciding to change their mind in the long run. But that process happens in the person. It doesn’t happen because of someone else.

So, if you find yourself in the middle of heated discussion don’t start looking for Perry Mason moments. Instead, you need to think about why you’re trying to change someone’s mind. Instead of trying to win an argument it’s better to consider your position and understand why you’re trying to win. I bet a little introspection will do a lot more good than looking for that wham line. You may not necessarily agree with their perspective but I bet you’ll have an easier time living with that than trying to change the mind of someone that’s set against it.

Five Minutes To Magic Time

Have you ever worked with someone that has the most valuable time in the world? Someone that counts each precious minute in their presence as if you’re keeping them from something very, very important that they could use to solve world hunger or cure cancer? If you haven’t then you’re a very lucky person indeed. Sadly, almost everyone, especially those in IT, has had the misfortune to be involved with someone whose time is more precious than platinum-plated saffron.

That’s not to say that we should be wasting the time of those we work with. Simple things like being late to meetings or not having your materials prepared are easy ways to help reduce the time of meetings or to make things run smoothly. Those items are common courtesies that should be extended to all the people you meet, from the cashier that takes your order at a fast food establishment to the most powerful people on the planet. No, this is about something deeper and more insidious.

No Time For Hugs

I’ve seen the kind of behavior I’ve described very often in the higher echelons of companies. People that live at the CxO level often have very little time to devote to anything that resembles thought. They’re busy strategizing and figuring out ways to keep the company profitable. They don’t have time to listen to people talk. Talking interrupts their brain functions. They need time to think.

If you think I’m being hyperbolic, ask yourself how many times you’ve been told to “simplify” something for a CEO when you present to them (if you’re even given the opportunity). I think this strip from Dilbert explains it succinctly:

https://dilbert.com/strip/1998-01-10

The higher up the food chain you go, the simpler it needs to be. But if the CEO is the most important person in the company, how is it that you need to make things easy for them to understand? They aren’t morons, right? They got this job somehow?

The insinuation is that the reason why you need to make it simple for them is because their time is too valuable. Needless talking and discussion takes them away from things that are more important. Like thinking and strategizing. Or something. So, if their time is the most value in the room, what does that say about your time? How does it feel to know that your efforts and research and theorizing are essentially wasted work because your time isn’t as important as the person you’re talking to?

This is even more egregious when you realize that your efforts to summarize something down to the most basic level is often met with a lot of questions about how you determined that conclusion. In essence, all the hard work you did to simplify your statements is undone because someone wants you to justify why you go to that conclusion. You know, the kinds of details you would have given in a presentation if you’d been given the time to explain!

Solution: Five Minute Meetings

Okay, so I know I’m going to get flack for this one. Everyone has the solution the meeting overload problem. Standup meetings, team catch ups, some other kind of crazy treadmill conference calls. But the real way to reduce your meeting stress is to show people how valuable time is for everyone. Not just them.

My solution: All meetings with CxO level people are now five minutes long. Period. End of story. You get to walk in, give your statement, and you walk out. No questions. No long Q&A. Just your conclusions. You say what you have to say and move on.

Sounds stupid, doesn’t it? And that’s kind of the point. When you are forced to boil your premise down to something like the Dilbert smiley face above, you’re doing yourself a disservice. All the detail and nuance goes right out the window. The only way you get to bring it back out is if someone in the room starts asking questions. And if you don’t give enough detail they almost always will. Which defeats the purpose of boiling it down in the first place!

Instead, push it back on the CxOs with the most valuable time. Make them see how hard you work. By refusing to answer any of their follow up questions. You see, if their time is so valuable, you need to show them how much you respect it. If they have follow up questions or require more details, they need to write all those interrogatories down in an email or an action item list and send it to you so you can get it done on your time. Make them wait for the answers. Because then they’ll see that this idea that their time is valuable is just an illusion.

It sounds awfully presumptuous of me to say that we need to waste the time the C-level suite. But a little bit of pushback goes a long way. Imagine how furious they’ll be when you walk out of the meeting after five minutes and don’t answer a single question. How dare this knowledge worker not bend their calendar to my desire to learn more?!? It’s ridiculous!

How about wondering how ridiculous it is for this person to limit your time? Or to not know anything ahead of time about the topic of discussion? Imagine telling someone to wait until you’re ready to talk to them after a meeting starts because you are more important than they are! The nerve!

However, once you stick to your plan a few times the people in the room will understand that meetings about topics should be as long as they need to be. And you should be given enough time to explain up front instead of talking for five minutes and getting interrupted with a thousand questions that you were prepared to answer anyway if they’d just given you the chance to present!

Watch how your meetings transform from interrogation scenes to actual presentations with discussions. Instead of only getting five minutes to talk you’ll be accorded all the time you need to fill in the details. Maybe you only needed ten minutes in the first place. But the idea is that now your time and expertise is just as valuable as everyone else on the team, from the bottom all the way to the top.


Tom’s Take

There needs to be an obligatory “no everyone is like this” disclaimer. I’ve met some very accommodating executives. And I’ve also met some knowledge workers that can’t present their way out of a paper bag. But the way to fix those issues is to make them get better at giving info and at listening to presentations. The way is not to artificially limit time to make yourself seem more important. When you give your people the time they need to get you the info you need, you’ll find that they are going to answer the questions you have a lot quicker than waiting with dread as the CEO takes the time to think about what they were going to be told anyway.

AI and Trivia

questions answers signage

Photo by Pixabay on Pexels.com

I didn’t get a chance to attend Networking Field Day Exclusive at Juniper NXTWORK 2019 this year but I did get to catch some of the great live videos that were recorded and posted here. Mist, now a Juniper Company, did a great job of talking about how they’re going to be extending their AI-driven networking into the realm of wired networking. They’ve been using their AI virtual assistant, named “Marvis”, for quite a while now to solve basic wireless issues for admins and engineers. With the technology moving toward the copper side of the house, I wanted to talk a bit about why this is important for the sanity of people everywhere.

Finding the Answer

Network and wireless engineers are walking storehouses of useless trivia knowledge. I know this because I am one. I remember the hello and dead timers for OSPF on NBMA networks. I remember how long it takes BGP to converge or what the default spanning tree bridge priority is for a switch. Where some of my friends can remember the batting average for all first basemen in the league in 1971, I can instead tell you all about LSA types and the magical EIGRP equation.

Why do we memorize this stuff? We live in a world with instant search at our fingertips. We can find anything we might need thanks to the omnipotent Google Search Box. As long as we can avoid sponsored results and ads we can find the answer to our question relatively quickly. So why do we require people to memorize esoteric trivia? Is it so we can win free drinks at the bar after we’re done troubleshooting?

The problem isn’t that we have to know the answer. It’s that we need to know the answer in order to ask the right question. More often than now we find ourselves stuck in the initial phase of figuring out the problem. The results are almost always the same – things aren’t working. Finding the cause isn’t always easy though. We have to find some nugget of information to latch onto in order to start the process.

One of my old favorites was trying to figure out why a network I was working with had a segmented spanning tree. One side of the network was working just fine but there were three switches daisy chained together that didn’t. Investigations turned up very little. Google searches were failing me. It wasn’t until I keyed in on a couple of differences that I found out that I had improperly used a BPDU filtering command because of a scoping issue. Sure, it only took me two hours of searching to find it after I discovered the problem. But if I hadn’t memorized the BDPU filtering and guard commands and their behavior I wouldn’t have even known to ask about them. So it’s super important to know how every minutia of every protocol works, right?

Presenting the Right Questions

Not exactly. We, as human computers, memorize the answers to more efficiently search through our database to find the right answers. If the problem takes 5 minutes to present we can eliminate a bunch of causes. If it’s happening in layer 3 and not layer 2 we can toss out a bunch of other stuff. Our knowledge is allowing us to discard useless possibilities and focus on the right result.

And it’s horribly inefficient. I can attest to that given my various attempts to learn OSPF hello and dead timers through osmosis of falling asleep in my big CCNP Routing book. The answers don’t crawl off the page and into your brain no matter how loudly you snore into it. So I spent hours learning something that I might use two or three times in my career. There has to be a better way.

Not coincidentally, that’s where the AI-driven systems from Mist, and now Juniper, come into play. Marvis is wonderful at looking at symptoms and finding potential causes. It’s what we do as humans. Except Marvis has no inherent biases. It also doesn’t misremember the values for a given protocol or get confused about whether or not OSPF point-to-point networks are broadcast or not. Marvis just knows what it was programmed with. But it does learn.

Learning is the key to how these AI and machine learning (ML) driven systems have to operate. People tend to discount solutions because they think there’s no way it could be that solution this time. For example, a haiku:

It’s not DNS.
Could it be DNS?
It was DNS.

DNS is often the cause of our problems even if we usually discount it out of hand in the first five minutes of troubleshooting. Even if it was only DNS 50% of the time we would still toss DNS as the root cause within the first five minutes because we’ve “trained” our brains to know what a DNS problem looks like without realizing how many things DNS can really affect.

But AI and ML don’t make these false correlations. Instead, they learn every time what the cause was. They can look at the network and see the failure state, present options based on the symptoms, and even if you don’t check in your changes they can analyze the network and figure out what change caused everything to start working again. Now, the next time the problem crops up, a system like Marvis can present you with a list of potential solutions with confidence levels. If DNS is at the top of the list, you might want to look into DNS first.

AI is going to make us all better troubleshooters because it’s going to make us all less reliant on poor memory. Instead of misremembering how a protocol should be configure, AI and ML will tell us how it should look. If something is causing routing loops or if layer 2 issues are happening because of duplex mismatches we’ll be able to see that quickly and have confidence it’s the right answer instead of just guessing and throwing things at the wall until they stick. Just like Google has supplanted the Cliff Claven people at the bar that are storehouses of useless knowledge, so too will AI and ML reduce our dependence on know-it-alls that may not have all the answers.


Tom’s Take

I’m ready to be forgetful. I’m tired of playing “stump the chump” in troubleshooting with the network playing the part of the stumper and me playing the chump. I’ve memorized more useless knowledge than I ever care to recall in my life. But I don’t want to have to do the work any longer. Instead, I want to apply my gifts to training algorithms with more processing power than me to do all the heavy lifting. I’m more than happy to look at DNS and EIGRP timers than try to remember if MTU and reliability are part of the K-values for this network.

Locked Up By Lock-In

When you start evaluating a solution, you are going to get a laundry list of features and functionality that you are supposed to use as criteria for selection. Some are important, like the ones that give you the feature set you need to get your job done. Others are less important for the majority of use cases. One thing tends to stand out for me though.

Since the dawn of platforms, I believe the first piece of comparison marketing has been “avoids lock-in”. You know you’ve seen it too. For those that may not be completely familiar with the term, “lock-in” describes a platform where all the components need to come from the same manufacturer or group of manufacturers in order to work properly. An example would be if a networking solution required you to purchase routers, switches, access points, and firewalls from a single vendor in order to work properly.

Chain of Fools

Lock in is the greatest asset a platform company has. The more devices they can sell you the more money they can get from you at every turn. That’s what they want. So they’re going to do everything they can to keep you in their ecosystem. That includes things like file formats and architectures that require the use of their technology or of partner technologies to operate correctly.

So, the biggest question here is “What’s wrong with that?” Note that I’m not a proponent of lock-in. Rather, I’m against the false appearance of choices. Sure, offering a platform with the promise of “no lock-in” is a great marketing tool. But how likely are you to actually follow through on that promise?

I touched on this a little bit earlier this year at Aruba Atmosphere 2019 when I talked about the promise of OpenConfig allowing hardware from different vendors to all be programmed in a similar way. The promise is grand that you’ll be able to buy an access point from Extreme and run it on an Aruba Controller while the access layer polices are programmed into Cisco switches. It’s the dream of interoperability!

More realistically, though, you’ll find that most people aren’t that concerned about lock-in. The false choice of being open to new systems generally comes down to one single thing: price. The people that I know that complain the most about vendor lock-in almost always follow it up with a complaint about pricing or licensing costs. For example:

The list could go on for three or four more pages. And the odds are good you’ve looked at one of those solutions already or you’re currently dealing with something along those lines. So, ask yourself how much pain vendor lock-in brings you aside from your checkbook?

The most common complaint, aside from price, is that the vendor solution isn’t “best of breed”. Which has always been code for “this particular piece sucks and I really wish I could use something else”. But there’s every possibility that the solution sucks because it has to integrate tightly with the rest of the platform. It’s easy to innovate when you’re the only game in town and trying to get people to buy things from you. But if you’re a piece of a larger puzzle and you’re trying to have eight other teams tell you what your software needs to do in order to work well with the platform, I think you can see where this is going.

How many times have you actually wished you could pull out a piece and sub in another one? Again, aside from just buying the cheapest thing off the shelf? Have you ever really hoped that you could sub in an Aerohive AP630 802.11ax (Wi-Fi 6) AP into your Cisco wireless network because they were first to market? Have you ever really wanted to rip out Cisco ISE from your integrated platform and try to put Aruba ClearPass in its place? Early adopters and frustrated users are some of the biggest opponents of vendor lock-in.

Those Three Words

I’m about to tell you why lock-in isn’t the demon you think it is. And I can do it in three words:

It. Just. Works.

Granted, that’s a huge stretch and we all know it. It really should be “well built software that meets all my functionality goals should just work”. But in reality, the reason why we all like to scream at lock-in when we’re writing checks for it is because the alternative to paying big bucks for making it “all just work” is for us to invest our time and effort into integrating two solutions together. And ask your nearest VAR or vendor partner how much fun that can be?

I used to spend a LOT of my time trying to get pieces and parts to integrate because schools don’t like to spend a lot of money on platforms. In Oklahoma they’re even allowed to get out of license agreements every year. Know what that means? A ton of legacy software that’s already paid for sitting around waiting to run on brand new hardware they just bought. And oh, by the way, can you make all that work together in this new solution we were sold by the Vendor of the Month Club?

And because they got the best deal or the best package, I had to spend my time and effort putting things together. So, in a way, the customer was trading money from the hardware and software to me for my wetware — the brain power needed to make it all work together. So, in a way, I was doing something even worse. I was creating my own lock-in. Not because I was building an integrated solution with one vendor’s pieces. But because I was building a solution that was custom and difficult to troubleshoot, even with proper documentation.


Tom’s Take

Lock-in isn’t the devil. You’re essentially trading flexibility for ease-of-use. You’re trading the ability to switch out pieces of a solution for the ability to integrate the pieces together without expending much effort. And yes, I realize that some lock-in solutions are harder to integrate than others. I’m looking at you, Cisco ISE. But, that just speaks to how hard it is to create the kind of environment that you get when you are trying to create all kinds of integrations. You’ll find that the idea of freedom of choice is much more appealing than actually having the ability to swap things out at will.

Procrastination Party

“I’ll get to that later.”

“I’m not feeling it right now.”

“I have to find an angle.”

“It will be there tomorrow.”

Any of those sound familiar? I know they do for me. That’s because procrastination is the beast that lives inside all of us. Slumbering until a time when it awakes and persuades us to just put things off until later. Can’t hurt, right?

Brain Games

The human brain is an amazing thing. It is the single largest consumer of nutrients and oxygen in the human body. It’s the reason why human babies are born practically helpless due to the size in relation to the rest of an infant. It’s the reason why we can make tools, ponder the existence of life in the universe, and write kick-ass rock and roll music.

But the human brain is lazy. It doesn’t like thinking. It prefers simple patterns and easy work. Given a choice, the human brain would rather do some kind of mindless repetitive task ad naseum instead of creating. When you think about it that makes a lot of sense from a biological perspective. Tasks that are easy don’t engage many resources. Which means the brain doesn’t have to operate as much and can conserve energy.

But we don’t want our brains to be lazy. We want to create and learn and do amazing things with our thoughts. Which means we have to overcome the inertia of procrastination. We have to force ourselves to move past the proclivity to be lazy in our thoughts. It is said that the hardest part of running isn’t the mileage but is instead getting up in the morning and putting on your shoes. Likewise, the hardest part of being creative isn’t the actual thinking but is instead starting the process of it in the first place.

Strategies for Anti-Procrastination

I have some methods I use to fight my tendency to procrastinate. I can’t promise they’ll work for you but the idea is that you base your strategies around the ideas and go from there.

  • Make Yourself Uncomfortable – I’m not talking about laying on a bed or nails or working in the freezing cold. What I mean is take yourself out of you comfort zone. Instead of sitting in my office when I need to write a few things, I intentionally go somewhere in public, like a coffee shop. Why? Because putting myself in a place with noice and uncomfortable chairs makes me focus on what I’m supposed to be doing. My brain can’t get lazy when it’s being stimulated from all sides. I have to apply some effort to drown out the conversation and that extra effort pushes me into action.
  • Set a Small Goal to Relax – This one works wonders for your brain. If it thinks there’s even a remote possibility in the near future that it can relax it’s going to race to that finish line as fast as possible. If you’re familiar with the Pomodoro Technique that’s basically what’s going on. Your brain sees the opportunity to be lazy five minutes out of every 30 so it pushes to get there. Except you’re tricking it by forcing it to do work to get there. You become more productive because you’re thinking you get to relax in ten or fifteen minutes when in fact you’re much more productive because you’ve secretly been focused the whole time.
  • Create a Zone For Yourself – This is kind of the opposite of the first point above but it works just as well. Your brain likes to do mindless repetitive tasks because they require very little energy. So why not use that against your lazy brain and trick it into thinking whatever you’re doing is actually “easy”? There’s a ton of ways to do this. My two favorites involve aural stimulation. There are a lot of folks that have a Coding Playlist or a Writing Setlist that they use to zone out and accomplish tasks that require some focus but are mindless in nature. Likewise, I often use noise generators to do the same thing. My current favorite is A Soft Murmur because it allows me to customize the noise that I want to help me shut off the distractions and focus on what I’m doing. I’ll often pair it with a sprint from the second point above to help me really dial in what I’m trying to work on.

Tom’s Take

Your mileage may very greatly on the items above. Your brain doesn’t work like mine. You know how you think and what it takes to motivate you. Maybe it’s massive amounts of coffee or a TV playing in the background. But knowing that your mind wants to shut off active processing and do repetitive things over and over again does wonders to help figure out how best to engage it and work smarter. You can’t always stop procrastination. But, with a little planning, you can put it off until tomorrow.

IT Hero Culture

I’ve written before about rock stars and IT super heroes. We all know or have worked with someone like this in the past. Perhaps we still do have someone in the organization that fits the description. But have you ever stopped to consider how it could be our culture that breeds the very people we don’t want around?

Keeping The Lights On

When’s the last time you got recognition for the network operating smoothly? Unless it was in response to a huge traffic spike or an attack that tried to knock you offline, the answer is probably never or rarely. Despite the fact that networks are hard to build and even harder to operate, we rarely get recognized for keeping the lights on day after day.

It’s not all that uncommon. The accounting department doesn’t get recognized when the books are balanced. The janitorial staff doesn’t get an exceptional call out when the floors are mopped. And the electric company doesn’t get a gold star because they really did keep the lights on. All of these things are examples of expected operation. When we plug something into a power socket, we expect it to work. When we plug a router in, we expect it to work as well. It may take more configuration to get the router working than the electrical outlet, but that’s just because the hard work of the wiring has already been done.

The only time we start to notice things is when they’re outside our expectation. When the accounting department’s books are wrong. When the floors are dirty. When the lights aren’t on. We’re very quick to notice failure. And, often, we have to work very hard to minimize the culture that lays blame for failure. I’ve already talked a lot about things like blameless post-mortems and other ways to attack problems instead of people. Companies are embracing the idea that we need to fix issues with our systems and not shame our people into submission for things they might not have had complete control over.

Put On The Cape

Have you ever thought about what happens in the other direction, though? I can speak from experience because I spent a lot of time in that role. As a senior engineer from a VAR, I was often called upon to ride in and save the day. Maybe it was after some other company had tried to install something and failed. Or perhaps it was after one of my own technicians had created an issue that needed to be resolved. I was ready on my white horse to ride in and save the day.

And it felt nice to be recognized for doing it! Everyone feels a bit of pride when you are the person to fix an issue or get a site back up and running after an outage. Adulation is a much better feeling than shame without a doubt. But it also beat apathy too. People don’t get those warm fuzzy feelings from just keeping the lights on, after all.

The culture we create that worships those that resolve issues with superhuman skill reinforces the idea that those traits are desirable in engineers. Think about which person you’d rather have working on your network:

  • Engineer A takes two months to plan the cutover and wants to make sure everything goes smoothly before making it happen.
  • Engineer B cuts over with very little planning and then spends three hours of the maintenance window getting all the systems back online after a bug causes an outage. Everything is back up and running before the end of the window.

Almost everyone will say they want Engineer A working for them, right? Planning and methodical reasoning beats a YOLO attitude any day of the week. But who do we recognize as the rockstar with special skills? Probably Engineer B. Whether or not they created their own issue they are the one that went above and beyond to fix it.

We don’t reward people for producing great Disaster Recovery documentation. We laud them for pulling a 36-hour shift to rebuild everything because there wasn’t a document in the first place. We don’t recognize people that spend an extra day during a wireless site survey to make sure they didn’t miss anything in a warehouse. But we really love the people that come in after-the-fact and spend countless hours fixing it.

Acknowledging Averages

Should we stop thanking people for all their hard work in solving problems? No. Because failure to appreciate true skills in a technical resource will sour them on the job quickly. But, if we truly want to stop the hero worshipping behavior that grows from IT, we have to start acknowledging the people that put in hard work day after day to stay invisible.

We need to give a pat on the back to an engineer that built a good script to upgrade switches. Or to someone that spent a little extra time making sure the site survey report covered everything in detail. We need to help people understand that it’s okay to get your job done and not make a scene. And we have to make sure that we average out the good and the bad when trying to ascertain root cause in outages.

Instead of lauding rock stars for spending 18 hours fixing a routing issue, let’s examine why the issue occurred in the first place. This kind of analysis often happens when it’s a consultant that has to fix the issue since a cost is associated with the fix, but it rarely happens in IT departments in-house. We have to start thinking of the cost of this rock star or white knight behavior as being something akin to money or capital in the environment.


Tom’s Take

Rock star culture and hero worship in IT isn’t going to stop tomorrow. It’s because we want to recognize the people that do the work. We want to hold those that go above and beyond up to those that we want to emulate them. But we should also be asking hard questions about why it was necessary for there to need to be a hero in the first place. And we have to be willing to share some of the adulation with those that keep the lights on between disasters that need heroes.

Positioning Policy Properly

Who owns the network policy for your organization? How about the security policy?Identity policy? Sound like easy questions, don’t they? The first two are pretty standard. The last generally comes down to one or two different teams depending upon how much Active Directory you have deployed. But have you ever really thought about why?

During Future:NET this week, those poll questions were asked to an audience of advanced networking community members. The answers pretty much fell in line with what I was expecting to see. But then I started to wonder about the reasons behind those decisions. And I realized that in a world full of cloud and DevOps/SecOps/OpsOps people, we need to get away from teams owning policy and have policy owned by a separate team.

Specters of the Past

Where does the networking policy live? Most people will jump right in with a list of networking gear. Port profiles live on switches. Routing tables live on routers. Networking policy is executed in hardware. Even if the policy is programmed somewhere else.

What about security policy? Firewalls are probably the first thing that come to mind. More advanced organizations have a ton of software that scans for security issues. Those policy decisions are dictated by teams that understand the way their tools work. You don’t want someone that doesn’t know how traffic flows through a firewall to be trying to manage that device, right?

Let’s consider the identity question. For a multitude of years the identity policy has been owned by the Active Directory (AD) admins. Because identity was always closely tied to the server directory system. Novell (now NetIQ) eDirectory and Microsoft AD were the kings of the hill when it came to identity. Today’s world has so much distributed identity that it’s been handed to the security teams to help manage. AD doesn’t control the VPN concentrator the cloud-enabled email services all the time. There are identity products specifically designed to aggregate all this information and manage it.

But let’s take a step back and ask that important question: why? Why is it that the ownership of a policy must be by a hardware team? Why must the implementors of policy be the owners? The answer is generally that they are the best arbiters of how to implement those policies. The network teams know how to translate applications in to ports. Security teams know how to create firewall rules to implement connection needs. But are they really the best people to do this?

Look at modern policy tools designed to “simplify” networking. I’ll use Cisco ACI as an example but VMware NSX certainly qualifies as well. At a very high level, these tools take into account the needs of applications to create connectivity between software and hardware. You create a policy that allows a database to talk to a front-end server, for example. That policy knows what connections need to happen to get through the firewall and also how to handle redundancy to other members of the cluster. The policy is then implemented automatically in the network by ACI or NSX and magically no one needs to touch anything. The hardware just works because policy automatically does the heavy lifting.

So let’s step back for moment and discuss this. Why does the networking team need to operate ACI or NSX? Sure, it’s because those devices still touch hardware at some point like switches or routers. But we’ve abstracted the need for anyone to actually connect to a single box or a series of boxes and type in lines of configuration that implement the policy. Why does it need to be owned by that team? You might say something about troubleshooting. That’s a common argument that whoever needs to fix it when it breaks is who needs to be the gatekeeper implementing it. But why? Is a network engineer really going to SSH into every switch and correct a bad application tag? Or is that same engineer just going to log into a web console and fix the tag once and propagate that info across the domain?

Ownership of policy isn’t about troubleshooting. It’s about territory. The tug-of-war to classify a device when it needs to be configured is all about collecting and consolidating power in an organization. If I’m the gatekeeper of implementing workloads then you need to pay tribute to me in order to make things happen.

If you don’t believe that, ask yourself this: If there was a Routing team and and Switching team in an organization, who would own the routed SVI interface on a layer 3 switch? The switching team has rights because it’s on their box. The routing team should own it because it’s a layer 3 construct. Both are going to claim it. And both are going to fight over it. And those are teams that do essentially the same job. When you start pulling in the security team or the storage team or the virtualization team you can see how this spirals out of control.

Vision of the Future

Let’s change the argument. Instead of assigning policy to the proper hardware team, let’s create a team of people focused on policy. Let’s make sure we have proper representation from every hardware stack: Networking, Security, Storage, and Virtualization. Everyone brings their expertise to the team for the purpose of making policy interactions better.

Now, when someone needs to roll out a new application, the policy team owns that decision tree. The Policy Team can have a meeting about which hardware is affected. Maybe we need to touch the firewall, two routers, a switch, and perhaps a SAN somewhere along the way. The team can coordinate the policy changes and propose an implementation plan. If there is a construct like ACI or NSX to automate that deployment then that’s the end of it. The policy is implemented and everything is good. Perhaps some older hardware exists that needs manual configuration of the policy. The Policy Team then contacts the hardware owner to implement the policy needs on those devices. But the Policy Team still owns that policy decision. The hardware team is just working to fulfill a request.

Extend the metaphor past hardware now. Who owns the AWS network when your workloads move to the cloud? Is it still the networking team? They’re the best team to own the network, right? Except there are no switches or routers. They’re all software as far as the instance is concerned. Does that mean your cloud team is now your networking team as well? Moving to the cloud muddies the waters immensely.

Let’s step back into the discussion about the Policy Team. Because they own the policy decisions, they also own that policy when it changes hardware or location. If those workloads for email or productivity suite move from on-prem to the cloud then the policy team moves right along with them. Maybe they add an public cloud person to the team to help them interface with AWS but they still own everything. That way, there is no argument about who owns what.

The other beautiful thing about this Policy Team concept is that it also allows the rest of the hardware to behave as a utility in your environment. Because the teams that operate networking or security or storage are just fulfilling requests from the policy team they don’t need to worry about anything other than making their hardware work. They don’t need to get bogged down in policy arguments and territorial disputes. They work on their stuff and everyone stays happy!


Tom’s Take

I know it’s a bit of stretch to think about pulling all of the policy decisions out of the hardware teams and into a separate team. But as we start automating and streamlining the processes we use to implement application policy the need for it to be owned by a particular hardware team is hard to justify. Cutting down on cross-faction warfare over who gets to be the one to manage the new application policy means enhanced productivity and reduced tension in the workplace. And that can only lead to happy users in the long run. And that’s a policy worth implementing.