Fast Friday- Perry Mason Moments

It’s the Thanksgiving holiday weekend in the US which means lots of people discussing things with their relatives. And, as is often the case, lots of arguments. It’s the nature of people to have a point of view and then to want to defend it. And it’s not just politics or other divisive topics. We see it all the time in networking too.

EIGRP vs OSPF. Cisco vs Juniper. ACI vs NSX. You name it and we’ve argued about it. Every viewpoint has a corresponding counterpart. Yes, there are good points for using one versus the other. But there are also times when every piece of factual information doesn’t matter because we “know” the right answer.

It’s those times when we run into what I call the “Perry Mason Problem”. It’s a reminder of the old Perry Mason TV show when the lawyer in the title would win a case with a carefully crafted statement that just ends any arguments. It’s often called a Wham Line or an Armor-Piercing Question. Basically, Mr. Mason would ask a question or make a statement that let all the air out of the argument. And often it would result in him winning the case without any further discussion.

Most people that argue keep searching for that magic Perry Mason moment. They want to win and they want to do it decisively. They want to leave their opponent speechless. They want to drop a microphone and walk away victorious.

And yet, that almost never happens in real life. No one is swayed by a single statement. No one changes their mind because of a question no matter how carefully crafted. Sure, it can add to them deciding to change their mind in the long run. But that process happens in the person. It doesn’t happen because of someone else.

So, if you find yourself in the middle of heated discussion don’t start looking for Perry Mason moments. Instead, you need to think about why you’re trying to change someone’s mind. Instead of trying to win an argument it’s better to consider your position and understand why you’re trying to win. I bet a little introspection will do a lot more good than looking for that wham line. You may not necessarily agree with their perspective but I bet you’ll have an easier time living with that than trying to change the mind of someone that’s set against it.

Five Minutes To Magic Time

Have you ever worked with someone that has the most valuable time in the world? Someone that counts each precious minute in their presence as if you’re keeping them from something very, very important that they could use to solve world hunger or cure cancer? If you haven’t then you’re a very lucky person indeed. Sadly, almost everyone, especially those in IT, has had the misfortune to be involved with someone whose time is more precious than platinum-plated saffron.

That’s not to say that we should be wasting the time of those we work with. Simple things like being late to meetings or not having your materials prepared are easy ways to help reduce the time of meetings or to make things run smoothly. Those items are common courtesies that should be extended to all the people you meet, from the cashier that takes your order at a fast food establishment to the most powerful people on the planet. No, this is about something deeper and more insidious.

No Time For Hugs

I’ve seen the kind of behavior I’ve described very often in the higher echelons of companies. People that live at the CxO level often have very little time to devote to anything that resembles thought. They’re busy strategizing and figuring out ways to keep the company profitable. They don’t have time to listen to people talk. Talking interrupts their brain functions. They need time to think.

If you think I’m being hyperbolic, ask yourself how many times you’ve been told to “simplify” something for a CEO when you present to them (if you’re even given the opportunity). I think this strip from Dilbert explains it succinctly:

https://dilbert.com/strip/1998-01-10

The higher up the food chain you go, the simpler it needs to be. But if the CEO is the most important person in the company, how is it that you need to make things easy for them to understand? They aren’t morons, right? They got this job somehow?

The insinuation is that the reason why you need to make it simple for them is because their time is too valuable. Needless talking and discussion takes them away from things that are more important. Like thinking and strategizing. Or something. So, if their time is the most value in the room, what does that say about your time? How does it feel to know that your efforts and research and theorizing are essentially wasted work because your time isn’t as important as the person you’re talking to?

This is even more egregious when you realize that your efforts to summarize something down to the most basic level is often met with a lot of questions about how you determined that conclusion. In essence, all the hard work you did to simplify your statements is undone because someone wants you to justify why you go to that conclusion. You know, the kinds of details you would have given in a presentation if you’d been given the time to explain!

Solution: Five Minute Meetings

Okay, so I know I’m going to get flack for this one. Everyone has the solution the meeting overload problem. Standup meetings, team catch ups, some other kind of crazy treadmill conference calls. But the real way to reduce your meeting stress is to show people how valuable time is for everyone. Not just them.

My solution: All meetings with CxO level people are now five minutes long. Period. End of story. You get to walk in, give your statement, and you walk out. No questions. No long Q&A. Just your conclusions. You say what you have to say and move on.

Sounds stupid, doesn’t it? And that’s kind of the point. When you are forced to boil your premise down to something like the Dilbert smiley face above, you’re doing yourself a disservice. All the detail and nuance goes right out the window. The only way you get to bring it back out is if someone in the room starts asking questions. And if you don’t give enough detail they almost always will. Which defeats the purpose of boiling it down in the first place!

Instead, push it back on the CxOs with the most valuable time. Make them see how hard you work. By refusing to answer any of their follow up questions. You see, if their time is so valuable, you need to show them how much you respect it. If they have follow up questions or require more details, they need to write all those interrogatories down in an email or an action item list and send it to you so you can get it done on your time. Make them wait for the answers. Because then they’ll see that this idea that their time is valuable is just an illusion.

It sounds awfully presumptuous of me to say that we need to waste the time the C-level suite. But a little bit of pushback goes a long way. Imagine how furious they’ll be when you walk out of the meeting after five minutes and don’t answer a single question. How dare this knowledge worker not bend their calendar to my desire to learn more?!? It’s ridiculous!

How about wondering how ridiculous it is for this person to limit your time? Or to not know anything ahead of time about the topic of discussion? Imagine telling someone to wait until you’re ready to talk to them after a meeting starts because you are more important than they are! The nerve!

However, once you stick to your plan a few times the people in the room will understand that meetings about topics should be as long as they need to be. And you should be given enough time to explain up front instead of talking for five minutes and getting interrupted with a thousand questions that you were prepared to answer anyway if they’d just given you the chance to present!

Watch how your meetings transform from interrogation scenes to actual presentations with discussions. Instead of only getting five minutes to talk you’ll be accorded all the time you need to fill in the details. Maybe you only needed ten minutes in the first place. But the idea is that now your time and expertise is just as valuable as everyone else on the team, from the bottom all the way to the top.


Tom’s Take

There needs to be an obligatory “no everyone is like this” disclaimer. I’ve met some very accommodating executives. And I’ve also met some knowledge workers that can’t present their way out of a paper bag. But the way to fix those issues is to make them get better at giving info and at listening to presentations. The way is not to artificially limit time to make yourself seem more important. When you give your people the time they need to get you the info you need, you’ll find that they are going to answer the questions you have a lot quicker than waiting with dread as the CEO takes the time to think about what they were going to be told anyway.

AI and Trivia

questions answers signage

Photo by Pixabay on Pexels.com

I didn’t get a chance to attend Networking Field Day Exclusive at Juniper NXTWORK 2019 this year but I did get to catch some of the great live videos that were recorded and posted here. Mist, now a Juniper Company, did a great job of talking about how they’re going to be extending their AI-driven networking into the realm of wired networking. They’ve been using their AI virtual assistant, named “Marvis”, for quite a while now to solve basic wireless issues for admins and engineers. With the technology moving toward the copper side of the house, I wanted to talk a bit about why this is important for the sanity of people everywhere.

Finding the Answer

Network and wireless engineers are walking storehouses of useless trivia knowledge. I know this because I am one. I remember the hello and dead timers for OSPF on NBMA networks. I remember how long it takes BGP to converge or what the default spanning tree bridge priority is for a switch. Where some of my friends can remember the batting average for all first basemen in the league in 1971, I can instead tell you all about LSA types and the magical EIGRP equation.

Why do we memorize this stuff? We live in a world with instant search at our fingertips. We can find anything we might need thanks to the omnipotent Google Search Box. As long as we can avoid sponsored results and ads we can find the answer to our question relatively quickly. So why do we require people to memorize esoteric trivia? Is it so we can win free drinks at the bar after we’re done troubleshooting?

The problem isn’t that we have to know the answer. It’s that we need to know the answer in order to ask the right question. More often than now we find ourselves stuck in the initial phase of figuring out the problem. The results are almost always the same – things aren’t working. Finding the cause isn’t always easy though. We have to find some nugget of information to latch onto in order to start the process.

One of my old favorites was trying to figure out why a network I was working with had a segmented spanning tree. One side of the network was working just fine but there were three switches daisy chained together that didn’t. Investigations turned up very little. Google searches were failing me. It wasn’t until I keyed in on a couple of differences that I found out that I had improperly used a BPDU filtering command because of a scoping issue. Sure, it only took me two hours of searching to find it after I discovered the problem. But if I hadn’t memorized the BDPU filtering and guard commands and their behavior I wouldn’t have even known to ask about them. So it’s super important to know how every minutia of every protocol works, right?

Presenting the Right Questions

Not exactly. We, as human computers, memorize the answers to more efficiently search through our database to find the right answers. If the problem takes 5 minutes to present we can eliminate a bunch of causes. If it’s happening in layer 3 and not layer 2 we can toss out a bunch of other stuff. Our knowledge is allowing us to discard useless possibilities and focus on the right result.

And it’s horribly inefficient. I can attest to that given my various attempts to learn OSPF hello and dead timers through osmosis of falling asleep in my big CCNP Routing book. The answers don’t crawl off the page and into your brain no matter how loudly you snore into it. So I spent hours learning something that I might use two or three times in my career. There has to be a better way.

Not coincidentally, that’s where the AI-driven systems from Mist, and now Juniper, come into play. Marvis is wonderful at looking at symptoms and finding potential causes. It’s what we do as humans. Except Marvis has no inherent biases. It also doesn’t misremember the values for a given protocol or get confused about whether or not OSPF point-to-point networks are broadcast or not. Marvis just knows what it was programmed with. But it does learn.

Learning is the key to how these AI and machine learning (ML) driven systems have to operate. People tend to discount solutions because they think there’s no way it could be that solution this time. For example, a haiku:

It’s not DNS.
Could it be DNS?
It was DNS.

DNS is often the cause of our problems even if we usually discount it out of hand in the first five minutes of troubleshooting. Even if it was only DNS 50% of the time we would still toss DNS as the root cause within the first five minutes because we’ve “trained” our brains to know what a DNS problem looks like without realizing how many things DNS can really affect.

But AI and ML don’t make these false correlations. Instead, they learn every time what the cause was. They can look at the network and see the failure state, present options based on the symptoms, and even if you don’t check in your changes they can analyze the network and figure out what change caused everything to start working again. Now, the next time the problem crops up, a system like Marvis can present you with a list of potential solutions with confidence levels. If DNS is at the top of the list, you might want to look into DNS first.

AI is going to make us all better troubleshooters because it’s going to make us all less reliant on poor memory. Instead of misremembering how a protocol should be configure, AI and ML will tell us how it should look. If something is causing routing loops or if layer 2 issues are happening because of duplex mismatches we’ll be able to see that quickly and have confidence it’s the right answer instead of just guessing and throwing things at the wall until they stick. Just like Google has supplanted the Cliff Claven people at the bar that are storehouses of useless knowledge, so too will AI and ML reduce our dependence on know-it-alls that may not have all the answers.


Tom’s Take

I’m ready to be forgetful. I’m tired of playing “stump the chump” in troubleshooting with the network playing the part of the stumper and me playing the chump. I’ve memorized more useless knowledge than I ever care to recall in my life. But I don’t want to have to do the work any longer. Instead, I want to apply my gifts to training algorithms with more processing power than me to do all the heavy lifting. I’m more than happy to look at DNS and EIGRP timers than try to remember if MTU and reliability are part of the K-values for this network.

The Future of Hidden Features

 

948AD2EC-79D1-4828-AF55-C71EA8715771You may have noticed last week that Ubiquiti added a new “feature” to their devices in a firmware updated. According to this YouTube video from @TomLawrenceTech, Ubiquiti built an new service that contacts a URL to “phone home” and check in with their servers. It got some heavy discussion going, especially on Reddit.

The consensus is that Ubiquiti screwed up here by not informing people they were adding the feature up front and also not allowing users to opt-out initially. The support people at Ubiquiti even posted a quick workaround of blocking the URL at a perimeter firewall to prevent the communications until they could patch in the option to opt-out. If this was an isolated incident I could see some manner of outcry about it, but the fact of the matter is that companies are adding these hidden features more and more every day.

The first issue comes from the fact that most release notes for apps any more are nothing aside from platitudes. “Hey, we fixed some bugs and stuff so turn on automatic updates so you get the best version of our stuff!” is somewhat common now when it comes to a list of changes. That has a lot to do with applications developers doing unannounced A/B testing with people. It’s not uncommon to have two identical version numbers of an app running two wildly different code releases or having dissimilar UIs just because some bean counter wants to know how well Feature X polls with a certain demographic.

Slipstreaming Stuff

While that’s all well and good for consumer applications, the trend is starting to seep into enterprise software as well. While we still get an exhaustive list of things that have been fixed in a release, we’re getting more than we bargained for on occasion as well. Doing a scrub of code to ensure that it actually fixes the bugs that you have in your environment is important for reliability purposes. When the quality of the code you’re trying to publish is of less-than-stellar quality, you have have to spend more time ensuring that the stated fixes aren’t going to cause issues during implementation.

But what about when the stated features aren’t the only things included? Could you imagine the nightmare of installing a piece of software to fix an issue only to find out that something that was hidden in the code, like a completely undocumented feature, caused some other issue elsewhere? And when I say “undocumented feature” I’m not using it as a euphemism for another bug. I’m talking about a service that got installed that no one knows about. Like a piece of an app that will be enabled at a later time for instance.

Remember Microsoft back in the early 90s? They were invincible. They had everything they wanted in the palm of their hand. And then their world exploded. They got sued by the US government in an anti-trust case. One of the things that came out of this lawsuit was a focus on undocumented features in Microsoft software. The government said that Microsoft was adding features to their application software that gave them performance advantages over other vendors. And only they knew about them.

Microsoft agreed to get rid of all undocumented features, including some of the Easter eggs that were hidden in their software. That irritated some people that enjoyed finding these fun little gifts. But in 2005, there was a great blog post that discussed why Microsoft has a strict “no Easter eggs” policy now. And reading it almost 15 years later makes it sound so prescient.

Security Sounds Simple, Right?

Undocumented features are a security risk. If there is code in your app that isn’t executing or hasn’t been completely checked against the interactions in your environment you’re adding a huge amount of risk. There can be interactions that someone doesn’t understand or that you have no way of seeing. And you can better believe that if the regulators of your industry find them before you do you’re going to have a lot of explaining to do.

That’s even if you notice it in the first place. How many times have you been told to whitelist a specific URL or IP stack to make something work? I can remember being told to whitelist 17.0.0.0/8 for Apple to make their push notification service work. That’s an awful lot of IP addresses to enable push notifications! And that’s a lot of data that could be corrupted or misused by a smart threat actor.

The solution we need to fix the problem is actually pretty simple. We need to push back against the idea that we can slip undocumented features into updates. Release notes need to list everyting that comes in the software package. We need to tell our vendors and companies that we have to have a full listing of the software features. And regulatory bodies need to be ready to share the blame when someone breaks those rules instead of punishing the people that had no idea there was something in the code that was misbehaving.


Tom’s Take

I’m not a fan of finding things I wasn’t expecting. At one point my wife and I had the latest Facebook app updates and somehow her UI looked radically different than mine. But if it’s on my phone or my home computer I don’t really have much to complain about. Finding undocumented apps and features in my enterprise software is a huge issue though. Security is paramount and undocumented code is an entry point to disaster. We are the only ones that can stem the tide by pushing back against this practice now before it becomes commonplace in the future.

In Defense of Support

We’re all in IT. We’ve done our time in the trenches. We’ve…seen things, as Roy Batty might say. Things you wouldn’t believe. But in the end we all know the pain of trying to get support for something that we’re working on. And we know how painful that whole process can be. Yet, how is it that support is universally “bad” in our eyes?

One Of Us

Before we launch into this discussion, I’ll give you a bit of background on me. I did inbound tech support for Gateway Computers for about six months at the start of my career. So I wasn’t supporting enterprises to start with but I’ve been about as far down in the trenches as you can go. And that taught me a lot about the landscape of support.

The first thing you have to realize is that most Tier 1 support people are, in fact, not IT nerds. They don’t have a degree in troubleshooting OSPF or are signatories to the fibre channel standards. They are generally regular people. They get a week or two of training and off they go. In general the people on the other end of the support phone number are better phone people than they are technical people. The trainers at my old job told me, “We can teach someone to be technical. But it’s hard to find someone who is pleasant on the phone.”

I don’t necessarily agree with that statement but it seems to be the way that most companies staff their support lines. Why? Ask yourself a quick question: How many times has Tier 1 support solved your issue? For most of us the answer is probably “never” or “extremely rarely”. That’s because we, as IT professionals, have spent a large amount of our time and energy doing the job of a Tier 1 troubleshooter already. We pull device logs and look at errors and Google every message we can find until we hit a roadblock. Then it’s time to call. However, you’re already well past what Tier 1 can offer.

Look at it from the perspective of the company offering the support. If the majority of people calling me are already past the point of where my Tier 1 people can help them, why should I invest in those people? If all they’re going to do is take information down and relay the call to Tier 2 support how much knowledge do they really need? Why hire a rock star for a level of support that will never solve a problem?

In fact, this is basically the job of Tier 1 support. If it’s not a known issue, like an outage or a massive bug that is plastered up everywhere, they’re going to collect the basic information and get ready to pass the call to the next tier of support. Yes, it sucks to have to confirm serial numbers and warranty status with someone that knows less about VLANs than you do. But if you were in the shoes of the Tier 2 technician would you want to waste 10 minutes of the call verifying all that info yourself? Or would you rather put your talent to use to get the problem solved?

Think about all the channel partners and certification benefits that let you bypass Tier 1 support. They’re all focused on the idea that you know enough about the product to get to the next level to help you troubleshoot. But they’re also implying that you know the customer’s device is in warranty and that you need to pull log files and other data before you open a ticket so there’s no wasted time. But you still need to take the time to get the information to the right people, right? Would you want to start troubleshooting a problem with no log files and only a very basic description of the issue? And consider that half the people out there just put “Doesn’t Work” or “Acting Weird” for the problem instead of something more specific.

Sight Beyond Sight

This whole mess of gathering info ahead of time is one of the reasons why I’m excited about the coming of network telemetry and network automation. That’s because you now have access to all the data you need to send along to support and you don’t need to remember to collect it. If you’ve ever logged into a switch and run the show tech-support command you probably recall with horror the console spam that was spit out for minutes upon minutes. Worse yet, you probably remember that you forgot to redirect that spam to a file on the system to capture it all to send along with your ticket request.

Commands like show tech-support are the kinds of things that network telemetry is going to solve. They’re all-in-one monsters that provide way too much data to Tier 2 support techs. If the data they need is in there somewhere it’s better to capture it all and spend hours poring through a text file than talk to the customer about the specific issue, right?

Now, imagine the alternative. A system that notices a ticket has been created and pulls the appropriate log files and telemetry. The system adds the important data to the ticket and gets it ready to send to the vendor. Now, instead of needing to provide all the data to the Tier 1 folks you can instead open a ticket at the right level to get someone working on the problem. Mean Time to Resolution goes down and everyone is happy. Yes, that does mean that there are going to be fewer Tier 1 techs taking calls from people that don’t have a way to collect the data ahead of time but that’s the issue with automating a task away from someone. If the truth be told they would be able to find a different job doing the same thing in a different industry. Or they would see the value of learning how to troubleshoot and move up the ladder to Tier 2.

However, having telemetry gathered automatically still isn’t enough for some. The future is removing the people from the chain completely. If the system can gather telemetry and open tickets with the vendor what’s stopping it from doing the same proactively? We already have systems designed to do things like mail hard disk drives to customers when the storage array is degraded and a drive needs to be replaced. Why can’t we extend things like that to other parts of IT? What if Tier 2 support called you for a change and said, “Hey, we noticed your routing protocol has a big convergence issue in the Wyoming office. We think we have a fix so let’s remote in and fix it together.” Imagine that!


Tom’s Take

The future isn’t about making jobs go away. It’s about making everything more efficient. I don’t want to see Tier 1 support people lose their jobs but I also know that most people consider their jobs pointless already. No one wants to deal with the info takers and call routers. So we need to find a way to get the right information to the people that need it with a minimum of effort. That’s how you make support work for the IT people again. And when that day comes and technology has caught up to the way that we use support, perhaps we won’t need to defend them anymore.

The End of SD-WAN’s Party In China

As I was listening to Network Break Episode 257 from my friends at Packet Pushers, I heard Greg and Drew talking about a new development in China that could be the end of SD-WAN’s big influence there.

China has a new policy in place, according to Axios, that enforces a stricter cybersecurity stance for companies. Companies doing business in China or with offices in China must now allow Chinese officials to get into their networks to check for security issues as well as verifying the supply chain for network security.

In essence, this is saying that Chinese officials can have access to your networks at any time to check for security threats. But the subtext is a little less clear. Do they get to control the CPE as well? What about security constructs like VPNs? This article seems to indicate that as of January 1, 2020, there will be no intra-company VPNs authorized by any companies in China, whether Chinese or foreign businesses in China.

Tunnel Collapse

I talked with a company doing some SD-WAN rollouts globally in China all the way back in 2018. One of the things that was brought up in that interview was that China was an unknown for American companies because of the likelihood of changing that model in the future. MPLS is the current go-to connectivity for branch offices. However, because you can put an SD-WAN head-end unit there and build an encrypted tunnel back to your overseas HQ it wasn’t a huge deal.

SD-WAN is a wonderful way to ensure your branches are secure by default. Since CPE devices “phone home” and automatically build encrypted tunnels back to a central location, such as an HQ, you can be sure that as soon as the device powers on and establishes global connectivity that all traffic will be secure over your VPN until you change that policy.

Now, what happens with China’s new policy? All traffic must transit outside of a VPN. Things like web traffic aren’t as bad but what about email? Or traffic destined for places like AWS or Azure? It was an unmentioned fact that using SD-WAN VPNs to transit through the content filters in place in China was a way around issues that might arise from accessing resources inside of a very well secured country-wide network.

With the policy change and enforcement guidelines set forth to be enacted in 2020, this could be a very big deal for companies hoping to use SD-WAN in China. First and foremost, you can’t use your intra-company VPN functions any longer. That effectively means that your branch office can’t connect to the HQ or the rest of your corporate network. Given some of the questions around intellectual property issues in China that might not be a bad thing. However, it is going to cause issues for your users trying to access the mail and other support services. Especially if they are hosted somewhere that is going to create additional scrutiny.

The other potential issue is whether or not Chinese officials are even going to allow you to use CPE of your own choosing in the future. If the mandate is that officials should have access to your network for security concerns, who is to say they can’t just dictate what CPE you should use in order to facilitate that access. Larger companies can probably negotiate for come kind of on-site server that does network scanning. But smaller branches are likely going to need to have an all-in-one device at the head end doing all the work. The additional benefit for the Chinese is that control of the head end CPE ensures that you can’t build a site-to-site VPN anywhere.

Peering Into The Future

Greg and Drew pontificate a bit on the future on what this means for organizations from foreign countries doing business in China in the future. I tend to agree with them on a few points. I think you’re going to see a push for Chinese offices of major companies treating them like zero-trust endpoints. All communications will be trading minimal information. Networks won’t be directly connected, either by VPN substitute or otherwise.

Looking further down the road makes the plans even more murky. Is there a way that you can certify yourself to have a standard for cybersecurity? We have something similar with regulations here in the US were we can submit compliance reports for various agencies and submit to audits or have audits performed by third parties. But if the government won’t take that as an answer how do you even go about providing the level of detail they want? If the answer is “you can’t”, then the larger discussion becomes whether or not you can comply with their regulations and reduce your business exposure while still making money in this market. And that’s a conversation no technology can solve.


Tom’s Take

SD-WAN gives us a wonderful set of features included in the package. Things like application inspection are wonderful to look at on a dashboard but I’ve always been a bigger fan of the automatic VPN service. I like knowing that as soon as I turn up my devices they become secure endpoints for all my traffic. Alas, all the technology in the world can be defeated by business or government regulation. If the rules say you can’t have a feature, you either have to play by the rules or quit playing the game. It’s up to businesses to decide how they’ll react going forward. But SD-WAN’s greatest feature may now have to be an unchecked box on that dashboard.

Locked Up By Lock-In

When you start evaluating a solution, you are going to get a laundry list of features and functionality that you are supposed to use as criteria for selection. Some are important, like the ones that give you the feature set you need to get your job done. Others are less important for the majority of use cases. One thing tends to stand out for me though.

Since the dawn of platforms, I believe the first piece of comparison marketing has been “avoids lock-in”. You know you’ve seen it too. For those that may not be completely familiar with the term, “lock-in” describes a platform where all the components need to come from the same manufacturer or group of manufacturers in order to work properly. An example would be if a networking solution required you to purchase routers, switches, access points, and firewalls from a single vendor in order to work properly.

Chain of Fools

Lock in is the greatest asset a platform company has. The more devices they can sell you the more money they can get from you at every turn. That’s what they want. So they’re going to do everything they can to keep you in their ecosystem. That includes things like file formats and architectures that require the use of their technology or of partner technologies to operate correctly.

So, the biggest question here is “What’s wrong with that?” Note that I’m not a proponent of lock-in. Rather, I’m against the false appearance of choices. Sure, offering a platform with the promise of “no lock-in” is a great marketing tool. But how likely are you to actually follow through on that promise?

I touched on this a little bit earlier this year at Aruba Atmosphere 2019 when I talked about the promise of OpenConfig allowing hardware from different vendors to all be programmed in a similar way. The promise is grand that you’ll be able to buy an access point from Extreme and run it on an Aruba Controller while the access layer polices are programmed into Cisco switches. It’s the dream of interoperability!

More realistically, though, you’ll find that most people aren’t that concerned about lock-in. The false choice of being open to new systems generally comes down to one single thing: price. The people that I know that complain the most about vendor lock-in almost always follow it up with a complaint about pricing or licensing costs. For example:

The list could go on for three or four more pages. And the odds are good you’ve looked at one of those solutions already or you’re currently dealing with something along those lines. So, ask yourself how much pain vendor lock-in brings you aside from your checkbook?

The most common complaint, aside from price, is that the vendor solution isn’t “best of breed”. Which has always been code for “this particular piece sucks and I really wish I could use something else”. But there’s every possibility that the solution sucks because it has to integrate tightly with the rest of the platform. It’s easy to innovate when you’re the only game in town and trying to get people to buy things from you. But if you’re a piece of a larger puzzle and you’re trying to have eight other teams tell you what your software needs to do in order to work well with the platform, I think you can see where this is going.

How many times have you actually wished you could pull out a piece and sub in another one? Again, aside from just buying the cheapest thing off the shelf? Have you ever really hoped that you could sub in an Aerohive AP630 802.11ax (Wi-Fi 6) AP into your Cisco wireless network because they were first to market? Have you ever really wanted to rip out Cisco ISE from your integrated platform and try to put Aruba ClearPass in its place? Early adopters and frustrated users are some of the biggest opponents of vendor lock-in.

Those Three Words

I’m about to tell you why lock-in isn’t the demon you think it is. And I can do it in three words:

It. Just. Works.

Granted, that’s a huge stretch and we all know it. It really should be “well built software that meets all my functionality goals should just work”. But in reality, the reason why we all like to scream at lock-in when we’re writing checks for it is because the alternative to paying big bucks for making it “all just work” is for us to invest our time and effort into integrating two solutions together. And ask your nearest VAR or vendor partner how much fun that can be?

I used to spend a LOT of my time trying to get pieces and parts to integrate because schools don’t like to spend a lot of money on platforms. In Oklahoma they’re even allowed to get out of license agreements every year. Know what that means? A ton of legacy software that’s already paid for sitting around waiting to run on brand new hardware they just bought. And oh, by the way, can you make all that work together in this new solution we were sold by the Vendor of the Month Club?

And because they got the best deal or the best package, I had to spend my time and effort putting things together. So, in a way, the customer was trading money from the hardware and software to me for my wetware — the brain power needed to make it all work together. So, in a way, I was doing something even worse. I was creating my own lock-in. Not because I was building an integrated solution with one vendor’s pieces. But because I was building a solution that was custom and difficult to troubleshoot, even with proper documentation.


Tom’s Take

Lock-in isn’t the devil. You’re essentially trading flexibility for ease-of-use. You’re trading the ability to switch out pieces of a solution for the ability to integrate the pieces together without expending much effort. And yes, I realize that some lock-in solutions are harder to integrate than others. I’m looking at you, Cisco ISE. But, that just speaks to how hard it is to create the kind of environment that you get when you are trying to create all kinds of integrations. You’ll find that the idea of freedom of choice is much more appealing than actually having the ability to swap things out at will.