Redundancy Is Not Resiliency

Most people carry a spare tire in their car. It’s there in case you get a flat and need to change the tire before you can be on your way again. In my old VAR job I drove a lot away from home and to the middle of nowhere so I didn’t want to rely on roadside assistance. Instead I just grabbed the extra tire out of the back if I needed it and went on my way. However, the process wasn’t entirely hitless. Even the pit crew for a racing team needs time to change tires. I could probably get it done in 20 minutes with appropriate cursing but those were 20 minutes that I wasn’t doing anything else beyond fixing a tire.

Spare tires are redundant. You have an extra thing to replace something that isn’t working. IT operations teams are familiar with redundant systems. Maybe you have a cold spare on the shelf for a switch that might go down. You might have a cold or warm data center location for a disaster. You could even have redundant devices in your enterprise to help you get back in to your equipment if something causes it to go offline. Well, I say that you do. If you’re Meta/Facebook you didn’t have them this time last year.

Don’t mistake redundancy for resilience though. Like the tire analogy above you’re not going to be able to fix a flat while you’re driving. Yes, I’ve seen the crazy video online of people doing that but aside from stunt driving you’re going to have to take some downtime from your travel to fix the tire. Likewise, a redundant setup that includes cold spares or out-of-band devices that are connected directly to your network could incur downtime if they go offline and lock you out of your management system. Facebook probably thought their out-of-band control system worked just fine. Until it didn’t.

The Right Gear for Resilience

At Networking Field Day 29 last week we were fortunate to see Opengear present again for the second time. I’m familiar with them from all the way back at Networking Field Day 2 in 2011 so their journey through the changes of networking over the past decade has been great to see. They make out-of-band devices and they make them well. They’re one of the companies that immediately spring to mind when you think about solutions for getting access to devices outside the normal network access method.

As a VAR there were times that I needed to make calls to locations in order to reboot devices or get console access to fix an issue. Whether it was driving 3 hours to press F1 to clear a failed power supply message or racing across town to restore phone service after locking myself out of an SSH session there are numerous reasons why having actual physical access to the console is important. Until we perfect quantum teleportation we’re going to have to solve that problem with technology. Here’s a video from the Networking Field Day session that highlights some of the challenges and solutions that Opengear has available:

Ryan Hogg brings up a great argument here for redundancy versus resiliency. Are you managing your devices in-band? Or do you have a dedicated management network? And what’s your backup for that dedicated network if something goes offline? VLAN separation isn’t good enough. In the event of a failure mode, such as a bridging loop or another attack that takes a switch offline you won’t be able to access the management network if you can’t sent packets through it. If the tire goes flat you’re stopped until it’s fixed.

Opengear solves this problem in a number of ways. The first is of course providing a secondary access method to your network. Opengear console devices have a cellular backup function that can allow you to access them in the event of an outage, either from the internal network or from the Internet going down. I can think of a couple of times in my career where I would have loved to have been able to connect to a cellular interface to undo a change that just happened that had unintended consequences. Sometimes reload in 5 doesn’t quite do the job. Having a reliable way to connect to your core equipment makes life easy for network operating systems that don’t keep from making mistakes.

However, as mentioned, redundancy is not resiliency. It’s not enough for us to have access to fix the problem while everything is down and the world is on fire. We may be able to get back in and fix the issue without needing to drive to the site but the users in that location are still down while we’re working. SD-WAN devices have offered us diverse connectivity options for a number of years now. If the main broadband line goes down just fail back to the cellular connection for critical traffic until it comes back up. Easy to do now that we have the proper equipment to create circuit diversity.

As outlined in the video above, Opengear has the same capability as well. If you don’t have a fancy SD-WAN edge device you can still configure Opengear console devices to act as a secondary egress point. It’s as simple as configuring the network with an IP SLA to track the state of the WAN link and installing the cellular route in the routing table if that link goes down. Once configured your users can get out on the backup link while you’re coming in to fix whatever caused the issue. If it’s the ISP having the issue you can log a ticket and confirm things are working on-site without having to jump in a car to see what your users see.

Resilience Really Matters

One of the things that Opengear has always impressed me with is their litany of use cases for their devices. I can already think of a ton of ways that I could implement something like this for customers that need monitoring and resilient connectivity options. Remote offices are an obvious choice but so too are locations with terrible connectivity options.

If you are working in a location with spotty connectivity you can easily deploy an Opengear device to keep an eye on the network and/or servers as well as providing an extra way for the site to get back online in the event of an issue. If the WAN circuit goes down you can just hop over to the cellular link until you get it fixed. Opengear will tell you something happened and you can log into the Lighthouse central management system to go there and collect data. If configured correctly your users may not even realize they’re offline! We’re almost at the point of changing the tire while we’re driving.


Tom’s Take

I am often asked if I miss working on networking equipment since I rarely touch it these days. As soon as I’m compelled to answer that question I remember all the times I had to drive somewhere to fix an issue. Wasted time can never be recovered. Resources cost money whether it’s money for a device or time spent going to fix one. I look at the capabilities that a company like Opengear has today and I wish I had those fifteen years ago and could deploy them to places I know needed them. In my former line of work redundant things were hard to come by. Resilient options were much more appealing because they offered more than just a back plan in case of failure. You need to pick resiliency every time because otherwise you’re going to be losing time replacing that tire when you could be rolling along fixing it instead.


Disclaimer

Opengear was a presenter at Networking Field Day 29 on September 7, 2022. I am an employee of Tech Field Day, which is the company that managed the event. This blog post represents my own personal thoughts about the presentation and is not the opinion of Tech Field Day. Opengear did not provide compensation for this post or ask for editorial approval. This post is my perspective alone.

Brand Protection

I woke up at 5am this morning to order a new iPhone. I did this because I wanted the new camera upgrades along with some other nice-to-haves. Why did I get an iPhone and not a new Samsung? Why didn’t I look at any of the other phones on the market? It’s because I am a loyal Apple customer at this point. Does that mean I think the iPhone is perfect? Far from it! But I will choose it in spite of the flaws because I know it has room to be better.

That whole story is repeated time and again in technology. People find themselves drawn to particular companies or brands. They pick a new phone or computer or car based on their familiarity with the way they work or the design choices that are made. But does that mean they have to be loyal to that company no matter what?

Agree to Disagree

One of the things that I feel is absolutely paramount to being a trusted advisor in the technology space is the ability to be critical of a product or brand. If you look at a lot of the ambassador or influencer program agreements you’ll see language nestled toward the bottom of the legalese. That language usually states you are not allowed to criticize the brand for their decisions or talk about them in a disparaging way. In theory the idea is important because it prevents people from signing up for the program and then using the platform to harshly and unfairly criticize the company.

However, the dark side of those agreements usually outweigh the benefits. The first issue is that companies will wield the power to silence you to great effect. The worst offenders will have you removed from the program and potentially even sue you. Samsung almost stranded bloggers 10 years ago because of some brand issues. At the time it seemed crazy that a brand would do that. Today it doesn’t seem quite so far-fetched.

The second issue is that those agreements are written in such a way as to be able to cause issues for you even if you didn’t realize you were doing something you weren’t supposed to be doing. Think about celebrities that have tweeted about a new Android phone and the tweet has metadata that says sent with Twitter for iPhone. How about companies that get very upset when you discuss companies that they see as competitors. Even if you don’t see them as competitors or don’t see the issue with it you may find yourself running afoul of the brand when they get mad about you posting a pic of their product next to the supposed competition.

In my career I’ve worked at a value-added reseller (VAR) where I found myself bound by certain agreements to talk positively about brands. I’ve also found myself on the wrong side of the table when that brand went into a bidding process with another VAR and then tried to tell me I could say bad things about them in the process because I was also their partner. The situation was difficult because I was selling against a partner that went with another company but I also needed to do the work to do the bid. Hamstringing me by claiming I had to play by some kind of weird rules ultimately made me very frustrated.

Blind Faith

Do companies really want ambassadors that only say positive things about the brand? Do they want people to regurgitate the marketing points with everyone and never discuss the downsides of the product? Would you trust someone that only ever had glowing things to say about something you were trying to buy?

The reality of our world today is that the way that people discuss products like this influences what we think about them. If the person doing the discussion never has a negative thing to say about a company then it creates issues with how they are perceived. It can create issues for a supposedly neutral or unbiased source if they only ever say positive things, especially if it later comes out they weren’t allowed to say something negative for fear they’d get silenced or sued.

Think about those that never say anything negative toward a brand or product. You probably know them by a familiar epithet: fanboys. Whether it’s Apple or Tesla or Android or Ford there are many people out there that aren’t just bound by agreement to always speak positively about something. They will go out of their way to attack those that speak ill of their favorite product. If you’ve every had an interaction with a fan online that left you shaking your head because you can’t understand why they don’t see the issues you know how difficult that conversation can be.

As a company, you want people discussing the challenges your product could potentially face. You want an honest opinion that it doesn’t fit in a particular vertical, for example. Imagine how upset a customer would be if they bought your product based on a review from a biased influencer only to find that it didn’t fit your need because the influencer couldn’t say anything negative. Would that customer be happy with your product? Would the community trust that influencer in the future?


Tom’s Take

Honesty isn’t negativity. You can be critical of something you enjoy and not insinuate you’re trying to destroy it. I’ll be the first person to point out the shortcomings of a product or company. I’ll be fair but honest. I’ll point out where the improvements need to be made. One of the joys of my day job at Tech Field Day is that I have the freedom to say what I want in my private life and not worry about my work agreements getting me in trouble as has happened with some in the past. I’ll always tell you straight up how I feel. That’s how you protect your brand. Not with glowing reviews but with honest discussion.

When Were You Last a Beginner?

In a couple of weeks I’m taking the opportunity to broaden my leadership horizons by attending the BSA leadership course known as Philmont Leadership Challenge. It’s a course that builds on a lot of the things that I’ve been learning and teaching for the past five years. It’s designed to be a sort of capstone for servant leadership and learning how to inspire others. I’m excited to be a part of it in large part because I get to participate for a change.

Being a member of the staff for my local council Wood Badge courses has given me a great opportunity to learn the material inside and out. I love being able to teach and see others grow into leaders. It’s also inspired me to share some of those lessons here to help others in the IT community that might not have the chance to attend a course like that. However the past 3 years have also shown me the value of being a beginner at something from time to time.

Square One

Everyone is new at something. No one is born knowing every piece of information they’ll need to know for their entire lives. We learn language and history and social skills throughout our formative years. When we get to our career we learn skills and trades and figure out how to do complex things easily. For some of us we also learn how to lead and manage others. It’s a process of building layer upon layer to be better at what we do. Those skills give us the chance to show how far we’ve come in a given area by the way we understand how the complex things we do interact.

One of my favorite stories about this process is when I first started studying for my CCIE back in 2008. I knew the first place I should look was the Cisco Press certification guide for the written exam. As I started reading through the copy I caught myself thinking, “This is easy. I already know this.” I even pondered why I bothered with those pesky CCNP routing books because everything I needed to know was right here!

The practitioners in the audience have already spotted the logical fallacy in my thinking. The CCIE certification guide was easy and remedial for me because I’d already spent so much time reading over those CCNP guides. And those CCNP guides only made sense to me because I’d studied for my CCNA beforehand. The advanced topics I was refreshing myself on could be expanded because I understood the rest of the information that was being presented already.

When you’re a beginner everything looks bigger. There’s so much to learn. It’s worrisome to try and figure out what you need to know. You spend your time categorizing things that might be important later. It can be an overwhelming process. But it’s necessary because it introduces you to the areas you have to understand. You can’t start off knowing everything. You need to work you way into it. You need to digest information and work with it before moving on to add more to what you’ve learned. Trying to drink from a firehose makes it impossible to do anything.

However, when you approach things from a perspective of an expert you lose some of the critical nature of being bad at something. You might think to yourself that you don’t need to remember a protocol number or a timer value because “they never worry about that anyway”. I’ve heard more than a few people in my time skip over valuable information at the start of a course because they want to get to the “good stuff” that they just know comes later. Of course, skipping over the early lessons means they’re going to be spending more time reviewing the later information because they missed the important stuff up front.

Those Who Teach

You might think to yourself that teaching something is a harder job. You need to understand the material well enough to instruct others and anticipate questions. You need to prep and practice. It’s not easy. But it also takes away some of the magic of learning.

Everyone has a moment in their journey with some technology or concept where everything just clicks. You can call it a Eureka moment or something similar but we all remember how it felt. Understanding how the pieces fit together and how you grasp that interconnection is one of the keys to how we process complex topics. If you don’t get it you may never remember it. Those moments mean a lot to someone at the start of their journey.

When you teach something you have to grasp it all. You may have had your Eureka moment already. You’re also hoping that you can inspire one in others. If you’re trying to find ways to impart the knowledge to others based on how you grasped it you may very well inspire that moment. But you also don’t have the opportunity to do it for yourself. We’re all familiar with the old adage that familiarity breeds contempt. It’s easy to fall into that trap with a topic you are intimately familiar with.

In your career have you ever asked a question about a technical subject to an expert that started their explanation with “it’s really easy…”? Most of us have. We’ve probably even said that phrase ourselves. But it’s important to remember that not everyone has had the same experiences. Not everyone knows the topic to the level that we know it. And not everyone is going to form the same connections to recall that information when they need it again. It may be simple to you but for a beginner it’s a difficult subject they’re struggling to understand. How they comprehend it relies heavily on how you impart that knowledge.

Wide Eyed Wonder

Lastly, the thing that I think is missing in the expert level of things is the wonder of learning something new for the first time. It’s easy to get jaded when you have to take in a new piece of information and integrate it into your existing view. It can be frustrating in cases where the new knowledge conflicts with old knowledge. We spent a lot of time learning the old way and now we have to change?

Part of the value of being a beginner is looking at things with fresh eyes. No doubt you’ve heard things like “this is the way we’ve always done it” in meetings before. I’ve written about challenging those assumptions in the past and how to go about doing it properly but having a beginner perspective helps. Pretend I’m new to this. Explain to me why we do it that way. Help me understand. By taking an approach of learning you can see the process and help fix the broken pieces or optimize the things that need to be improved.

Even if you know the subject inside and out it can be important to sit back and think through it from the perspective of a beginner. Why is a vanilla spanning tree timer 50 seconds? What can be improved in that process? Why should things not be hurried. What happens when things go wrong? How long does it take for them to get fixed? These are all valid beginner questions that help you understand how others look at something you’re very familiar with. You’ll find that being able to answer them as a beginner would will lead to even more understanding of the process and the way things are supposed to work.


Tom’s Take

There are times when I desperately want to be new at something again. I struggle with finding the time to jump into a new technology or understand a new concept because my tendency is to want to learn everything about it and there are many times when I can’t. But the value of being new at something isn’t just acquiring new knowledge. It’s learning how a beginner thinks and seeing how they process something. It’s about those Eureka moments and integrating things into your process. It’s about chaos and change and eventually understanding. So if you find yourself burned out it’s important to stop and ask when you were last a beginner.

Certification Comfort Food

I’m a big fan of comfort food. Maybe more than I should be. The idea of something simple and tasty just hits the right spot a lot of time, especially when I’m stressed or don’t have time to do something more involved. I know I really need to be better about cooking but you can’t beat a quick meal that uses something simple and gets the job done, right?

Now, before you ask yourself what I’m on about this week, I want you to think about that analogy in terms of certifications and learning. When we’re starting out in the industry or we’re learning a new skill we have to pick up basic ideas. The more advanced or radical the technology the more we need the kinds of explanations that make the concepts simple to understand. We need the equivalent of learning comfort food. Simple, digestible, and easy to prepare.

Climbing the Ladder

As our skills improve we have the choice to continue on and develop our capabilities to greater depths. Perhaps we want to learn everything there is to know about BGP and policies. We could even parlay that networking knowledge into new adjacencies that build on our skill sets. We also have the option of staying in the basic level and honing those skills. Instead of learning VXLAN we could spend a thousand hours practicing all the ways that you can configure a VLAN.

Which way is right? Is there a need to make a choice? People are going to feel more comfortable doing one thing over the other in almost every case. If you’re like me you want to get to the bottom of every mystery and explore every nuance of something. Once you figure it out you’re going to want to move on to the next hard problem to solve. You become a voracious reader and consumer of knowledge and before you know it you’ve run out of things to consume. It’s partially the reason why I’ve been such a prolific writer for the past twelve years. I’ve been creating the content that I wanted to consume so others can benefit.

The other side of the choice is being content with the skills you have. This is in no way a negative thing. Not everyone that cooks needs to be a four star chef that makes perfect risotto and Beef Wellington every time. There is a place for everyone that learns enough to accomplish their goals and decides that is enough for them. If the above option is the “pull” model where one is trying to pull in new knowledge as fast as possible then this is the “push” version where people must be pushed to learn additional things. Your company might move to cloud and that would facilitate a need to pick up cloud operations skills to complement the ones you have for the network or the virtualization cluster. You’re not actively seeking the knowledge until it’s needed.

Boiling the Mudpuddle

It’s all well and good when you can recognize which type of learner you are. It’s also important to know where your resources are aimed. If your top destinations for content are part of the “push” model and aim at a lower level when you’re someone that wants to grow and investigate new areas you’re going to hit a wall eventually and sour on them.

A personal story for me comes when I was racing through my certification journey in the early part of my career. Once I started with Cisco I was consuming books left and right. Every time I went into the book store I picked up a new tome to teach me more about routing or remote access networks or even firewalls. I would consume that content whenever I could and apply those lessons to my job or my certification process. Eventually I knew I was reaching a limit because there were fewer and fewer books in the bookstore that taught me things I wanted to know. It made me realize there is a target market for these resources.

Things like certification guides are aimed at a wide market. They want to teach skills to the widest possible audience. Not everyone needs to know the ins and outs of EVPN but most everyone in networking needs to know how a switch forwards frames. If you want to sell the most books which would you write about? You’d write the one that covers the most people. It’s a reality of the market. Content for the entry level and the broadest group sells the best. In today’s world the book has been replaced by the blog and the YouTube channel.

As mentioned, I started my blogging career because of the above bookstore issue. Once I started learning things that weren’t in every book I wanted to share those ideas. That got me to Tech Field Day and eventually to different things. It also made me realize that while my content may never have hundreds of thousands of readers for every post it would serve people that needed to find those lessons or understand those topics in a depth that was beyond a paragraph or two in a 400-page encyclopedia of terminology.

To me, the certification comfort food is that entry-level content. It’s always going to be there. It’s simple to write about, especially when you have good analogies to frame new concepts for people. It’s tasty when you’re starving. And you can make a very good living doing it. But if you’re the kind of person that wants to try new tastes and break away from the comfort and ease you’re going to need to figure out your own path. You need to experiment and make mistakes and struggle to conceptualize what you’re talking about. You need to expand your horizons and do new things and then tell the world how you did it. Like a recipe blog or TikTok channel for cooking you’re going to need to put your crazy ideas out there and see how it goes.


Tom’s Take

There are a lot of great creators out there that have made a very good place for themselves teaching newcomers the basics of how things work. I applaud them and wish them nothing but success. I also know that’s not for me. I started writing about my CCIE studies and the challenges I was solving the real world. Now I write about the state of the market or the changing of tech or how to build and lead teams. It’s very representative of my journey as well as the journeys of those in the community that I talk to. My very nature won’t let me stay in a little bubble and create the same things in new ways. I’m going to push the envelope and explore new things. It means I might not land in everyone’s top list but it also means I won’t be bored. Why be mac-n-cheese when I really need to be risotto?

The Puzzle of Peering with Kentik

If you’ve worked at an ISP or even just closely with them you’ve probably hearing the term peering quite a bit. Peering is essentially a reciprocal agreement to provide access to networks between two providers. Provider A agrees to allow Provider B to send traffic over and through their network in exchange for the same access in the other direction. Sounds easy, right? On a technical level it is pretty easy. You simply set up a BGP session with the partner provider and make sure all the settings match and you’ve got things rolling.

The technical part isn’t usually where peering gets complicated. Instead it’s almost always related to the business side of things. The policy and negations that have to happen for a good peering agreement take way more time that hammering out some BGP configuration stanzas. The amount of traffic to be sent, the latency requirements, and even the cost of the agreement are all things that have to be figured out before the first hello packet can be exchanged. This agreement is always up for negotiation too, since the traffic patterns can change before you realize it and put you at a disadvantage.

Peerless Data Collection

If you want to get the most out of your peering arrangement you need to know what’s going on. You need to have statistics about the key points of your agreement. You need to know if you’re holding up your end of the bargain as well as the company you’re working with. If you walk into a peering negotiation without the right data you’re going to be working from a disadvantage right away.

For example, did your partner company take all of the traffic they agreed to accept in the peering contract? Or did they have issues that forced you to send the traffic along a different route? Were you forced to send that traffic across a different route that had a higher cost? Did your users complain about network speed because of congestion outside of your control? If you can’t put your fingers on the answers to these questions quickly you’re going to find yourself with lots of angry users and customers not to mention peering partners that want answers from you as well.

Recently I had a chance to listen to a great presentation from Kentik during Networking Field Day: Service Provider. Nina Bargisten laid out some of the challenges that Kentik customers face with peering arrangements and how Kentik is helping to solve them:

One of the points that Nina discusses is that capacity planning is a huge undertaking for ISPs. With the supply chain issues that we’re currently facing in 2022 it’s not easy to order equipment to alleviate congestion problems. Even under somewhat normal circumstances it’s not likely that an ISP is going to go out and order a lot of new hardware just to deal with congestion. They might change some polices to route traffic in different directions but ultimately the decision has to be made about how to get customer packets through and out of their network to the ultimate destination.

Peering agreements can help with congestion. Adding more exit points to your network means some flows can exit through a different provider and either get to their destination faster or prevent a larger connection from being overwhelmed and congested. It’s not unlike having multiple options to use to arrive a destination when driving. Some streets are better for smaller amounts of traffic compared to larger highways and interstates that provide high-speed travel.

As mentioned above, it’s critical that you have data on your traffic and its performance. Are you sending everything through one route? Are you peering with providers that are getting less than half of the traffic load they agreed to take? These are all questions you have to ask to create a capacity plan. If you’re hearing complaints about congestion but you see that only two of your outbound connections are running a full capacity while the rest are sitting idle then you don’t have a congestion issue as much as you have a configuration problem to solve.

Kentik’s solution allows you to see what’s going on and help you make better decisions about the routes that traffic should be taking. As demonstrated above, their dashboard collects data from your network as well as many others and can tell you when you need to be configuring polices to send traffic to low volume peers instead of relying on congested links. It will also help you see trends for when links become congested and allow you to set thresholds to divide your traffic appropriately before it becomes an issue.

There’s a lot more info in the video above to help you with your capacity planning and peering negotiations. It all comes down to a simple maxim: Information is key. You can’t solve these puzzles without knowing what you have and what you need. If you’re just going to keep throwing peering agreements at a problem until it goes away you’re going to fail. You won’t solve your real issues by just adding another connection that never gets used. Instead you can use Kentik’s platform to provide the kinds of insights that will help you create value for your customers and save money at the same time.


Tom’s Take

Service providers think about traffic differently than enterprise admins. They have to worry about it coming into the network and leaving again. Instead of worrying about a couple of links to the wider Internet they have to worry about dozens. If you think it’s hard keeping track of all that data for the enterprise you can just imagine how hard it is when you scale it up to the service provider level. Thankfully companies like Kentik are applying their expertise to provide actionable information to help you make the right choices and maybe even negotiate some better deals.

If you’d like to learn more about this presentation, make sure you check out the full presentation on the Tech Field Day site or go to http://Kentik.com

Authority and Responsibility

Congratulations on your promotion! You’re now a manager or leader for your team. You now have to make sure everyone is getting their things done. That also means lots of reports and meetings with your manager about what’s happening and all the new rules that have to be followed in the future. Doesn’t this all sound nice?

In truth we all want to be able to help out as much as possible. Sometimes that means putting in extra work. For many it also means being promoted to a position of responsibility in a company leading a team or group of teams. That means you will have some new responsibilities and also some new authority. But what’s the difference? And why is one more foundational than the other?

Respect My Authority

Authority is “power to influence or command thought, opinion, or behavior”. It means you have the ability to tell people what to do. You give orders and they are followed. You tell your team the direction that you want things to go and it happens. If it doesn’t there are consequences. When you tell someone they are the boss this is what they usually picture.

Responsibility is “the quality of being responsible,” where responsible means “liable to be called on to answer”. Responsibility is being the one to discuss what happens with the people under your charge. You talk about successes and failures and ultimately serve as the face of the group. When your boss starts looking for someone to tell them what’s going on you’re the one that needs to provide the answers.

As mentioned, many people think leadership and management is about the first thing and less about the second. I’d argue that you’ve worked for them before and it hasn’t been enjoyable. Having orders barked at you or threats of disciplinary action if goals aren’t accomplished are hallmarks of someone that’s focused on authority or on a “power trip”. It’s usually a very unpleasant experience, especially if that person later gets more power or is promoted to a higher level.

Responsibility is what the rest of the population thinks of when you discuss leadership. It’s being accountable for the people you lead. It’s more about celebrating their successes with others when appropriate as well as explaining what happened when there wasn’t the success you’d hoped for. These leaders are often much easier to work for because they empower those they work with and shield you from bad managers and bosses that only want someone other than themselves to accept the responsibility for failure.

A good leader will exhibit qualities of both of these traits to a degree. However, I would argue that the biggest difference between good leaders and bad bosses is how they handle responsibility. Responsibility is the more important of the two qualities to have. That’s because you can delegate authority but you can never delegate responsibility.

Read that last part again. Slowly.

In a formal leadership role, such as a military command, you delegate authority to accomplish things. Officers delegate authority to non-commissioned officers who then may delegate to a lower level like a team leader. At no point is there only one single person issuing all the orders from on high with the expectation that they will be followed by everyone beneath them. Leaders like CEOs may have a vision for how things need to be done but they leave the authority to accomplish those goals up to the leaders closer to the task at hand.

Delegating authority ensures that things are accomplished with efficiency. Could you imagine how difficult it would be for a military command to rely on a single general to give them every single order that was necessary for them to function? That might have worked in antiquity with smaller armies but in a modern force you have to delegate authority to junior officers or enlisted soldiers in order to keep things running smoothly. You also have to trust that the people you’ve placed in that role will get things accomplished. It doesn’t always work out the way you’d like but that’s part of the role of developing good leaders.

Responsibility Bites

What about the other, more important thing? Responsibility can’t be delegated. If the captain of a ship puts a junior officer in charge and something happens? In the example of the USS Fitzgerald colliding with a merchant ship the sailors in charge of the bridge were relieved of command and the ship’s commander faced disciplinary action. Someone had to answer for the collision. The person that caused it faced disciplinary actions but so too did the people in charge. In a different situation removed from the military it might have been easy for the commander to claim they weren’t on duty or they had told someone else to do it but the legal tradition of the US Navy is that the commander of the ship is always responsible for the actions of their crew. They must answer for problems, including colliding with another ship.

Responsibility can’t be delegated. If you are the leader for your team you must answer for their actions. If their actions create success that’s an easy conversation to have. If their actions lead to problems or liability then you also must answer for those as well. You can’t just take credit for the good things. You must also provide the interface when your manager or boss needs to discuss the bad things too. Responsibility for your team fosters the connections that reinforce teamwork. It’s easy to claim it wasn’t your fault that something happened if you weren’t around for it. The best leaders accept that whatever happened must have been because of a lack of training or some other deficiency and answer for it while working to correct the issue. They take the heat to allow for time to fix the issue, either through training or through personnel replacement.

If you’re now staring to see the value of working for an organization where leaders delegate authority to a good team and accept responsibility for their actions, both good and bad, then you know how valuable that can be. Morale will go up, productivity will increase, and most importantly you’ll be training the next generation of leaders in that mold so they become effective.

However, if you’re wondering what it feels like to work in an environment that is the exact opposite, imagine a role where your boss tells you that you must be the one to answer for your actions and that they aren’t responsible for what happens. When someone complains your boss is the first to point out that it’s not their fault. When there is success they claim it was all due to their leadership. When you complain that the rules don’t allow you to be effective your boss tells you that’s just the way it is and you can’t change anything so you need to get used to it.

If that sounds familiar you’re not alone. If that sounds like the role you’re currently in perhaps it’s time to work for a better leader.


Tom’s Take

Good leaders know when to help and when to get out of the way. They don’t take charge. They take responsibility. They highlight success as a team effort and answer when success isn’t there so it can be fixed. It doesn’t have to be as strict as a military command. By delegating authority and being responsible you can set an example for everyone you work with and everyone you work for. If the culture of your organization is the exact opposite it’s time to go somewhere you are valued because bad leaders will soon have no one to take responsibility for them and they won’t be able to boss anyone around they way they really want to.

Why 2023 is the Year of Wi-Fi 6E

If you’re like me, you chuckle every time someone tells you that next year is the year of whatever technology is going to be hot. Don’t believe me? Which year was the Year of VDI again? I know that writing the title of this post probably made you shake your head in amusement but I truly believe that we’ve hit the point of adoption of Wi-Fi 6E next year.

Device Support Blooms

There are rumors that the new iPhone 14 will adopt Wi-Fi 6E. There were the same rumors when the iPhone 13 was coming out and the iPhone rumor mill is always a mixed bag but I think we’re on track this time. Part of the reason for that is the advancements made in Wi-Fi 6 Release 2. The power management features for 6ER2 are something that should appeal to mobile device users, even if the name is confusing as can be.

Mobile phones don’t make a market. If they were the only driver for wireless adoption the Samsung handsets would have everyone on 6E by now. Instead, it’s the ecosystem. Apple putting a 6E radio in the iPhone wouldn’t be enough to tip the scales. It would take a concerted effort of adoption across the board, right? Well, what else does Apple have on deck that can drive the market?

The first thing is the rumored M2 iPad Pro. It’s expected to debut in October 2022 and feature upgrades aside from the CPU like wireless charging. One of the biggest features would be the inclusion of a Wi-Fi 6E radio as well to match the new iPhone. That would mean both of Apple’s mobile devices could enjoy the faster and less congested bandwidth of 6 GHz. The iPad would also be easier to build a new chip around compared to the relatively cramped space inside the iPhone. Give the professional nature of the iPad Pro one might expect an enterprise-grade feature like 6E support to help move some units.

The second thing is the looming M2 MacBook Pro. Note for this specific example I’m talking about the 14” and 16” models that would features the Pro and Max chips, not the 13” model running a base M2. Apple packed the M1 Pro and M1 Max models with new features last year, including more ports and a snazzy case redesign. What would drive people to adopt the new model so soon? How about faster connectivity? Given that people are already complaining that the M1 Pro has slow Wi-Fi Apple could really silence their armchair critics with a Wi-Fi 6E radio.

You may notice that I’m relying heavily on Apple here as my reasoning behind the projected growth of 6E in 2023. It’s not because I’m a fanboy. It’s because Apple is one of the only companies that controls their own ecosystem to the point of being able to add support for a technology across the board and drive adoption among their user base. Sure, we’ve had 6E radios from Samsung and Dell and many others for the past year or so. Did they drive the sales of 6E radios in the enterprise? Or even in home routers? I’d argue they haven’t. But Apple isn’t the only reason why.

Oldie But Goodie

The last reason that 2023 is going to be the year of Wi-Fi 6E is because of timing. Specifically I’m talking about the timing of a refresh cycle in the enterprise. The first Wi-Fi 6 APs started coming into the market in 2019. Early adopters jumped at the chance to have more bandwidth across the board. But those APs are getting old by the standards of tech. They may still pass traffic but users that are going back to the office are going to want more than standard connectivity. Especially if those users splurged for a new iPhone or iPad for Christmas or are looking for a new work laptop of the Macintosh variety.

Enterprises may not have been packed with users for the past couple of years but that doesn’t mean the tech stood still. Faster and better is always the mantra of the cutting edge company. The revisions in the new standards would make life easier for those trying to deploy new IoT sensors or deal with with congested buildings. If enterprise vendors adopt these new APs in the early part of the year it could even function as an incentive to get people back in the office instead of the slow insecure coffee shop around the corner.

One other little quirky thing comes from an report that Intel is looking to adopt Wi-Fi 7. It may just be the cynic in me talking but as soon as we start talking about a new technology on the horizon people start assuming that the “current” cutting edge tech is ready for adoption. It’s the same as people that caution you not to install a new operating system until after the first patch or service release. Considering that Wi-Fi 6 Release 2 is effectively Wi-Fi 6E Service Pack 1 I think the cynics in the audience are going to think that it’s time to adopt Wi-Fi 6E since it’s ready for action.


Tom’s Take

Technology for the sake of tech is always going to fail. You need drivers for adoption and usage. If cool tech won the day we’d be watching Betamax movies or HD-DVD instead of streaming on Netflix. Instead, the real winners are the technologies that get used. So far that hasn’t been Wi-Fi 6E for a variety of reasons. However, with the projections of releases coming soon from Apple I think we’re going to see a massive wave of adoption of Wi-Fi 6E in 2023. And if you’re reading this in late 2023 or beyond and it didn’t happen, just mentally change the title to whatever next year is and that will be the truth.

Enforcing SLAs with Real Data

I’m sure by now you’ve probably seen tons of articles telling you about how important it is to travel with location devices in your luggage. The most common one I’ve seen is the Apple AirTag. The logic goes that if you have one in your checked suitcase that you’ll know if there are any issues with your luggage getting lost right away because you’ll be notified as soon as you’re separated from it. The advice is sound if you’re someone that checks your bag frequently or has it lost on a regular basis.

The idea behind using technology to enforce an agreement is a great one. We make these agreements all the time, especially in networking. These service level agreements (SLAs) are the way we know we’re getting what we pay for. Take a leased line, for example. You typically pay for a certain speed and a certain amount of availability. The faster the link or the more available it is the more it costs. Any good consumer is going to want to be sure they’re paying for the right service. How can you verify you’re getting what you’re paying for?

For a long time this was very hard to do. Sure, you could run constant speed tests to check the bandwidth. However, the reliability was the part that was typically the more expensive thing to add to the service. Making a circuit more reliable means adding more resources on the provider side to ensure it stays up. Allocating those resources means someone needs to pay for them and that is usually on the customer side.

It’s easy to tell when a link goes down during working hours. You can see that you’re not getting traffic out of your network. But what about those times when the circuit isn’t being as heavily utilized? What about the middle of the night or holidays? How can you ensure that you’re getting the uptime you pay for?

Trust and Verify

This is actually one of the groundbreaking areas that SD-WAN pioneered for networking teams when it came out years ago. Because you have a more modern way of maintaining the network as well as a way to route application traffic based on more than source and destination address you can build in some fancy logic to determine the reliability of your circuits too.

One of the biggest reasons in the past to use a leased line over a broadband circuit was that reliability factor. You need MPLS because it’s more stable than a cable modem. You get guaranteed bandwidth. It’s way more reliable. Those are the claims that you might hear when you talk to the salesperson at your ISP. But do they really hold water? The issue for years is that you had no way of knowing because your analytics capabilities were rudimentary in most cases. You could monitor the link interface but that was usually from the central office and not the remote branch. And depending on the polling interval you could miss downtime events.

SD-WAN changed that because now you could put an intelligent device on the edge that constantly monitored the link for throughput and reliability. The first thought was to do this for the broadband links to see how congested and unreliable they could be do know when to switch traffic to the more stable link. Over time admins started putting those same monitors on the MPLS and leased line circuits as well. They found that while broadband, such as cable and DSL, was typically more reliable than the sales people would have you believe it was the relatively unreliable leased circuits that surprised everyone.

Much like the above example of lost luggage you previously had to take the airlines at their word when they said something was lost. No details were available. Did your bag ever leave the airport when you flew out? Did it make it to the right location but get stuck somewhere? Did your bag get on the wrong plane and end up in Cleveland instead of Las Vegas? Because the airline reporting systems were so opaque you had no idea. Now, with the advent of Tile and AirTags and many others, you can see exactly where it is at all times.

The same thing happened with SD-WAN analytics. Now, armed with proof that links weren’t as reliable as the SLA, admins could choose to do one of two things. They could either force the provider to honor the agreement and provide better service to meet the agreement. Or the company could reduce their contract to the level of service they are actually getting instead of the one that is promised by not delivered. Having the data and a way to prove the reliability helped with the negotiations.

Once the providers knew that organizations had the ability to verify things they also had to up their game by including ways to monitor their own performance to ensure that they were meeting their own metrics. Overall the situation led to better results because having technology to enforce agreements makes all sides aware of what’s going on.


Tom’s Take

I’ve only had a couple of bags misplaced in my travel time so I haven’t yet had the need to go down the AirTag route. I’ve done it on occasion for international travel because knowing when my bag is about to come out on the carousel helps me figure out how much time it’s going to take me to get through customs. The idea of using technology to make things more transparent is important to me on a bigger scale though. If you can’t verify the promises you make then you shouldn’t make them. Having the AirTag or SD-WAN software to ensure we’re getting what we pay for are just parts of a bigger opportunity to provide better experiences.

All Problems Are Hardware Problems

When I was a lad in high school I worked for Walmart. I learned quite a bit about retail at my early age but one of the fascinating things I used in the late 1990s was a wireless inventory unit, colloquially known as a Telxon. I was amazed by the ability to get inventory numbers on a device without a cable. Since this was prior to the adoption of IEEE 802.11 it was a proprietary device that only worked with that system.

Flash forward to the 2020s. I went to Walmart the other day to look for an item and I couldn’t find it. I asked one of the associates if it was in stock. They said they could check and pulled out their phone. To my surprise they were able to launch an app and see that it was in stock in the back. As I waited for them to return with the item I thought about how 25 years of progress had changed that hardware solution into something software focused.

Hardware Genesis

All problems start as hardware problems. If there’s a solution to an issue you’re going to build something first. Need to get somewhere fast? Trains or cars. Need to get there faster? Planes. Communcations? Phones or the Internet. All problems start out by inventing a thing that does something.

Technology is all about overcoming challenges with novel solutions. Sometimes those leaps are major, like radio. Other times the tool is built on that technology, like wireless. However they are built they almost always take physical form first. The reasoning is simple in my view. You have to have the capability to build something before it can be optimized.

If you’re sitting there saying to yourself that a lot of our solutions today are software-focused you would be correct. However, you’re also making some assumptions about hardware ubiquity already. Sure, the smartphone is a marvel of software simplicity that provides many technological solutions. It’s also a platform that relies on wireless radio communications, Internet, and cloud computing. If you told someone in 2005 that cellular phones would be primarily used as compute devices they would have laughed at you. Because the hardware at the time was focused on audio communications and only starting to be used for other things, like texting or PDA functions.

Hardware exists first to solve the issue at hand. Once we’ve built something that can accomplish the goal we can then optimize it and make it more common. Servers seem like a mainstay of our tech world today but client/server architecture wasn’t always the king of the hill. Cloud computing seems like an obvious solution today but AWS and GCP haven’t always existed. Servers needed to become commonplace before the idea to cluster them together and offer their capacity as a rentable service emerged.

Software to the Rescue

Software doesn’t like unpredictability. Remember the Telxon example above? Those devices ran proprietary software that interfaced with a single server. There was no variation. If you wanted to use the inventory software on a different device you were out of luck. There was zero flexibility. It was a tool that was designed for a purpose. So many of the things in our lives are built the same way. Just look at your home phone, for example. That is, if you still have one. It’s a simple device that doesn’t even run software as we know it. Just a collection of electrical switches and transistors that accomplish a goal.

However, we have abstracted the interface of a telephone into a device that provides flexibility. Any smart phone you see is a computer running software with a familiar interface to make a phone call. There’s no specialized hardware involved. Just an interface to a system that performs the same functions that a traditional phone would. There are no wires. No switches like an old telecom engineer would recognize. Just a software platform that sends voice over the Internet to a receiving station.

Software becomes an option when we’ve built a hardware environment that is sufficiently predictable as to allow the functions to be abstracted. The Walmart Telxon can function as an app on a smartphone now because we can write an app to perform those same functions. We’ve also created interfaces into inventory systems that can be called by software apps and everyone that works for Walmart either carries a phone or has access to a device that can run the software.


Tom’s Take

Programs and platforms provide us with the flexibility to do things any time we want. But they only have that capability because of the infrastructure we’ve built out. We have to build the hardware before we can abstract the functions into software. The most complicated unsolved problem you can think of today will eventually be solved by a piece of kit that will eventually become a commodity and replicated as a series of functions that run on everything years later. That’s the way that things have always been and how they will always be.

Friday Mobility Field Day Thoughts

I’m finishing up Mobility Field Day 7 this week and there’s been some exciting discussion here around a lot of technology. I think my favorite, and something I’m going to talk about more, is the continuing battle between 5G and Wi-Fi. However, there’s a lot going on that I figured I’d bring up to whet your appetite for the videos.

  • What is mission critical? When you think about all the devices that are in your organization that absolutely must work every time what does that look like? And what are you prepared to do to make them work every time? If it’s a safety switch or some other kind of thing that prevents loss of life are you prepared to spend huge amounts of money to make it never fail?
  • Operations teams don’t need easier systems. They need systems that remove complexity. The difference in those two things is subtle but important. Easier means that things are simplified to the point of almost being unusable. Think Apple Airport or even some Meraki devices. Whereas reduced complexity means that you’ve made the up front configuration easy but enabled the ability to configure other features in different places. Maybe that’s by giving your engineers the ability to log in to an Advanced dashboard or something like that.
  • When you’re trying to figure out where your audience is on a subject, always aim slightly above their technical level. If all goes well you will pull them up to where you want them and they’ll appreciate the opportunity to stretch their thinking to meet you there. If not you’ll provide them lots of great material to learn about when they get there later.

Tom’s Take

Technology changes quickly but the way we teach it doesn’t need to if we do it right the first time. By taking the time to aim high and educate instead of retelling something we’ve already said a few times we create content that endures.