Time to Talk

It’s a holiday week here in the US so most people are working lighter days or just taking the whole week off. They’re looking forward to spending time with family and friends. Perhaps they’re already plotting their best strategy for shopping during Black Friday and snagging a new TV or watch. Whatever the case may be there’s lots things going on all over.

One thing that I feel needs to happen is conversation. Not just the kind of idle conversation that we make when we don’t know what to talk about. I also don’t mean the kinds of deep conversations that we need to prepare ourselves to have. I’m talking about the ones where we learn. The ones we have with friends and family where we pick up tidbits of stories and preserve them for the future.

It sounds rather morbid but these conversations aren’t going to be available forever. Our older loved ones are getting older every year. Time marches on and we never know when that time I going to run out. I have several friends that have lost loved ones this year and still others that have realized the time is growing shorter. Mortality is something that reminds us how important those experiences can be.

This year, talk to your friends. Listen to the stories of your family. Make the time to really hear them. That might mean turning off the football game or skipping that post-turkey nap. But trust me when I say that you’ll appreciate that time more when you realize you won’t have it any more.

Play To Your Team Strengths

This past weekend I went to a training course for an event that I’m participating in next year. One of the quotes that came up during the course was about picking the team that will help you during the event. The quote sounded something like this:

Get the right people on the right bus in the right seats and figure out where you want to go.

Sounds simple, right? Right people, right bus, right seats. Not everyone is going to be a good fit for your team and even if they are they may not be in the right position to do their best work. But how do you know what they’re good at?

Not-So-Well Rounded

Last night, I listened to this excellent Art of Network Engineering episode. The guest was a friend of mine in the industry, Mike Bushong (@MBushong). He’s a very talented person and he knows how to lead people. He’s one of the people that would love to work for given the opportunity. He’s also very astute and he has learned a lot of lessons about enabling people on a team.

One of the things he discussed in the episode was about people’s strengths. Not just things you’re good at but qualities you would say are your strongest assets. Aspects of you that you say are core to what you do. Maybe you’re good at writing emails but is that a strength? Or is concise communication your real strength? Are you good at seeing patterns? Or are you analytical? Every task you excel at isn’t a strength until you can see the underlying pieces that you really are good at doing.

One of the things that Mike brought up in the episode that really resonated with me was the idea that our performance evaluation system is really built around pointing out weaknesses and encouraging (or forcing) people to work on them. I can understand having people work on some core skills that are necessary in a business environment, such as time management or communications. You have to be good at those in order to succeed in your career.

However, having people pick up new skills or focus on aspects of what they do almost in a vacuum doesn’t really help as much as managers might think it does. Could you imagine telling someone like Steph Curry he needs to work on his dunking skills? Or perhaps telling someone like Steven Tyler that, while his singing is great, he really should spend more time playing the drums to be a more well-rounded band member?

The idea of telling professionals to concentrate on something other than their strength is ludicrous. Albert Pujols isn’t going to be the base stealing king for a baseball team. His strength is hitting home runs. Why try to make him into someone that runs instead of someone that hits? Yet when you ask a manager about reviewing someone’s performance they’ll tell you they need that team member to be well-rounded or they need to pick up this other skill that isn’t necessarily their strength but is needed.

I’m a decent writer. I’ve found over the past fifteen years that one of my strengths is in written communications. I can distill information and convey it to others through written words. I’ve managed to adapt that skill into being good at public speaking and giving presentations. What I would not be good at doing is being a full-time project manager with responsibilities for managing timelines for multiple teams. It’s because I realize that I need to be good enough at time management to get my work done but it is by no means a strength for me. For my manager to tell me to spend more time focusing on my weakness and less on my strength is doing me and the team a disservice.

Strong Bus

When you’re putting together your team, you need the right people on the bus. That part is pretty easy, right? But if you don’t know what they’re good at you don’t know if they’re the right people for your bus. If you have a team of people that are very detail oriented do you need to add another detail oriented person to do design work? Or would it be better to have someone with a strength in creativity? If you already have a group introverts that work best in isolation do you want to add another? Or do you need an outgoing personality in a role that interfaces with other teams or customers?

When you’re putting the right people in the right seats on the bus, make sure you’re looking at playing to their strengths instead of forcing them into a role to help them grow. Most people love a good challenge or like to step outside of their comfort zone now and then. However not everyone is able to take on a role that is something they consider a weakness. Instead of pushing people to the point of being uncomfortable and making them the wrong person in the wrong seat we need to help them succeed. An example might be having someone with limited experience creating marketing material perhaps working to come up with written pieces of the deliverables instead of the entire design. Or maybe having them do a single flyer or handout instead of the entire job. Giving them the opportunity to stretch a little and succeed is a much better way to help them in their career instead of throwing them in the deep end of the pool and hoping they can swim.


Tom’s Take

You should listen to the entire podcast episode with Mike Bushong. It encapsulates why he’s such a well-liked and effective leader. He knows his limitations and he works to overcome them. He identifies the right people for his team and he puts them in the places they can make the most impact to get things accomplished. He knows that a good leader makes the people around them better by enabling them to succeed. Some of these lessons are things that I’ve learned over the past few years through Wood Badge and the way he’s phrased them helped me internalize them a bit better. Play to people’s strengths and you’ll be happily surprised at how far they can go with you.

Continuity is Not Recovery

It was a long weekend for me but it wasn’t quite as long as it could have been. The school district my son attends is in the middle of a ransomware attack. I got an email from them on Friday afternoon telling us to make sure that any district-owned assets are powered off until further notice to keep our home networks from being compromised. That’s pretty sound advice so we did it immediately.

I know that the folks working on the problem spent the whole weekend trying to clean it up and make sure there isn’t any chance of getting reinfected. However, I also wondered how that would impact school this week. The growing amount of coursework that happens online or is delivered via computer is large enough that going from that to a full stop of no devices is probably jarring. That got me to thinking once more about the difference between continuity and recovery

Keeping The Lights On

We talk about disaster recovery a lot. Backups of any kind are designed to get back what was lost. Whether it’s a natural disaster or a security incident you want to be able to recover things back to the way they were before the disaster. We talk about making sure the data is protected and secured, whether from attackers or floods or accidental deletion. It’s a sound strategy but I feel it’s a missing a key component.

Aside from getting your data back, which is called the recovery point objective (RPO), you also need to consider how long it’s going to take to get you there. That’s called the recovery time objective (RTO). RTO tells you how long it will be until you can get your stuff back. For a few files the RTO could be minutes. For an entire data center it could be weeks. The RTO can even change based on the nature of the disaster. If you lose power to the building due to a natural disaster you may not even be able to start recovery for days which will extend the RTO due to circumstances outside your control.

For a business or organization looking to stay up and running during a disaster, RTO is critical but so too is the need for business continuity. How critical is it? The category was renamed to “Disaster Recovery and Business Continuity” many years ago. It’s not enough to get your data back. You have to stay up and running as much as possible during the process. You’ve probably experienced this if you’ve ever been to a store that didn’t have working registers or the ability to process credit cards. How can you pay for something if you can’t ring it up or process a payment option?

Business continuity isn’t about the data. It’s about keeping the lights on while you recover it. In the case of my son’s school they’re going to teach the old fashioned way. Lectures and paper are going to replace videos and online quizzes. Teachers are thankfully very skilled in this manner. They’ve spent hundreds if not thousands of hours in a classroom instructing with a variety of techniques. Are your employees equally as skilled when everything goes down? Could they get the job done if your Exchange Server goes down or they’re unable to log into Salesforce?

Back to Good, Eventually

In order to make sure you have a business left to recover you need to have some sort of a continuity plan. Especially in a world where cyberattacks are common you need to know what you have to do to keep things going while you work on fixing the damage. Most bad actors are counting on you not being able to conduct business as a driver to pay the ransom. If you’re losing thousands of dollars per minute you’re more likely to cave in and pay than try to spend days or weeks recovering.

Your continuity plan needs to exist separately from your backup RTO objectives. It may sound pessimistic but you need to have a plan for what happens if the RTO is met but also one for what happens if you miss your RTO. You don’t want to count on a quick return to normal operations as your continuity plan only to find out you’re not going to get there.

The other important thing to keep in mind is that continuity plans need to be functional, not perfect. You use the systems you use for a reason. Credit card machines make processing payments quick and easy. If they’re down you’re not going to have the same functionality. Yes, using the old manual process with paper slips and carbon copies is a pain and takes time. It’s also the only way you’re going to be able to take those payments when you can’t use the computer.

You also need to plan around how to handle your continuity plan. If you’re suddenly using more paper, such as invoices or credit card slips, where do you store those? How will you process them once the systems come back online? Will you need to destroy anything after it’s entered? Does that need to happen in a special way? All of these questions should be asked now so there is time to debate them instead of waiting until you’re in the middle of a disaster to solve them.


Tom’s Take

Disasters are never fun and we never really want them to happen. However we need to make sure we’re ready when they do. You need to have a plan for how to get everything back as well as how to keep doing everything you can until that happens. You may not be able to do 100% of the things you could before but if you don’t try to at least do some of them you’re going to lose a lot more in the long run. Have a plan and make sure everyone knows what to do when disaster strikes. Don’t count on getting everything back as the only way to recovery.

Wi-Fi 6E Growing Pains For Apple

You may have seen that the new iPad Pro has Wi-Fi 6E support. That caused a lot of my wireless friends to jump out and order one, as I expected. As I previously mentioned, 2023 is going to be a big year for Wi-Fi 6E. I was wrong about the 6E radio on the new iPhone but given the direction that Apple is going with the iPad Pro and probably the MacBook as well we’re in for a lot of fun. Why? Because Apple is changing their stance on how to configure 6GHz networks.

An SSID By Any Other Name

If you’ve ever set up wireless networks before you know there are some different suggestions about how to configure the SSIDs with multiple bands. One school of thought says that you need to combine both 2.4GHz and 5GHz in the same SSID and let the device figure out which one is the best to use. This is the way that I have mine set up at home.

However, if you do a quick Google search you’ll find a lot of other wisdom that suggests creating two different SSIDs that only work on a single band. The thought process here is that the device can’t distinguish between the different bands once it makes a decision so it will be stuck on one or the other. While that doesn’t sound like a bad idea it does have issues in practice. What if your device chooses 2.4GHz when you didn’t want it to? What if your device is stuck on 5GHz at the limit of the noise floor in your office and you want it to swap to the other band for better throughput instead of adding another AP?

There are several reasons to have more control over how the frequency band is chosen. Sadly, Apple has never allowed the device to choose the band when joining a network. The only way to influence that selection has been to create different networks, which leads to management issues and other challenges for devices that are unable to join one network or another. The management issues made the planning process rather challenging.

Now, with the advent of a device that has a Wi-Fi 6E radio in the 6GHz range, Apple has changed their thinking about how a network should operate. In a new support post, Apple now clarifies that the SSID names should not be different for the three different bands. There’s no other mention of what happens at a device level as far as band selection.

In a different tech support article, Apple describes what happens if you don’t give them the same name. If you join a 6GHz-only network on the new iPad Pro, the device will detect there is no corresponding 5GHz network and search for one from the same AP and let you join it as well. The article for this even mentions the ominous “limited compatibility”, even if the dialog box doesn’t. If you choose to join this split SSID setup there is another confirmation box that encourages you to go tweak your SSID settings to make the name the same on both networks. I’m not sure if that same prompt comes up for 2.4GHz networks too. Maybe I can borrow someone’s iPad to test it.

Disabling New Tech

Even though Apple has never allowed users to select the band that they want to use on an SSID there is a new feature for 6GHz that gives you the opportunity to work around any issues you have with this new band. In the settings for the SSID there is a toggle for “Wi-Fi 6E Mode” that allows you to disable 6GHz on that SSID until enabled again. This way you can use the recommended settings for the SSID per Apple but still disable the pieces that might be broken.

Interestingly, this toggle only appears for 6E networks according to the support article. There’s still no way to toggle between 2.4GHz and 5GHz. However, adding this support to the network settings should be easy to carry down into the other bands. Whether or not Apple does it is a much different matter. Also, the setting isn’t currently in MacOS Ventura. That could be because there isn’t a 6E radio available in a Mac yet and the setting might not show up until there’s a supported radio. Time will tell when Apple releases a MacBook with a built-in Wi-Fi 6E radio.


Tom’s Take

After months of professionals saying that Apple needs to release support for Wi-FI 6E it’s finally here. It also brings new capabilities from the software side to control how the 6E radios are used. Is it completely baked and foolproof? Of course not. Getting the radios into the iPad was the first step. By introducing them now with software for troubleshooting and configurations and following it up with a likely 6GHz MacBook and iMac lineup soon there will be plenty of time to work out the issues by the time the iPhone 15 gets support for Wi-Fi 6E. Apple is clearly defining their expectations for how an SSID should work so you have plenty of time to work through things or change your design documents before the explosion of Wi-Fi 6E clients arrives en masse in 2023.

Cleaning Out The Cruft

I spent the weekend doing something I really should have done a long time ago. I went through my piles of technology that I was going to get around to using one day and finally got rid of anything I didn’t recognize. Old access points, old networking gear, and even older widgets that went to devices that I don’t even remember owning.

Do you have one of these piles? Boxes? Corners of your office or cave? The odds are good there’s a pile of stuff that you keep thinking you’re eventually going to get around to doing something with some day. Except some day hasn’t come yet. So maybe it’s time to get rid of that pile. Trust me you’re going to feel better for getting rid of that stuff.

What to do with it? It needs to be properly recycled so don’t just toss it in the trash can. Anything with electric circuits needs to be properly disposed of so look for an electronics recycling facility. Yes, there are stories that electronics recycling isn’t all it’s cracked up to be but it’s better than polluting with e-waste everywhere.

Consider donating the devices to a trade school or other maker space. Maybe they won’t work properly as intended but giving students the chance to take them apart is a much better option than just junking it all. Maybe you’ll inspire the next group of scientists and inventors because your old 802.11g access point fascinated them when they pulled it apart.

No matter whether you recycle or donate you should go through it all. Consolidate and be honest with yourself. If you don’t recognize it or haven’t used it in the last few months you’re not going to miss it.

Hedgehog – The Network OS Distro?

You’ve probably seen by now that there’s a new entrant into the market for network operating systems. Hedgehog came out of stealth mode this week to fanfare from the networking community. If you read through the website you might question why I labeled them as a network operating system. While they aren’t technically the OS I think it’s more important to look at them as an OS distribution.

Cacophony of Choice

Hedgehog starts from a very simple premise. Cloud networking is where we’re all headed. Whether or not you’re running entirely on-premises, fully in the public cloud, or in some kind of super-multi-hybrid cloud offering you’re all chasing the same thing. You want a stable system that acts as a force multiplier for your operations teams to reduce deployment times for users to get their builds done. It’s been said before but the idea of cloud is to get IT out of the way of the business.

Streamlining processes means automating a lot of the things that were formerly done by people. That means building repeatable and consistent tools to make that happen. If anyone has ever worked on AWS or Google Cloud you have lots of access to that tooling. Perhaps it’s not as full-featured as rolling your own toolset but that’s the tradeoff for running in someone else’s cloud. Notice that I left Microsoft Azure off that list.

Azure’s networking stack has been built on SONiC, a LInux-based NOS that has been built to scale in the cloud and solve the challenges that Microsoft has faced in their hyperscale data centers. They’ve poured resources into making SONiC exactly what they needed. However, one of the challenges that is faced when that happens is some of those things don’t scale down to the enterprise. I’m not saying you shouldn’t use SONiC. I’m saying that it’s not easy to adapt SONiC to what you want to do if you’re not Microsoft.

Speedy Adoption

In a way, it’s the same problem that Linux faced 25 years ago. If you really wanted to run it on a system you could download the source code and compile it on your system to get the kernel running. However a kernel doesn’t do much without software running on top of it. How could I write blog posts or check the time or get on the Internet without other applications? That need for additional resources to make the process of using Linux easier and more complete is where we got the rise of the Linux distribution, often shortened to distro.

Distros made adopting Linux easier. You didn’t have to go out and find sources for programs to run them on your system after compiling the kernel. You could just install everything like a big package and get going. It also meant that you could swap out programs and tools much easier than other operating systems. Ever tried to get rid of Notepad on Windows? It’s practically a system tool. On the other hand I think most Linux users can tell me their five favorite text editors off the top of their head. The system is very extensible.

Hedgehog acts like the distro of yore for SONiC. It makes the process of obtaining the OS much easier than it would be otherwise and includes a toolset that amplifies your networking experience. The power of cloud networking comes from optimization and orchestration. Hedgehog gives you that capability. It allows you to run the same kinds of tooling that you would use in the cloud on your enterprise data center networking devices.

If you’re starting to standardize on Kubernetes to make your applications more portable to the cloud then Hedgehog and SONiC can help you. If you’re looking to build more edge computing functionality into the stack Hedgehog has you covered. The Hedgehog team is building the orchestration capabilities that are needed in the enterprise right now to help you leverage SONiC. Because that tooling doesn’t exist outside of Microsoft right now you can believe that the Hedgehog team is addressing the needs of enterprise operations teams. They are the drivers for SONiC adoption. Making sure they can take care of daily tasks is paramount.

The distro launched Linux into the enterprise. Clunky DIY tooling can only scale so far. If you want to be serious about adopting cloud-first mentality in your organization you need to make sure you’re using proven tools that scale and don’t fall apart every time you try to make a minor change. Your data center isn’t Facebook or Google or Azure. However the lessons learned there and the way that we apply them at the enterprise level will go a long way to providing the advantages of the cloud for every day use. Thanks to companies like Hedgehog that are concentrating on the way to bring that market we have a chance to see it sooner than we’d hoped.

To learn more about Hedgehog and how they make SONiC easier for the enterprise, make sure to check out their website at https://githedgehog.com/

Why Do We Accept Bad Wireless Clients?

We recorded a fun roundtable discussion last week during Mobility Field Day that talked about the challenges that wireless architects face in their daily lives. It’s about an hour but it’s packed with great discussions about hard things we deal with:

One of the surprises for me is that all the conversations came back to how terrible wireless clients can be. The discussion kept coming back to how hard it is to find quality clients and how we adjust our expectations for the bad ones.

Driven to Madness

Did you know that 70% of Windows crashes are caused by third-party drivers? That’s Microsoft’s own research saying it. That doesn’t mean that Windows is any better or more stable with their OS design compared to Linux or MacOS. However, I’ve fiddled with drivers on Linux and I can tell you how horrible that experience can be1. Windows is quite tolerant of hardware that wouldn’t work anywhere else. As long as the manufacturer provides a driver you’re going to get something that works most of the time.

Apply that logic to a wireless networking card. You can buy just about anything and install it on your system and it will mostly work. Even with reputable companies like Intel you have challenges though. I have heard stories of driver updates working in one release and then breaking horribly in another. I’ve had to do the dance of installing beta software to make a function work at the expense of stability of the networking stack. Anyone that has ever sent out an email cautioning users to not update any drivers on their system knows the pain that can be caused by bad drivers corrupting clients.

That’s just the software we can control. What if it’s an OS we can’t do anything about? More and more users are turning to phones and tablets for their workhorse devices. Just a causal glance at Youtube will reveal a cornucopia of using a tablet as a daily driver machine. Those devices aren’t immune to driver challenges. They just come in a hidden package during system updates. Maybe the developers decided to roll out a new feature. Maybe they wanted to test a new power management algorithm. Maybe they’re just chaotic neutral and wanted to disrupt the world. Whatever the reason you’re stuck with the results. If you can’t test it fast enough you may find your users updated their devices chasing a feature. Most companies stop signing the code for the older version shortly after issuing an update so downgrading is impossible. Then what? You have a shiny brick? Maybe you have to create a special network that disables features for them? There are no solid answers.

Pushing Back

My comment in the roundtable boils down to something simple: Why do we allow this to happen? Why are we letting client manufacturers do this? The answer is probably more elegant than you realize. We do it because users expect every device to work. Just like the Windows driver issues you wouldn’t plug something into a computer and then expect it to not work, right? Wireless is no different to the user. They want to walk in somewhere and connect. Whether it’s a coffee shop or their home office or the corporate network it needs to be seamless and friction-free.

Would you expect the same of an Ethernet cable? or a PATA hard drive? Would you expect to be able to bring a phone from home and plug it into your corporate PBX? Of course not. Part of the issue is a lack of visible incompatibility. If you know the Ethernet cable won’t plug into a device you won’t try to connect it. If the cable for your disk drive isn’t compatible with your motherboard you get a different drive. With wireless we expect the nerds in the back to “make it work”. Wireless is one of the best protocols at making things work poorly just to say it is up and running. If you had an Ethernet network with 15% packet loss you’d claim it was broken. Yet Wi-Fi will connect and drop packets due to bad SNR and other factors because it’s designed to work under adverse conditions.

Why do we tolerate bad clients? Why don’t we push back against the vendors and tell them to do better? The standard argument is that we don’t control the client manufacturing process. How are we supposed to tell vendors to support a function if we can’t make our voices heard? While we may not be able to convince Intel or Apple or Samsung to build in support for specific protocols we can affect that change with consumption. If you work in an enterprise and you need support for something, say 802.11r, you can refuse to purchase a device until it’s supported.

But wait, you say, I don’t control that either. You may not control the devices but you control the network to which they attach. You can tell your users that the device isn’t supported. Just like a PATA hard disk or a floppy drive you can tell users that what they want to do won’t work and you need to do something different. If they want to use their personal iPad for work or their ancient laptop to connect they need to update it or use a different communications method. If your purchasing department wants to save $10 per laptop because they come with inferior wireless cards you can push back and tell them that the specs aren’t compatible with the network setup. Period, full stop, end of sentence.


Tom’s Take

The power to solve bad clients won’t come from companies that make money doing the least amount of work possible. It won’t come from companies that don’t provide feedback in the form of lost sales. It will come when someone puts their foot down and refuses to support any more bad client hardware and software. If the Wi-Fi Alliance won’t enforce good client connectivity it’s time we do it for them.

If you disagree I’d love to hear what you think. Is there a solution I’m not seeing? Or are we just doomed to live with bad client devices forever?


  1. If you say Winmodem around me I will scream. ↩︎

Monday Mobility Quick Thoughts

I’m getting ready for Mobility Field Day 8 later this week and there’s been a lot of effort making sure we’re ready to go. That means I’ve spent lots of time thinking about event planning instead of writing. So I wanted to share some quick thoughts with you ahead of this week as well as WLPC Europe next week.

  • I remain convinced than half of the objections that are raised by oversight organizations when it comes to adopting new technology come from the fact they got caught flat-footed and weren’t ready for it to be popular. Whether it’s the Wi-Fi 6E safety issue or the report earlier this year from the FAA about 5G and airports it just seems like organizations spend less time doing actual investigation and more time writing press releases about how they are ready to figure it all out yet.
  • I also remain cautiously optimistic that the new Apple devices rumored to be coming out later this year, namely the iPad Pro and MacBook Pro with M2 chips, will have Wi-Fi 6E support. Yes, the iPhone didn’t. It’s also a smaller device with less room to add new hardware. The iPad and MacBook have historically gotten new chips before the smaller mobile device does. If I’m wrong then I guess we’ll get to see if 6E is enough of a factor to get people to ditch their Apple device for a Google or Samsung one.
  • As we rely more and more on software to expand the capabilities of our hardware I think we’re going to see more and more companies working toward the model of hardware-as-a-service. As in you lease the equipment from them for a monthly payment and, in return, you get to have a base level of features that can be expanded in higher “tiers” of service. Expect some more on this idea in the near future with the launch of solutions like Nile.

Tom’s Take

Make sure you tune in for Mobility Field Day 8 and don’t forget to tell us what you think! Maybe by next year we’ll have lots of Wi-Fi 7 content to discuss.

Intelligence and Wisdom

I spent the last week at the Philmont Leadership Challenge in beautiful Cimarron, NM. I had the chance to learn a bit more about servant leadership and work on my outdoor skills a little. I also had some time to reflect on an interesting question posed to me by one of the members of my crew.

He asked me, “You seem wise. How did you get so wise?” This caught me flat-flooted for a moment because I’d never really considered myself to be a very wise person. Experienced perhaps but not wise like Yoda or Gandalf. So I answered him as I thought more about it.

Intelligence is knowing what to do. Wisdom is knowing what not to do.

The more I thought about that quote the more I realized the importance of the distinction.

Basic Botany

There’s another saying that people tweeted back at me when I shared the above quote. It’s used in the context of describing Intelligence and Wisdom for Dungeons and Dragons roleplaying:

Intelligence is knowing that a tomato is a fruit. Wisdom is not putting tomatoes in a fruit salad.

It’s silly and funny but it gets right to the point and is a different version of my other observation. Intelligence is all about the acquisition of knowledge. Think about your certification journey. You spend all your time learning the correct commands for displaying routing tables or how to debug a device and figure out what’s going on. You memorize arguments so you can pass the exam without the use of the question mark.

Intelligence is focused on making sure you have all the knowledge you can ever use. Whether it’s an arcane spell book or Routing TCP/IP Volume 1 you’re working with the kinds of information that you need to ingest in order to get things done. Think of it like a kind of race to amass a fortune in facts.

However, as pointed out above, intelligence is often lacking in the application of that knowledge. Assembling a storehouse full of facts doesn’t do much to help you when it comes to applying that knowledge to produce outcomes. You can be a very intelligent person and still not know what to do with it. You may have heard someone say that a person is “book smart” or is lacking is “common sense”. These are both ways to say that someone is intelligent by maybe not wise.

Applied Science

If intelligence is all about acquisition of knowledge then wisdom is focused on application. Just because you know what commands are used to debug a router doesn’t mean you need to use them all the time. There are apocryphal stories of freshly minted CCIEs walking in to the data center for an ISP and entering debug ip packet detail on the CLI only to watch the switch completely exhaust itself and crash in the middle of the day. The command was correct for what they wanted to accomplish. What was missing was the applied knowledge that a busy switch wouldn’t be able to handle the additional processing load of that much data being streamed to the console.

Wisdom isn’t gained from reading a book. It’s gained from applying knowledge to situations. No application of that knowledge is going to be perfect every time. You’re going to make mistakes. You’re going to do things that cause problems. You’re going to need to fix mistakes and learn as you go. Along the way you’re going to find a lot of things that don’t work for a given situation. That’s where wisdom is gained. You’re not failing. You’re learning what doesn’t work so you don’t apply it incorrectly.

A perfect example of this came just a couple of days ago. The power in my office was out which meant the Internet was down for everyone. A major crisis for sure! I knew I needed to figure out what was going on so I started the troubleshooting process. I knew how electricity worked and what needed to be checked. Along the way I kept working and trying to figure out where the problem was. The wisdom I gained along the way from working with series circuits and receptacles helped me narrow things down to one wall socket that had become worn out and needed to be replaced. More wisdom told me to make sure the power was turned off before I started working on the replacement.

I succeeded not because I knew what to do as much as knowing what not to do when applying the knowledge. I didn’t have to check plugs I knew weren’t working. I knew things could be on different circuits. I knew I didn’t have to mess with working sockets either. All the knowledge of resistance and current would only serve me correctly if I knew where to put it and how to work around the issues I saw in the application of that knowledge.

Not every piece of wisdom comes from unexpected outcomes. It’s often just as important to do something that works and see the result so you can remember it for the next time. The wisdom comes in knowing how to apply that knowledge and why it only works in certain situations. If you’ve every worked with someone that troubleshoots really complex problems with statements like “I tried this crazy thing once and it worked” you know exactly how this can be done.


Tom’s Take

Intelligence has always been my strong point. I read a lot and retain knowledge. I’m at home when I’m recalling trivia or absorbing new facts. However I’ve always worried that I wasn’t very wise. I make simple mistakes and often forget how to use the information I have on hand. However, when I shared the quote above I finally realized that all those mistakes were just me learning how to apply the knowledge I’d gained over time. Wisdom isn’t a passage in a book. It’s not a fact. It’s about knowing when to use it and when not to use it. It’s about learning in a different way that matters just as much as all the libraries in the world.

Redundancy Is Not Resiliency

Most people carry a spare tire in their car. It’s there in case you get a flat and need to change the tire before you can be on your way again. In my old VAR job I drove a lot away from home and to the middle of nowhere so I didn’t want to rely on roadside assistance. Instead I just grabbed the extra tire out of the back if I needed it and went on my way. However, the process wasn’t entirely hitless. Even the pit crew for a racing team needs time to change tires. I could probably get it done in 20 minutes with appropriate cursing but those were 20 minutes that I wasn’t doing anything else beyond fixing a tire.

Spare tires are redundant. You have an extra thing to replace something that isn’t working. IT operations teams are familiar with redundant systems. Maybe you have a cold spare on the shelf for a switch that might go down. You might have a cold or warm data center location for a disaster. You could even have redundant devices in your enterprise to help you get back in to your equipment if something causes it to go offline. Well, I say that you do. If you’re Meta/Facebook you didn’t have them this time last year.

Don’t mistake redundancy for resilience though. Like the tire analogy above you’re not going to be able to fix a flat while you’re driving. Yes, I’ve seen the crazy video online of people doing that but aside from stunt driving you’re going to have to take some downtime from your travel to fix the tire. Likewise, a redundant setup that includes cold spares or out-of-band devices that are connected directly to your network could incur downtime if they go offline and lock you out of your management system. Facebook probably thought their out-of-band control system worked just fine. Until it didn’t.

The Right Gear for Resilience

At Networking Field Day 29 last week we were fortunate to see Opengear present again for the second time. I’m familiar with them from all the way back at Networking Field Day 2 in 2011 so their journey through the changes of networking over the past decade has been great to see. They make out-of-band devices and they make them well. They’re one of the companies that immediately spring to mind when you think about solutions for getting access to devices outside the normal network access method.

As a VAR there were times that I needed to make calls to locations in order to reboot devices or get console access to fix an issue. Whether it was driving 3 hours to press F1 to clear a failed power supply message or racing across town to restore phone service after locking myself out of an SSH session there are numerous reasons why having actual physical access to the console is important. Until we perfect quantum teleportation we’re going to have to solve that problem with technology. Here’s a video from the Networking Field Day session that highlights some of the challenges and solutions that Opengear has available:

Ryan Hogg brings up a great argument here for redundancy versus resiliency. Are you managing your devices in-band? Or do you have a dedicated management network? And what’s your backup for that dedicated network if something goes offline? VLAN separation isn’t good enough. In the event of a failure mode, such as a bridging loop or another attack that takes a switch offline you won’t be able to access the management network if you can’t sent packets through it. If the tire goes flat you’re stopped until it’s fixed.

Opengear solves this problem in a number of ways. The first is of course providing a secondary access method to your network. Opengear console devices have a cellular backup function that can allow you to access them in the event of an outage, either from the internal network or from the Internet going down. I can think of a couple of times in my career where I would have loved to have been able to connect to a cellular interface to undo a change that just happened that had unintended consequences. Sometimes reload in 5 doesn’t quite do the job. Having a reliable way to connect to your core equipment makes life easy for network operating systems that don’t keep from making mistakes.

However, as mentioned, redundancy is not resiliency. It’s not enough for us to have access to fix the problem while everything is down and the world is on fire. We may be able to get back in and fix the issue without needing to drive to the site but the users in that location are still down while we’re working. SD-WAN devices have offered us diverse connectivity options for a number of years now. If the main broadband line goes down just fail back to the cellular connection for critical traffic until it comes back up. Easy to do now that we have the proper equipment to create circuit diversity.

As outlined in the video above, Opengear has the same capability as well. If you don’t have a fancy SD-WAN edge device you can still configure Opengear console devices to act as a secondary egress point. It’s as simple as configuring the network with an IP SLA to track the state of the WAN link and installing the cellular route in the routing table if that link goes down. Once configured your users can get out on the backup link while you’re coming in to fix whatever caused the issue. If it’s the ISP having the issue you can log a ticket and confirm things are working on-site without having to jump in a car to see what your users see.

Resilience Really Matters

One of the things that Opengear has always impressed me with is their litany of use cases for their devices. I can already think of a ton of ways that I could implement something like this for customers that need monitoring and resilient connectivity options. Remote offices are an obvious choice but so too are locations with terrible connectivity options.

If you are working in a location with spotty connectivity you can easily deploy an Opengear device to keep an eye on the network and/or servers as well as providing an extra way for the site to get back online in the event of an issue. If the WAN circuit goes down you can just hop over to the cellular link until you get it fixed. Opengear will tell you something happened and you can log into the Lighthouse central management system to go there and collect data. If configured correctly your users may not even realize they’re offline! We’re almost at the point of changing the tire while we’re driving.


Tom’s Take

I am often asked if I miss working on networking equipment since I rarely touch it these days. As soon as I’m compelled to answer that question I remember all the times I had to drive somewhere to fix an issue. Wasted time can never be recovered. Resources cost money whether it’s money for a device or time spent going to fix one. I look at the capabilities that a company like Opengear has today and I wish I had those fifteen years ago and could deploy them to places I know needed them. In my former line of work redundant things were hard to come by. Resilient options were much more appealing because they offered more than just a back plan in case of failure. You need to pick resiliency every time because otherwise you’re going to be losing time replacing that tire when you could be rolling along fixing it instead.


Disclaimer

Opengear was a presenter at Networking Field Day 29 on September 7, 2022. I am an employee of Tech Field Day, which is the company that managed the event. This blog post represents my own personal thoughts about the presentation and is not the opinion of Tech Field Day. Opengear did not provide compensation for this post or ask for editorial approval. This post is my perspective alone.