About networkingnerd

Tom Hollingsworth, CCIE #29213, is a former network engineer and current organizer for Tech Field Day. Tom has been in the IT industry since 2002, and has been a nerd since he first drew breath.

Wi-Fi 6 Release 2, Or Why Naming Conventions Suck

I just noticed that the Wi-Fi Alliance announced a new spec for Wi-Fi 6 and Wi-Fi 6E. Long-time readers of this blog will know that I am a fan of referring to technology by the standard, not by a catch term that serves as a way to trademark something, like Pentium. Anyway, this updated new standard for wireless communications was announced on January 5th at CES and seems to be an entry in the long line of embarrassing companies that forget to think ahead when naming things.

Standards Bodies Suck

Let’s look at what’s included in the new release for Wi-Fi 6. The first and likely biggest thing to crow about is uplink multi-user MIMO. This technology is designed to enhance performance and reduce latency for things like video conferencing and uploading data. Essentially, it creates multi-user MIMO for data headed back the other direction. When the standard was first announced in 2018 who knew we would have spent two years using Zoom for everything? This adds functionality to help alleviate congestion for applications that upload lots of data.

The second new feature is power management. This one is aimed primarily at IoT devices. The combination of broadcast target wake time (TWT), extended sleep time, and multi-user spatial multiplexing power save (SMPS) are all aimed at battery powered devices. While the notes say that it’s an enterprise feature I’d argue this is aimed at the legion of new devices that need to be put into deep sleep mode and told to wake up at periodic intervals to transmit data. That’s not a laptop. That’s a sensor.

Okay, so why are we getting these features now? I’d be willing to bet that these were the sacrificial items that were holding up the release of the original spec of 802.11ax. Standards bodies often find themselves in a pickle because they need to get the specifications out the door so manufacturers can start making gear. However, if there are holdups in the process it can delay time-to-market and force manufacturers to wait or take a gamble on the supported feature set. And if there is a particular feature that is being hotly debated it’s often dropped because of the argument or because it’s too complex to implement.

These features are what has been added to the new specification, which doesn’t appear to change the 802.11ax protocol name. And, of course, these features must be added to new hardware in order to be available, both in radios and client devices. So don’t expect to have the hot new Release 2 stuff in your hands just yet.

A Marketing Term By Any Other Name Stinks

Here’s where I’m just shaking my head and giggling to myself. Wi-Fi 6 Release 2 includes improvements for all three supported bands of 802.11ax – 2.4GHz, 5GHz, and 6GHz. That means that Wi-Fi 6 Release 2 supersedes Wi-Fi 6 and Wi-Fi 6E, which were both designed to denote 802.11ax in the original supported spectrums of 2.4 and 5GHz and then to the 6GHz spectrum when it was ratified by the FCC in the US.

Let’s all step back and realize that the plan to simplify the naming convention of the Wi-Fi alliance for marketing has failed spectacularly. In an effort to avoid confusing consumers by creating a naming convention that just counts up the Wi-Fi Alliance has committed the third biggest blunder. They forgot to leave room for expansion!

If you’re old enough you probably remember Windows 3.1. It was the biggest version of Windows up to that time. It was the GUI I cut my teeth on. Later, there were new features that were added, which meant that Microsoft created Windows 3.11, a minor release. There was also a network-enabled version, Windows for Workgroups 3.11, which included still other features. Was Windows 3.11 just as good as Windows for Workgroups 3.11? Should I just wait for Windows 4.0?

Microsoft fixed this issue by naming the next version Windows 95, which created a bigger mess. Anyone that knows about Windows 95 releases know that the later ones had huge new improvements that made PCs easier to use. What was that version? No, not Windows 97 or whatever the year was. No, it was Windows 95 OEM Service Release 2 (Win95OSR2). That was a mouthful for any tech support person at the time. And it showed why creating naming conventions around years was a dumb idea.

Now we find ourselves in the mess of having a naming convention that shows major releases of the protocol. Except what happens when we have a minor release? We can’t call it by the old name because people won’t be impressed that it contains new features. Can we add a decimal to the name? No, because that will mess up the clean marketing icons that have already been created. We can’t call it Wi-Fi 7 because that’s already been reserved for the next protocol version. Let’s just stick “release 2” on the end!

Just like with 802.11ac Wave 2, the Wi-Fi Alliance is backed into a corner. They can’t change what they’ve done to make things easier without making it more complicated. They can’t call it Wi-Fi 7 because there isn’t enough difference between Wi-Fi 6 and 6E to really make it matter. So they’re just adding Release 2 and hoping for the best. Which will be even more complicated when people have to start denoting support for 6GHz, which isn’t universal, with monikers like Wi-Fi 6E Release 2 or Wi-Fi 6 Release 2 Plus 6E Support. This can of worms is going to wiggle for a long time to come.


Tom’s Take

I sincerely hope that someone that advised the Wi-Fi Alliance back in 2018 told them that trying to simplify the naming convention was going to bite them in the ass. Trying to be cool and hip comes with the cost of not being able to differentiate between minor version releases. You trade simplicity for precision. And you mess up all those neat icons you built. Because no one is going to legitimately spend hours at Best Buy comparing the feature sets of Wi-Fi 6, Wi-Fi 6E, and Wi-Fi 6 Release 2. They’re going to buy what’s on sale or what looks the coolest and be done with it. All that hard work for nothing. Maybe the Wi-Fi Alliance will have it figured out by the time Wi-Fi 7.5 Release Brown comes out in 2025.

Make Sure You Juggle The Right Way in IT

When my eldest son was just a baby, he had toys that looked like little baseballs. Long story short, I decided to teach myself to juggle with them. I’d always wanted to learn and thought to myself “How hard can it be?” Well, the answer was harder than I thought and it took me more time that I realized to finally get the hang of it.

One of the things that I needed to learn is that adding in one more ball to track while I’m trying to manage the ones that I had wasn’t as simple as it sounded. You would think that adding in a fourth ball should only be about 25% harder than the three you had been working with before. Or, you might even believe the statistical fallacy that you’re only going to fail about a quarter of the time and be successful the rest. The truth is that adding in one more object makes your entire performance subpar until you learn to adjust for it.

Clogging Up the Pipe

I mention this example because the most obvious application for the juggling metaphor is in Quality of Service (QoS). If you’ve ever read any of the training material related to QoS over the years, you’ll know that an oversubscribed link doesn’t perform poorly for the packets that are added in at the end. When a link hits the point of saturation all of the data flowing down the pipe is impacted in some way, whether it’s delays or or dropped packets or even application timeouts.

We teach that you need to manage congestion on the link as a whole and not just the data that is added that takes you over the stated rate. This is why we have queuing methods that are specifically tuned for latency sensitive traffic like voice or video. You can’t assume that traffic that gets stuffed in at the start will be properly handled. You can’t assume that all data is just going to line up in an orderly fashion and wait its turn. Yes, the transmission queue on the device is going to process the packets in a serial manner, but you can’t know for sure what packets are going to be shoved in the queue without some form of management.

It’s important to understand that QoS is about the quality of the experience for all consumers of the link and not just a select group. That’s why texts will teach you about priority queuing methods and why they’re so inefficient. If the priority queues are the only ones getting served then the regular queues will fail to send traffic. If users get creative and try to mark their packets as priority then the priority queue becomes no better than the regular queue.

QoS for Your Brain

All of these lessons for juggling packets and prioritizing them within reason don’t just resonate with technology. The same principles apply to the work you do and the projects and tasks that you take on. I can’t tell you the number of times I’ve thought to myself “I can just handle this one little extra thing and it won’t make a big difference.” Except it does make a big difference in the long run. Because adding one more task to my list is just like adding one more ball to the juggling list. It’s not additive. It adds a whole new dimension to what you’re working on.

Just like with the bandwidth example, the one extra piece added to the end makes the whole experience worse overall. Now you’re juggling more than you can handle. Instead of processing what you have efficiently and getting things done on time you’re flipping back and forth trying to make sure that all the parts are getting worked on properly and, in the end, using too much time inefficiently. Add in the likelihood that this new task is “important” and gets placed near the top of the list and you can quickly see how the priority queue example above is fitting. When every task is critical, there are no critical tasks.

Prioritize Before The Piles Happen

As luck would have it, the best way to deal with these issues of juggling too many tasks is the same as dealing with oversubscription on a link. You need to understand what your ability to deal with tasks looks like. Maybe you can handle eight things a day. Are those eight complex things? Eight easy things? Four of each? You need to know what it takes to maximize your productivity. If you don’t know what you can handle then you’ll only find out you’re oversubscribed when you take on one thing too many. And it’s too late to turn back after that.

Next, you need to manage the tasks you have in some way. Maybe it’s a simple list. But it’s way easier if the list has a way to arrange priority and deal with complicated or less critical tasks after the important stuff is done first. Remember that something being complex and critical is going to be a challenge. Easy tasks can be knocked out and crossed off your list sooner. You can also make sure that tasks that need to happen in a certain order are arranged in that way.

Lastly, you need to model the QoS drop method. Which means saying “no” to things that are going to oversubscribe you. It seems inelegant and will lead to others getting frustrated that you can’t get the work done. However, they also need to understand that if you can’t get the work done because you’re tasked with too much you’re going to do a poor job anyway. It’s better to get things done in a timely manner and tell people to come back later than take on more than you can do and disappoint everyone. And if someone tried to get creative and tell you their task is too important to put off, remind them that every task is critical to someone and you decide how important things are.


Tom’s Take

This is absolutely a case of “do as I say, not as I do”. I’m the world’s worst for taking on more than I can handle to avoid making other people feel disappointed. No matter how many times I remind myself that I can’t take on too much I have been known to find myself in a situation where I’m oversubscribed and my performance is suffering because of it. Use this as an opportunity to get a better handle on juggling things on your side. I never got good enough to juggle more than four at once and I’m okay with that. Don’t feel like you have to take on more than you can or else you’ll end up working in a circus.

Double the Fun in 2022

It’s January 1 again. The last 365 days have been fascinating for sure. The road to recovery doesn’t always take the straightest path. 2021 brought some of the the normal things back to us but we’re still not quite there yet. With that in mind, I wanted to look back at some of the things I proposed last year and see how they worked out for me:

  • Bullet Journaling: This one worked really well. When I remembered to do it. Being able to chart out what I was working on and what I needed to be doing helped keep me on track. The hardest part was remembering to do it. As I’ve said before, I always think I have a great memory and then remember that I forgot I don’t. Bullet journaling helped me get a lot of my tasks prioritized and made sure that the ones that didn’t get done got carried over to be finished later. I kind of stopped completely at the end of the year when things got hectic and I think that is what led me to feeling like everything was chaotic. I’m going to start again for 2022 and make sure to add some more flair to what I’m doing to make it stick for real this time.
  • More Video Content: This one was a mixed bag. I did record a full year of Tomversations episodes as well as the Rundown and various episodes of the On-Premise IT Roundtable podcast. The rest of my plans didn’t quite come to fruition but I think there’s still a spot for me to do things in 2022 to increase the amount of video content I’m doing. The reason is simple: more and more people are consuming content in video form instead of reading it. I think I can find a happy medium for both without increasing the workload of what I’m doing.
  • More Compelling Content: This was the part I think I did the most with. A considerable number of my posts this year were less about enterprise IT technical content and more about things like planning, development, and soft skills. I spent more time talking about the things around tech than I did talking about the tech itself. While that does have a place I wonder if it’s as compelling for my audience as the other analysis that I do. Given that my audience has likely shifted a lot over the last decade I’m not even sure what people read my blog for any longer. Given the number of comments that I get on IPv6 posts that were written five or six years ago I may not even be sure who would be interested in the current content here.

Okay, 2021 was a mixed bag of success and areas for improvement. My journaling helped me stay on task but I still felt a lot of the pressure of racing from task to task and my grand ideas of how to create more and do more ultimately fell away as things stayed busy. So, where to go from here?

  • More Analytical Content: Some of the conversations I’ve had over the year remind me that I have a unique place in the industry. I get to see a lot of what goes on and I talk to a lot of people about it. That means I have my own viewpoint on technologies that are important. While I do a lot of this for work, there are some kinds of analysis that are better suited for this blog. I’m going to spend some time figuring those out and posting them here over the year to help create content that people want to read.
  • Saying No to More Things: Ironically enough, one of the things I need to get better at in 2022 is turning things down. It’s in my nature to take on more than I can accomplish to make sure that things get done. And that needs to stop if I’m going to stay sane this year. I’m going to do my best to spread out my workload and also to turn down opportunities that I’m not going to be able to excel at doing. It may be one of the hardest things I do but I need to make it happen. Only time will tell how good I am and turning people down.
  • Getting In Front of Things: This one is more of a procedural thing for me but it’s really important. Rather than scrambling at the last minute to finish a script or get something confirmed, I’m going to try my hardest to plan ahead and make sure I’m not racing through chaos. With all the events I have coming up, both work and personal, I can’t afford to leave things to the last minute. So I’m going to be trying really hard to think ahead. We’ll see how it goes.

Tom’s Take

My January 1 post is mostly for me to keep myself honest over the year. It’s a way for me to set goals and stick to them, or at the very least come back to the next January 1 and see where I need to improve. I hope that it helps you a bit in your planning as well!

Holiday Networking Thoughts from 2021

It’s the Christmas break for 2021, which means lots of time spent doing very little work-related stuff. I’m currently putting together a Lego set, playing Metroid Dread and working on beating Ocarina of Time again.

As I waited for updates to download on Christmas morning I remembered how many packets must be flying across the wire to update software and operating systems for consoles. Even having done a few of the updates the night before I could see the traffic to those servers started to get a bit congested. It’s like Black Friday but for the latest patches to keep your games running. Add in the content that needs to be installed now in order to make that game disc work, or the download-only consoles for sale, and you can see that network engineers aren’t going to be a dying profession any time soon.

I’m a bit jaded because I come from a time when you didn’t need to be constantly connected to use software or need to download an update every few days. Heck, some of the bugs in Ocarina of Time have been there for over twenty years because those cartridges are not designed to be patched, having been created before a time when you could barely get online with a modem, let alone wirelessly connect a console.

I also am happy that upgrading devices in the house means fewer and fewer older units performing poorly on the wireless network. As more devices require me to connect them to the network for updates or app connectivity, I’m reminded that things like the Xbox 360 need low data rates enabled to work properly and that makes me sad. But I also can’t turn them off for fear that nothing will work and my children will scream. I don’t think spending a ton of money to get rid of an 802.11b client is really that big of a deal but I’m happy to see them go when I get the chance.

Likewise, I’m going to need to upgrade my APs a bit now that I have clients that can actually use 802.11ax (Wi-Fi 6). Even the older clients will get a performance boost. So It’s a matter of catching a good AP on sale and getting it done. Since I don’t use big box APs I just have to look a bit harder.


Tom’s Take

Make sure you give a shoutout to your friendly neighborhood network engineer for all their hard work making sure the services we’re currently consuming stayed up while the skeleton crew was carrying the pager this weekend. We’ve seen a lot of services crash on Christmas morning in recent years because of unexpected load. Also, give yourselves a hand for keeping your own network up long enough to download the latest DLC for a game or ensure that your new smart appliance can talk to the fancy app you need to use to control it. Let’s make it through the rest of the year with the change freeze intact and start 2022 off on the right foot with no outages.

A Recipe for Presentation Success

When I was a kid, I loved to help my mother bake. My absolute favorite thing to make was a pecan pie. I made sure I was always the one that got to do the work to fix it during the holidays. When I was first starting out I made sure I followed the recipe to the letter. I mixed everything in the order that it was listed. One of the first times I made the pie I melted the butter and poured it into the mixture which also had an egg. To my horror I saw the egg starting to cook and scramble in the bowl due to the hot butter. When I asked my mom she chuckled and said, “Now you get to learn about why the recipe isn’t always right.”

Throughout my career in IT and in presentations, I’ve also had to learn about why even if the recipe for success is written down properly there are other things you need to take into account before you put everything together. Just like tempering a mixture or properly creaming butter and sugar together, you may find that you need to do some things in a different order to make it all work correctly.

Step by Out of Step

As above, sometimes you need to know how things are going to interact so you do them in the right order. If you pour hot liquid on eggs you’re going to cook them. If you do a demo of your product without providing context for what’s happening you’re likely going to lose your audience. You need to set things up in the proper order for it all to make sense.

Likewise if you spend all your time talking about a problem that needs to be solved without telling your listeners that you solve the problem you’re going to have them focused on what’s wrong, not on how you fix it. Do you want them thinking about how you get a flat tire when you run over a nail? Or do you want them to buy your tires that don’t go flat when you run over sharp objects? It’s important to sell your product, not the problem.

It’s also important to know when to do those things out of order. Does your demo do something magical or amazing with a common issue? It might be more impactful to have your audience witness what happens before explaining how it works behind the scenes. It’s almost like a magician revealing their trick. Wow them with the result before you pull back the curtain to show them how it’s done.

The feel for how to do this varies from presentation to presentation. Are you talking to an audience that doesn’t understand the topic at all? You need to start with a lead-in or some other kind of level setting so no one gets lost. Are they experienced and understand the basics? You should be able to jump in at a higher level and show off a few things before going into detail. You have to understand whether or not you’re taking to a group of neophytes or a crowd of wizened veterans.

A counterpoint to this is the crowd of people that might be funding your project or startup. If they’re a person that gets pitched daily about “the problem” or they have a keen understanding of the market, what exactly are you educating them about when you open with a discussion of the issues? Are you telling them that you know what they are? Or are you just trying to set a hook? Might be worth explaining what you do first and then showing how you attack the problem directly.

Weaving a Story

The other thing that I see being an issue in presentations is the lack of a story. A recipe tells a story if you listen. Things have relationships. Liquids should be mixed together. Dry ingredients should be combined beforehand. Certain pieces should be put on last. If you put the frosting on a cake before you put it in the oven you’re going to be disappointed. It’s all part of the story that links the parts together.

Likewise, your presentation or lesson should flow. There should be a theme. It should make sense if you watch it. You can have individual pieces but if you tie it all together you’re going to have a better time of helping people understand it.

When I was growing up, TV shows didn’t tell longer stories. Episodes of the Addams Family or Gilligan’s Island stood alone. What happened in the first season didn’t matter in the next. Later, the idea of a narrative arc in a story started appearing. If you watch Babylon 5 today you’ll see how earlier episodes introduce things that matter later. Characters have growth and plot threads are tied up before being drawn out into new tapestries. It’s very much a job of weaving them all together.

When you present, do your sections have a flow? Do they make sense to be together? Or does it all feel like an anthology that was thrown together? Even anthologies have framing devices. Maybe you’re brining in two different groups that have different technologies that need to be covered. Rather than just throwing them out there you could create an overview of why they are important or how they work together. It’s rare that two things are completely unrelated, especially if you’re presenting them together.


Tom’s Take

If all you ever did was list out ingredients for recipes you’d be missing the important parts. They need to be combined in a certain order. Things need to go together properly. Yes, you’re going to make mistakes when you do it for the first time and you don’t understand the importance of certain things. But that learning process should help you put them together the way they need to be arranged. Take notes. Ask for feedback. And most importantly, know when it’s time to change the recipe to help you make it better the next time.

Is Disaggregation Going to Be Cord Cutting for the Enterprise?

There’s a lot of talk in the networking industry around disaggregation. The basic premise is that by decoupling the operating system from the hardware you can gain the freedom to run the devices you want from any vendor with the software that does what you want it to do. You can standardize or mix-and-match as you see fit. You gain the ability to direct the way your network works and you control how things will be going forward.

To me it sounds an awful lot like the trend of “cutting the cord” or unsubscribing from cable TV service and picking and choosing how you want to consume your content. Ten years ago the idea of getting rid of your cable TV provider was somewhat crazy. In 2021 it seems almost a given that you no long need to rely on your cable provider for entertainment. However, just like with the landscape of the post-cable cutting world, I think disaggregation is going to lead to a vastly different outcome than expected.

TNSTAAFL

Let’s get one thing out of the way up front: This idea of “freedom” when it comes to disaggregation and cord cutting is almost always about money. Yes, you want the ability to decide what software runs on your system. You don’t want to have unnecessary features or channels in your lineup. But why? I think maybe 5% of the community is worried about code quality or attack surfaces. The rest? They want to pay less for the software or hardware by unbundling the two. Instead of getting better code for their switches they’re really just chasing a lower cost per unit to run things. If that weren’t the case, why do so many of these NOS vendors run on Linux?

Yes, that feels like a bit of shot but reality speaks volumes over the pleasantries we often spout. The value of disaggregation is a smaller bottom line. Code quality can be improved over time with the proper controls in place. Hell, you could even write your own NOS given the right platform and development resources. However, people don’t want to build the perfect NOS or help vendors with the code issues. They want someone to build 90% of the perfect NOS and then sell it to them cheaply so they can run it on a cheap whitebox switch.

This is an issue that is faced by developers the world over. Look at the number of apps in the various mobile app stores that have a free entry point or are a “Freemium” business model. You don’t pay up front but as soon as you find a feature you really like it’s locked behind a subscription model. Why? Because one-time purchases don’t fund development. If everyone buys your app and then expects you to keep providing features for it and not just bug fixes, where does the investment for that development come from? Work requires resources – time or money. If you’re not getting paid for something you have to invest more time to make it work the way you want.

Vendors of disaggregated systems are finding themselves in a similar quandary. How do we charge enough for the various features we want to put into the system to be able to develop new features? The common way I see this done is to put in the most basic features that customers would want and then wait for someone to ask for something to be added. If the customer is asking for it the odds are good they’ll be willing to pay for it. You can even get them to buy your software now and sign an agreement that you’ll include the new feature in a few weeks in order to be sure your development resources aren’t wasted.

There are other ways, such as relying on single merchant silicon platforms or developing tight relationships with other vendors in the market, but ultimately it comes back to the question of resources. What are you willing to invest to make this happen? And what are you willing to accept as a cost that must be paid to get what you think you want?

The Buffet of Plenty…of Stuff You Don’t Want

The other aspect of this comparison is how the cable TV market responded to cord cutting. People started leaving cable TV for apps like Netflix and Hulu because they were cheaper than paying for a full cable subscription and had most of the content that people wanted. For the few pieces that weren’t available there were workarounds. By and large, you could find most of what you wanted in an auxiliary app when you occasionally wanted it.

So is this how things are today? Or did the market shift to the response of what customer behavior was? I think you’ll find that you’re not paying a single lump sum for content if you cut the cord for your cable provider. However, you are paying a large portion of that investment in separate apps that offer a portion of the content on-demand. And that’s why separating things is going to lead to new market dynamics.

The first behavior we saw was every media company coming up with their own app to host content. Instead of having a Disney channel on cable you now had a number of Disney apps that replicated the content channels. Later they merged into a single app with all the content. But was it all the content you wanted? Or was it all the content they owned? The drive for companies to create apps was not to offer customers a way to consume content along with their existing subscriptions. It was to provide a landing page for content you couldn’t find anywhere else.

That’s where phase two kicks in. Once you’ve created the destination, you need to make it the only place to be. That means removing content from other locations. Netflix started losing content when the creators started taking control of their own content. Soon it was necessary to create custom content to replace what was lost. Now, instead of buying a cable subscription and getting all the channels you had to sign up for five different apps, each comprising one or two of the channels you used to watch. Disney content is in the Disney app. NBC content is in another. The idea of channel surfing is gone. The back catalog of content added to the apps served more to entice people to keep their subscriptions during droughts of fresh new content.

How does this whole model break down in the enterprise? Well, going back to our earlier discussion about features being added to devices, what are you going to have to do to get new functions in your operating system? Are you going to require the vendor to write them on their schedule? Are you going to use a separate app or platform? Why should the vendor support some random feature that might not get much adoption and would take a significant amount of resources to build? Why not just make you do it yourself?

The idea is that you gain freedom and cheaper software. The hope is that you can build an enterprise network for half of what it would normally cost. The reality is that you’re going to gain less functionality and spend more time integrating things together on your own instead of just putting in a turnkey solution. And yes, there are people out there that are nodding their heads and saying they would love to do this. They want the perfect network with the perfect cheap NOS and whitebox hardware. But do you want this to be your only job for the rest of your career?

Once you build things the way you want them you become the only person that can work on them. You become the only source of support for your solution. If it’s a custom snowflake of a network you are the only person that can fix the snow issues. Traditional software and hardware may be unwieldy and difficult to troubleshoot but you can also call a support line where people have been paid to get training on how to implement and fix issues. If you built it yourself you’re the person that has to pick up the phone to fix it. Unless you want to train your team to support it too. Which takes time and money. So your savings between the two solutions are going to evaporate. And if you want the NOS vendor or the hardware supplier to support more functions to make it all easier you’re going to drive the price of the equipment up. So instead of writing one big check to the old guard you’re writing a bunch of little ones to every part of the new infrastructure you helped create.


Tom’s Take

I know it sounds like I’m not a fan of all this disaggregation stuff. In fact, I am a huge proponent of it. I just don’t buy the “freedom” excuse. My business background helps me understand the resource contention issues. My history of supporting snowflake implementations reminds me that you have to be able to turn your work over to someone else at some point in the future. Disaggregation has a lot of positive effects. You can mix and match your software and hardware and make it much easier to support for your own purposes. You no longer have to take a completed project and find workarounds to fit it to your needs. You get what you want. But don’t think you’re going to be able to get exactly what you need without some work of your own. Just like the cable cord cutting craze, you’re going to find out that you’re getting something totally different in the short term and a much different consumption model when the market shifts to the demands of the consumers. Don’t get complacent with your solutions and be ready to adapt when the suppliers force your hand.

You Down with IoT? You Better Be!

Did you see the big announcement from AWS re:Invent that Amazon has a preview of a Private 5G service? It probably got buried under the 200 other announcements that came out on so many other things so I’ll forgive you for missing it. Especially if you also managed to miss a few of the “hot takes” that mentioned how Amazon was trying to become a cellular provider. If I rolled my eyes any harder I might have caused permanent damage. Leave it to the professionals to screw up what seems to be the most cut-and-dried case of not reading the room.

Amazon doesn’t care about providing mobile service. How in the hell did we already forget about the Amazon (dumpster) Fire Phone? Amazon isn’t trying to supplant AT&T or Verizon. They are trying to provide additional connectivity for their IoT devices. It’s about as clear as it can get.

Remember all the flap about Amazon Sidewalk? How IoT devices were going to use 900 MHz to connect to each other if they had no other connectivity? Well, now it doesn’t matter because as long as one speaker or doorbell has a SIM slot for a private 5G or CBRS node then everything else can connect to it too. Who’s to say they aren’t going to start putting those slots in everything going forward? I’d be willing to bet the farm that they are. It’s cheap compared to upgrading everything to use 802.11ax radios or 6 GHz technology. And the benefits for Amazon are legion.

It’s Your Density

Have you ever designed a wireless network for a high-density deployment? Like a stadium or a lecture hall? The needs of your infrastructure look radically different compared to your home. You’re not planning for a couple of devices in a few dozen square feet. You’re thinking about dozens or even hundreds of devices in the most cramped space possible. To say that a stadium is one of the most hostile environments out there is underselling both the rabid loyalty of your average fan and the wireless airspace they’re using to post about how the other team sucks.

You know who does have a lot of experience designing high density deployments with hundreds of devices? Cellular and mobile providers. That’s because those devices were designed from the start to be more agreeable to hostile environments and have higher density deployments. Anyone that can think back to the halcyon days of 3G and how crazy it got when you went to Cisco Live and had no cell coverage in the hotel until you got to the wireless network in the convention center may disagree with me. But that exact scenario is why providers started focusing more on the number of deployed devices instead of the total throughput of the tower. It was more important in the long run to get devices connected at lower data rates than it was to pump up the wattage and get a few devices to shine at the expense of all the other ones that couldn’t get connected.

In today’s 5G landscape, it’s all about the clients. High density and good throughput. And that’s for devices with a human attached to them. Sure, we all carry a mobile phone and a laptop and maybe a tablet that are all connected to the Wi-Fi network. With IoT, the game changes significantly. Even in your consumer-focused IoT landscape you can probably think of ten devices around you right now that are connected to the network, from garage door openers to thermostats to light switches or light bulbs.

IoT at Work

In the enterprise it’s going to get crazy with industrial and operational IoT. Every building is going to have sensors packed all over the place. Temperature, humidity, occupancy, and more are going to be little tags on the walls sampling data and feeding it back to the system dashboard. Every piece of equipment you use on a factory floor is going to be connected, either by default with upgrade kits or with add-on networking gear that provides an interface to the control system. If it can talk to the Internet it’s going to be enabled to do it. And that’s going to crush your average Wi-Fi network unless you build it like a stadium.

On the other hand, private 5G and private LTE deployments are built for this scale. And because they’re lightly regulated compared to full-on provider setups you can do them easily without causing interference. As long as someone that owns a license for your frequency isn’t nearby you can just set things up and get moving. And as soon as you order the devices that have SIM slots you can plug in your cards and off you go!

I wouldn’t be shocked to see Amazon start offering a “new” lineup of enterprise-ready IoT devices with pre-installed SIMs for Amazon Private 5G service. Just buy these infrastructure devices from us and click the button on your AWS dashboard and you can have on-prem 5G. Hell, call it Network Outpost or something. Just install it and pay us and we’ll take care of the rest for you. And as soon as they get you locked in to their services they’ve got you hooked. Because if you’re already using those devices with 5G, why would you want to go through the pain on configuring them for the Wi-Fi?

This isn’t a play for consumers. Configuring a consumer-grade Wi-Fi router from a big box store is one thing. Private 5G is beyond most people, even if it’s a managed service. It also offers no advantages for Amazon. Because private 5G in the consumer space is just like hardware sales. Customers aren’t going to buy features as much as they’re shopping for the lowest sticker price. In the enterprise, Amazon can attach private 5G service to existing cloud spend and make a fortune while at the same time ensuring their IoT devices are connected at all times and possibly even streaming telemetry and collecting anonymized data, depending on how the operations contracts are written. But that’s a whole different mess of data privacy.


Tom’s Take

I’ve said it before but I’ll repeat it until we finally get the picture: IoT and 5G are now joined at the hip and will continue to grow together in the enterprise. Anyone out there that sees IoT as a hobby for home automation or sees 5G as a mere mobile phone feature will be enjoying their Betamax movies along with web apps on their mobile phones. This is bigger than the consumer space. The number of companies that are jumping into the private 5G arena should prove the smoke is hiding a fire that can signal that Gondor is calling for aid. It’s time you get on board with IoT and 5G and see that. The future isn’t a thick client with a Wi-Fi stack that you need to configure. It’s a small sensor with a SIM slot running on a private network someone else fixes for you. Are you down with that?

A Gift Guide for Sanity In Your Home IT Life

If you’re reading my blog you’re probably the designated IT person for your family or immediate friend group. Just like doctors that get called for every little scrape or plumbers that get the nod when something isn’t draining over the holidays, you are the one that gets an email or a text message when something pops up that isn’t “right” or has a weird error message. These kinds of engagements are hard because you can’t just walk away from them and you’re likely not getting paid. So how can you be the Designated Computer Friend and still keep your sanity this holiday season?

The answer, dear reader, is gifts. If you’re struggling to find something to give your friends that says “I like you but I also want to reduce the number of times that you call me about your computer problems” then you should definitely read on for more info! Note that I’m not going to fill this post will affiliate links or plug products that have sponsored anything. Instead, I’m going to just share the classes or types of devices that I think are the best way to get control of things.

Step 1: Infrastructure Upgrades

When you go visit your parents for Thanksgiving or some other holiday check in, are they still running the same wireless network they got when they got their high-speed Internet? Is their Wi-Fi SSID still the default with the password printed on the side of the router/modem combo? Then you’re going to want to upgrade their experience to keep your sanity for the next few holidays.

The first thing you need to do it get control of their wireless setup. You need to get some form of wireless access point that wasn’t manufactured in the early part of the century. Most of the models on the market have Wi-Fi 6 support now. You don’t need to go crazy with a Wi-Fi 6E model for your loved ones right now because none of their devices will support it. You just need something more modern with a user interface that wasn’t written to look like Windows 3.1.

You also need to see about an access point that is controlled via a cloud console. If you’re the IT person in the group you probably already use some form control for your home equipment. You don’t need a full Meraki or Juniper Mist setup to lighten your load. That is, unless you already have one of those dashboards set up and you have spare capacity. Otherwise you could look at something like Ubiquiti as a middle ground.

Why a cloud controller AP? Because then you can log in and fix things or diagnose issues without needing to spend time talking to less technical users. You can find out if they have an unstable Internet connection or change SSID passwords at the drop of a hat. You can even set up notifications for those remote devices to let you know when a problem happens so you can be ready and waiting for the call. And you can keep tabs on necessary upgrades and such so you aren’t fielding calls when the next major exploit comes out and your parents call you asking if they’re going to get infected by this virus. You can just tell them they’re up-to-date and good to go. The other advantage of this method is that when you upgrade your own equipment at home you can just waterfall the old functional gear down to them and give them a “new to you” upgrade that they’ll appreciate.

Step 2: Device Upgrades

My dad was notorious for using everything long past the point of needing to be retired. It’s the way he was raised. If there’s a hole you patch it. If it breaks you fix it. If that fix doesn’t work you wrap it in duct tape and use it until it crumbles to dust. While that works for the majority of things out there it does cause issues with technology far too often.

He had a iPad that he loved. He didn’t use it all day, every day but he did use it frequently enough to say that it was his primary computing device. It was a fourth-generation device, so it fell out of fashion a few years ago. When he would call me and ask me questions about why it was behaving a certain way or why he couldn’t download some new app from the App Store I would always remind him that he had an older device that wasn’t fast enough or new enough to run the latest programs or even operating software. This would usually elicit a grumble or two and then we would move on.

If you’re the Designated IT Person and you spend half your time trying to figure out what versions of OS and software are running on a device, do yourself a favor and invest in a new device for your users just to ease the headaches. If they use a tablet as their primary computing device, which many people today do, then just buy a new one and help them migrate all the data across to the new one while you’re eating turkey or opening presents.

Being on later hardware ensures that the operating system is the latest version with all the patches for security that are needed to keep your users safe. It also means you’re not trying to figure out what the last supported version of the software was that works with the rest of the things. I’ve played this game trying to get an Apple Watch to connect to an older phone with mismatched software as well as trying to get support for newer wireless security on older laptops with very little capability to do much more than WPA1. The amount of hours I burned trying to make the old junk work with the new stuff would have been better served just buying a new version of the same old thing and getting all their software moved over. Problems seem to just disappear when you are running on something that was manufactured within the last five years.

Step 3: Help Them Remember

This is probably my biggest request: Forgotten passwords. Either it’s the forgotten Apple ID or maybe the wireless network password. My parents and in-laws forget the passwords they need to log into things all the time. I finally broke down and taught them how to use a password management tool a few years ago and it made all the difference in the world. Now, instead of them having to remember what their password was for a shopping site they can just set it to automatically fill everything in. And since they only need to remember the master password for their app they don’t have to change it.

Better yet, most of these apps have a secure section for notes. So all those other important non-password things that seem to come up all the time are great to put in here. Social Security Numbers, bank account numbers, and so much more can be put in one central location and made easy to access. The best part? If you make it a shared vault you can request access to help them out when they forget how to get in. Or you can be designated as a trusted party that can access the account in the event of a tragedy. Getting your loved ones used to using password vaults now makes it much easier to have them storing important info there in case something happens down the road that requires you to jump in without their interaction. Trust me on this.


Tom’s Take

Your loved ones don’t need knick knacks and useless junk. If you want to show them you love them, give them the gift of not having to call you every couple of days because they can’t remember the wireless password or because they keep getting this error that says their app isn’t support on this device. Invest in your sanity and their happiness by giving them something that works and that has the ability for you to help manage it from the background. If you can make it stable and useful and magically work before they call you with a problem you’re going to find yourself a happier person in the years to come.

IP Class is Now in Session

You may have seen something making the rounds on Twitter this week about a couple of proposed drafts designed to alleviate the problems with IPv4 exhaustion by repurposing some old IP spaces that aren’t available for use right now. Specifically:

Ultimately, this is probably going to fail for a variety of reasons and looks like it’s more of a suggestion than anything else but I wanted to take a moment to talk about why this isn’t an effective way of fixing address issues.

Error Bearers

The first reason that the Schoen drafts are going to fail is because most of the operating systems in the world won’t allow you to use reserved spaces for a system address. Because we knew years ago that certain spaces were marked as non-usable the logic was configured into the system to disallow the use of those spaces. And even if the system isn’t configured to disallow that space there’s no guarantee the traffic is going to be transmitted.

Let’s take 127/8 as a good example. Was it a smart idea to mark 16 million addresses as loopback host-only space? Nope. But that ship has sailed and we aren’t going to be able to easily fix it. Too many systems will see any address starting with 127 in first octet and assume it’s a loopback address. In much the same way as people have been known to assume the entire 192/8 address space is RFC1918 reserved instead of 192.168.0.0/16. Logic rules and people making decisions aren’t going to trust any space being used in that manner. Even if you did something creative like using NAT and only using it internally you’re not going to be able to patch every version of every operating system in your organization.

We modify rules all the time and then have to spend years updating those modifications. Take area codes in North America for example. The old rules used to say that an area code had to have a zero or a one for the middle digit – ([2-9][0-1][2-9]) to use the Cisco UCM parlance. If your middle digit was something other than a zero or a one it wasn’t a valid NANP area code. As we began to expand the phone system in 1995 we changed those rules and now have area codes with all manner of middle numbers.

What about prefixes? Those follow rules too. NANP prefixes must not start with a zero or a one – (area code) [2-9]XX-XXXX is the way they are coded. Prefixes that start with a zero or a one are invalid and can’t be used. If we suddenly decided that we needed to open up the numbers in existing area codes and include prefixes that start with those forbidden numbers we would need to reset all the dialing rules in systems all over the country. I know that I specifically programmed my CUCM servers to send an immediate error if you dialed a prefix with a zero or a one. And all of them would have to be manually reconfigured for such a change.

In much the same way, the address spaces that are reserved today as invalid would need to be patched out of systems from home computers to phones to networking equipment. And even if you think you got it all you’re going to miss one and wonder why it isn’t working. Worse yet, it might even silently fail because you may be able to transmit data to 95% of the systems out there but some intermediate system may discard your packets as invalid and never tell you what happened. You’ll spend hours or days chasing a problem you may not even be able to fix.

Avoiding the Solutions

The easiest way to look at these proposals is by understanding that people are really, really, really in love with IPv4. Despite the fact that using the effort of the changes necessary to implement these reserved spaces would be better spent on IPv6 adoption we still get these things being submitted. There is a solution but people don’t want to use it. The modern Internet relies so much on the cloud that it would be simple to enable IPv6 in your provider space and use your engineering talent to help provide better adoption for that instead. We’re already seeing that all over places with address space has been depleted for a while now.

It may feel easier to spend more effort to revitalize the IPv4 space we all know and love. It may even feel triumphant when we’re able to reclaim address space that was wasted and use it for something productive instead of just teaching that you can’t configure devices with those spaces. And millions of devices will have IP address space to use, or more accurately there will be millions of addresses available to sell to people that will waste them anyway. Then what?

The short term gain from opening up IPv4 space at the expense of not developing IPv6 adoption is a fallacy that will end in pain. We can keep putting policy duct tape on the IPv4 exhaustion problem but we are eventually going to hit a wall we can’t overcome. The math doesn’t work when your address space is only 32 bits in total. That’s why IPv6 expanded the amount of information in the address space.

Sure, there have been mistakes in the way that IPv6 address space has been allocated and provisioned. Those mistakes would need to eventually be corrected and other configurations would need to be done in order to efficiently utilize the space. Again, the effort should be made to fix problems with a future-proof solution instead of trying our hardest to keep the lights on with the old system that’s falling apart for a few more years.


Tom’s Take

The race to find every last possible way to utilize the IPv4 space is exactly what I expected when we’re in the death throes of using it instead of IPv6. The easy solutions are done. The market and hunger for IPv4 space is only getting stronger. Instead of weaning the consumers off their existing setups and moving them to something future proof we’re feeding their needs for short term gains. If the purpose of this whole exercise was to get more address space to be rationed out for key systems to keep them online longer I might begrudgingly accept it. However, knowing that it would likely be opened up and fed to providers to be auctioned off in blocks to be ultimately wasted means all the extra effort is for no gain. These IETF drafts have a lot of issues and we’re better off letting them expire in May 2022. Because if we take up this cause and try to make them a reality we’re going to have to relearn a lot of lessons of the past we’ve forgotten.

The Process Will Save You

I had the opportunity to chat with my friend Chris Marget (@ChrisMarget) this week for the first time in a long while. It was good to catch up with all the things that have been going on and reminisce about the good old days. One of the topics that came up during our conversation was around working inside big organizations and the way that change processes are built.

I worked at IBM as an intern 20 years ago and the process to change things even back then was arduous. My experience with it was the deployment procedures to set up a new laptop. When I arrived the task took an hour and required something like five reboots. By the time I left we had changed that process and gotten it down to half an hour and only two reboots. However, before we could get the new directions approved as the procedure I had to test it and make sure that it was faster and produced the same result. I was frustrated but ultimately learned a lot about the glacial pace of improvements in big organizations.

Slow and Steady Finishes the Race

Change processes work to slow down the chaos that comes from having so many things conspiring to cause disaster. Probably the most famous change management framework is the Information Technology Infrastructure Library (ITIL). That little four-letter word has caused a massive amount of headaches in the IT space. Stage 3 of ITIL is the one that deals with changes in the infrastructure. There’s more to ITIL overall, including asset management and continual improvement, but usually anyone that takes ITIL’s name in vain is talking about the framework for change management.

This isn’t going to be a post about ITIL specifically but about process in general. What is your current change management process? If you’re in a medium to large sized shop you probably have a system that requires you to submit changes, get the evaluated and approved, and then implemented on a schedule during a downtime window. If you’re in a small shop you probably just make changes on the fly and hope for the best. If you work in DevOps you probably call them “deployments” and they happen whenever someone pushes code. Whatever the actual name for the process is you have one whether you realize it or not.

The true purpose of change management is to make sure what you’re doing to the infrastructure isn’t going to break anything. As frustrating as it is to have to go through the process every time the process is the reason why. You justify your changes and evaluate them for impact before scheduling them. As opposed to something that can be termed as “Change and find out” kind of methodologies.

Process is ugly and painful and keeps you from making simple mistakes. If every part of a change form needs to be filled out you’re going to complete it to make sure you have all the information that is needed. If the change requires you to validate things in a lab before implementation then it’s forcing you to confirm that it’s not going to break anything along the way. There’s even a process exception for emergency changes and such that are more focused on getting the system running as opposed to other concerns. But whatever the process is it is designed to save you.

ITIL isn’t a pain in the ass on accident. It’s purposely built to force your justify and document at every step of the process. It’s built to keep you from creating disaster by helping you create the paper trail that will save you when everything goes wrong.

Saving Your Time But Not Your Sanity

I used to work with a great engineer name John Pross. John wrote up all the documentation for our migrations between versions of software, including Novell NetWare and Novell Groupwise. When it came time to upgrade our office Groupwise server there was some hesitation on the part of the executive suite because they were worried we were going to run into an error and lock them out of their email. The COO asked John if he had a process he followed for the migration. John’s response was perfect in my mind:

“Yes, and I treat every migration like the first one.”

What John meant is that he wasn’t going to skip steps or take shortcuts to make things go faster. Every part of the procedure was going to be followed to the letter. And if something came up that didn’t match what he thought the output should have been it was going to stop until he solved that issue. John was methodical like that.

People like to take shortcuts. It’s in our nature to save time and energy however we can. But shortcuts are where the change process starts falling apart. If you do something different this time compared to the last ten times you’ve done it because you’re in a hurry or you think this might be more efficient without testing it you’re opening yourself up for a world of trouble. Maybe not this time but certainly down the road when you try to build on your shortcut even more. Because that’s the nature of what we do.

As soon as you start cutting corners and ignoring process you’re going to increase the risk of creating massive issues rapidly. Think about something as simple as the Windows Server 2003 shutdown dialog box. People used to reboot a server on a whim. In Windows 2003, the server had a process that required you to type in a reason why you were manually shutting the server down from the console. Most people that rebooted the server fell into two camps: Those that followed their process and typed in the reason for the reboot and those that just typed “;Lea;lksjfa;ldkjfadfk” as the reason and then were confused six months from now when doing the post-mortem on an issue and cursing their snarky attitude toward reboot documentation.

Saving the Day

Change process saves you in two ways. The first is really apparent: it keeps you from making mistakes. By forcing you to figure out what needs to happen along the way and document the whole process from start to finish you have all the info you need to make things successful. If there’s an opportunity to catch mistakes along the way you’re going to have every opportunity to do that.

The second way change process saves you is when it fails. Yes, no process is perfect and there are more than a few times when the best intentions coupled with a flaw in the process created a beautiful disaster that gave everyone lots of opportunity to learn. The question always comes back to what was learned in that process.

Bad change failures usually lead to a sewer pipe of blame being pointed in your direction. People use process failures as a change to deflect blame and avoid repercussions for doing something they shouldn’t have or trying to increase their stock in the company. The truly honest failure analysis doesn’t blame anyone but the failed process and tries to find a way to fix it.

Chris told me in our conversation that he loved ITIL at one of his former jobs because every time it failed it led to a meaningful change in the process to avoid failure in the future. These are the reasons why blameless post-mortem discussions are so important. If the people followed the process and the process the people aren’t at fault. The process is incorrect or flawed and needs to be adjusted.

It’s like a recipe. If the instructions tell you to cook something for a specific amount of time and it’s not right, who is to blame? Is it you because you did what you were told? Is the recipe? Is it the instructions? If you start with the idea that you did the process right and start trying to figure out where the process is wrong you can fix the process for next time. Maybe you used a different kind of ingredient that needs more time. Or you made it thinner than normal and that meant cooking it too long this time. Whatever the result, you end up documenting the process and changing things for the future to prevent that mistake from happening again.

Of course, just like all good frameworks, change processes shouldn’t be changed without analysis. Because changing something just to save time or take a shortcut defeats the whole purpose! You need to justify why changes are necessary and prove they provide the same benefit with no additional exposure or potential loss. Otherwise you’re back to making changes and hoping you don’t get burned this time.


Tom’s Take

ITIL didn’t really become a popular thing until after I left IBM but I’m sure if I were still there I’d be up to my eyeballs in it right now. Because ITIL was designed to keep keyboard cowboys like me from doing things we really shouldn’t be doing. Change management process are designed to save us at every step of the way and make us catch our errors before they become outages. The process doesn’t exist to make our lives problematic. That’s like saying a seat belt in a car only exists to get in my way. It may be a pain when you’re dealing with it regularly but when you need it you’re going to wish you’d been using it the whole time. Trust in the process and you will be saved.