Holiday Networking Thoughts from 2021

It’s the Christmas break for 2021, which means lots of time spent doing very little work-related stuff. I’m currently putting together a Lego set, playing Metroid Dread and working on beating Ocarina of Time again.

As I waited for updates to download on Christmas morning I remembered how many packets must be flying across the wire to update software and operating systems for consoles. Even having done a few of the updates the night before I could see the traffic to those servers started to get a bit congested. It’s like Black Friday but for the latest patches to keep your games running. Add in the content that needs to be installed now in order to make that game disc work, or the download-only consoles for sale, and you can see that network engineers aren’t going to be a dying profession any time soon.

I’m a bit jaded because I come from a time when you didn’t need to be constantly connected to use software or need to download an update every few days. Heck, some of the bugs in Ocarina of Time have been there for over twenty years because those cartridges are not designed to be patched, having been created before a time when you could barely get online with a modem, let alone wirelessly connect a console.

I also am happy that upgrading devices in the house means fewer and fewer older units performing poorly on the wireless network. As more devices require me to connect them to the network for updates or app connectivity, I’m reminded that things like the Xbox 360 need low data rates enabled to work properly and that makes me sad. But I also can’t turn them off for fear that nothing will work and my children will scream. I don’t think spending a ton of money to get rid of an 802.11b client is really that big of a deal but I’m happy to see them go when I get the chance.

Likewise, I’m going to need to upgrade my APs a bit now that I have clients that can actually use 802.11ax (Wi-Fi 6). Even the older clients will get a performance boost. So It’s a matter of catching a good AP on sale and getting it done. Since I don’t use big box APs I just have to look a bit harder.


Tom’s Take

Make sure you give a shoutout to your friendly neighborhood network engineer for all their hard work making sure the services we’re currently consuming stayed up while the skeleton crew was carrying the pager this weekend. We’ve seen a lot of services crash on Christmas morning in recent years because of unexpected load. Also, give yourselves a hand for keeping your own network up long enough to download the latest DLC for a game or ensure that your new smart appliance can talk to the fancy app you need to use to control it. Let’s make it through the rest of the year with the change freeze intact and start 2022 off on the right foot with no outages.

A Recipe for Presentation Success

When I was a kid, I loved to help my mother bake. My absolute favorite thing to make was a pecan pie. I made sure I was always the one that got to do the work to fix it during the holidays. When I was first starting out I made sure I followed the recipe to the letter. I mixed everything in the order that it was listed. One of the first times I made the pie I melted the butter and poured it into the mixture which also had an egg. To my horror I saw the egg starting to cook and scramble in the bowl due to the hot butter. When I asked my mom she chuckled and said, “Now you get to learn about why the recipe isn’t always right.”

Throughout my career in IT and in presentations, I’ve also had to learn about why even if the recipe for success is written down properly there are other things you need to take into account before you put everything together. Just like tempering a mixture or properly creaming butter and sugar together, you may find that you need to do some things in a different order to make it all work correctly.

Step by Out of Step

As above, sometimes you need to know how things are going to interact so you do them in the right order. If you pour hot liquid on eggs you’re going to cook them. If you do a demo of your product without providing context for what’s happening you’re likely going to lose your audience. You need to set things up in the proper order for it all to make sense.

Likewise if you spend all your time talking about a problem that needs to be solved without telling your listeners that you solve the problem you’re going to have them focused on what’s wrong, not on how you fix it. Do you want them thinking about how you get a flat tire when you run over a nail? Or do you want them to buy your tires that don’t go flat when you run over sharp objects? It’s important to sell your product, not the problem.

It’s also important to know when to do those things out of order. Does your demo do something magical or amazing with a common issue? It might be more impactful to have your audience witness what happens before explaining how it works behind the scenes. It’s almost like a magician revealing their trick. Wow them with the result before you pull back the curtain to show them how it’s done.

The feel for how to do this varies from presentation to presentation. Are you talking to an audience that doesn’t understand the topic at all? You need to start with a lead-in or some other kind of level setting so no one gets lost. Are they experienced and understand the basics? You should be able to jump in at a higher level and show off a few things before going into detail. You have to understand whether or not you’re taking to a group of neophytes or a crowd of wizened veterans.

A counterpoint to this is the crowd of people that might be funding your project or startup. If they’re a person that gets pitched daily about “the problem” or they have a keen understanding of the market, what exactly are you educating them about when you open with a discussion of the issues? Are you telling them that you know what they are? Or are you just trying to set a hook? Might be worth explaining what you do first and then showing how you attack the problem directly.

Weaving a Story

The other thing that I see being an issue in presentations is the lack of a story. A recipe tells a story if you listen. Things have relationships. Liquids should be mixed together. Dry ingredients should be combined beforehand. Certain pieces should be put on last. If you put the frosting on a cake before you put it in the oven you’re going to be disappointed. It’s all part of the story that links the parts together.

Likewise, your presentation or lesson should flow. There should be a theme. It should make sense if you watch it. You can have individual pieces but if you tie it all together you’re going to have a better time of helping people understand it.

When I was growing up, TV shows didn’t tell longer stories. Episodes of the Addams Family or Gilligan’s Island stood alone. What happened in the first season didn’t matter in the next. Later, the idea of a narrative arc in a story started appearing. If you watch Babylon 5 today you’ll see how earlier episodes introduce things that matter later. Characters have growth and plot threads are tied up before being drawn out into new tapestries. It’s very much a job of weaving them all together.

When you present, do your sections have a flow? Do they make sense to be together? Or does it all feel like an anthology that was thrown together? Even anthologies have framing devices. Maybe you’re brining in two different groups that have different technologies that need to be covered. Rather than just throwing them out there you could create an overview of why they are important or how they work together. It’s rare that two things are completely unrelated, especially if you’re presenting them together.


Tom’s Take

If all you ever did was list out ingredients for recipes you’d be missing the important parts. They need to be combined in a certain order. Things need to go together properly. Yes, you’re going to make mistakes when you do it for the first time and you don’t understand the importance of certain things. But that learning process should help you put them together the way they need to be arranged. Take notes. Ask for feedback. And most importantly, know when it’s time to change the recipe to help you make it better the next time.

Is Disaggregation Going to Be Cord Cutting for the Enterprise?

There’s a lot of talk in the networking industry around disaggregation. The basic premise is that by decoupling the operating system from the hardware you can gain the freedom to run the devices you want from any vendor with the software that does what you want it to do. You can standardize or mix-and-match as you see fit. You gain the ability to direct the way your network works and you control how things will be going forward.

To me it sounds an awful lot like the trend of “cutting the cord” or unsubscribing from cable TV service and picking and choosing how you want to consume your content. Ten years ago the idea of getting rid of your cable TV provider was somewhat crazy. In 2021 it seems almost a given that you no long need to rely on your cable provider for entertainment. However, just like with the landscape of the post-cable cutting world, I think disaggregation is going to lead to a vastly different outcome than expected.

TNSTAAFL

Let’s get one thing out of the way up front: This idea of “freedom” when it comes to disaggregation and cord cutting is almost always about money. Yes, you want the ability to decide what software runs on your system. You don’t want to have unnecessary features or channels in your lineup. But why? I think maybe 5% of the community is worried about code quality or attack surfaces. The rest? They want to pay less for the software or hardware by unbundling the two. Instead of getting better code for their switches they’re really just chasing a lower cost per unit to run things. If that weren’t the case, why do so many of these NOS vendors run on Linux?

Yes, that feels like a bit of shot but reality speaks volumes over the pleasantries we often spout. The value of disaggregation is a smaller bottom line. Code quality can be improved over time with the proper controls in place. Hell, you could even write your own NOS given the right platform and development resources. However, people don’t want to build the perfect NOS or help vendors with the code issues. They want someone to build 90% of the perfect NOS and then sell it to them cheaply so they can run it on a cheap whitebox switch.

This is an issue that is faced by developers the world over. Look at the number of apps in the various mobile app stores that have a free entry point or are a “Freemium” business model. You don’t pay up front but as soon as you find a feature you really like it’s locked behind a subscription model. Why? Because one-time purchases don’t fund development. If everyone buys your app and then expects you to keep providing features for it and not just bug fixes, where does the investment for that development come from? Work requires resources – time or money. If you’re not getting paid for something you have to invest more time to make it work the way you want.

Vendors of disaggregated systems are finding themselves in a similar quandary. How do we charge enough for the various features we want to put into the system to be able to develop new features? The common way I see this done is to put in the most basic features that customers would want and then wait for someone to ask for something to be added. If the customer is asking for it the odds are good they’ll be willing to pay for it. You can even get them to buy your software now and sign an agreement that you’ll include the new feature in a few weeks in order to be sure your development resources aren’t wasted.

There are other ways, such as relying on single merchant silicon platforms or developing tight relationships with other vendors in the market, but ultimately it comes back to the question of resources. What are you willing to invest to make this happen? And what are you willing to accept as a cost that must be paid to get what you think you want?

The Buffet of Plenty…of Stuff You Don’t Want

The other aspect of this comparison is how the cable TV market responded to cord cutting. People started leaving cable TV for apps like Netflix and Hulu because they were cheaper than paying for a full cable subscription and had most of the content that people wanted. For the few pieces that weren’t available there were workarounds. By and large, you could find most of what you wanted in an auxiliary app when you occasionally wanted it.

So is this how things are today? Or did the market shift to the response of what customer behavior was? I think you’ll find that you’re not paying a single lump sum for content if you cut the cord for your cable provider. However, you are paying a large portion of that investment in separate apps that offer a portion of the content on-demand. And that’s why separating things is going to lead to new market dynamics.

The first behavior we saw was every media company coming up with their own app to host content. Instead of having a Disney channel on cable you now had a number of Disney apps that replicated the content channels. Later they merged into a single app with all the content. But was it all the content you wanted? Or was it all the content they owned? The drive for companies to create apps was not to offer customers a way to consume content along with their existing subscriptions. It was to provide a landing page for content you couldn’t find anywhere else.

That’s where phase two kicks in. Once you’ve created the destination, you need to make it the only place to be. That means removing content from other locations. Netflix started losing content when the creators started taking control of their own content. Soon it was necessary to create custom content to replace what was lost. Now, instead of buying a cable subscription and getting all the channels you had to sign up for five different apps, each comprising one or two of the channels you used to watch. Disney content is in the Disney app. NBC content is in another. The idea of channel surfing is gone. The back catalog of content added to the apps served more to entice people to keep their subscriptions during droughts of fresh new content.

How does this whole model break down in the enterprise? Well, going back to our earlier discussion about features being added to devices, what are you going to have to do to get new functions in your operating system? Are you going to require the vendor to write them on their schedule? Are you going to use a separate app or platform? Why should the vendor support some random feature that might not get much adoption and would take a significant amount of resources to build? Why not just make you do it yourself?

The idea is that you gain freedom and cheaper software. The hope is that you can build an enterprise network for half of what it would normally cost. The reality is that you’re going to gain less functionality and spend more time integrating things together on your own instead of just putting in a turnkey solution. And yes, there are people out there that are nodding their heads and saying they would love to do this. They want the perfect network with the perfect cheap NOS and whitebox hardware. But do you want this to be your only job for the rest of your career?

Once you build things the way you want them you become the only person that can work on them. You become the only source of support for your solution. If it’s a custom snowflake of a network you are the only person that can fix the snow issues. Traditional software and hardware may be unwieldy and difficult to troubleshoot but you can also call a support line where people have been paid to get training on how to implement and fix issues. If you built it yourself you’re the person that has to pick up the phone to fix it. Unless you want to train your team to support it too. Which takes time and money. So your savings between the two solutions are going to evaporate. And if you want the NOS vendor or the hardware supplier to support more functions to make it all easier you’re going to drive the price of the equipment up. So instead of writing one big check to the old guard you’re writing a bunch of little ones to every part of the new infrastructure you helped create.


Tom’s Take

I know it sounds like I’m not a fan of all this disaggregation stuff. In fact, I am a huge proponent of it. I just don’t buy the “freedom” excuse. My business background helps me understand the resource contention issues. My history of supporting snowflake implementations reminds me that you have to be able to turn your work over to someone else at some point in the future. Disaggregation has a lot of positive effects. You can mix and match your software and hardware and make it much easier to support for your own purposes. You no longer have to take a completed project and find workarounds to fit it to your needs. You get what you want. But don’t think you’re going to be able to get exactly what you need without some work of your own. Just like the cable cord cutting craze, you’re going to find out that you’re getting something totally different in the short term and a much different consumption model when the market shifts to the demands of the consumers. Don’t get complacent with your solutions and be ready to adapt when the suppliers force your hand.

You Down with IoT? You Better Be!

Did you see the big announcement from AWS re:Invent that Amazon has a preview of a Private 5G service? It probably got buried under the 200 other announcements that came out on so many other things so I’ll forgive you for missing it. Especially if you also managed to miss a few of the “hot takes” that mentioned how Amazon was trying to become a cellular provider. If I rolled my eyes any harder I might have caused permanent damage. Leave it to the professionals to screw up what seems to be the most cut-and-dried case of not reading the room.

Amazon doesn’t care about providing mobile service. How in the hell did we already forget about the Amazon (dumpster) Fire Phone? Amazon isn’t trying to supplant AT&T or Verizon. They are trying to provide additional connectivity for their IoT devices. It’s about as clear as it can get.

Remember all the flap about Amazon Sidewalk? How IoT devices were going to use 900 MHz to connect to each other if they had no other connectivity? Well, now it doesn’t matter because as long as one speaker or doorbell has a SIM slot for a private 5G or CBRS node then everything else can connect to it too. Who’s to say they aren’t going to start putting those slots in everything going forward? I’d be willing to bet the farm that they are. It’s cheap compared to upgrading everything to use 802.11ax radios or 6 GHz technology. And the benefits for Amazon are legion.

It’s Your Density

Have you ever designed a wireless network for a high-density deployment? Like a stadium or a lecture hall? The needs of your infrastructure look radically different compared to your home. You’re not planning for a couple of devices in a few dozen square feet. You’re thinking about dozens or even hundreds of devices in the most cramped space possible. To say that a stadium is one of the most hostile environments out there is underselling both the rabid loyalty of your average fan and the wireless airspace they’re using to post about how the other team sucks.

You know who does have a lot of experience designing high density deployments with hundreds of devices? Cellular and mobile providers. That’s because those devices were designed from the start to be more agreeable to hostile environments and have higher density deployments. Anyone that can think back to the halcyon days of 3G and how crazy it got when you went to Cisco Live and had no cell coverage in the hotel until you got to the wireless network in the convention center may disagree with me. But that exact scenario is why providers started focusing more on the number of deployed devices instead of the total throughput of the tower. It was more important in the long run to get devices connected at lower data rates than it was to pump up the wattage and get a few devices to shine at the expense of all the other ones that couldn’t get connected.

In today’s 5G landscape, it’s all about the clients. High density and good throughput. And that’s for devices with a human attached to them. Sure, we all carry a mobile phone and a laptop and maybe a tablet that are all connected to the Wi-Fi network. With IoT, the game changes significantly. Even in your consumer-focused IoT landscape you can probably think of ten devices around you right now that are connected to the network, from garage door openers to thermostats to light switches or light bulbs.

IoT at Work

In the enterprise it’s going to get crazy with industrial and operational IoT. Every building is going to have sensors packed all over the place. Temperature, humidity, occupancy, and more are going to be little tags on the walls sampling data and feeding it back to the system dashboard. Every piece of equipment you use on a factory floor is going to be connected, either by default with upgrade kits or with add-on networking gear that provides an interface to the control system. If it can talk to the Internet it’s going to be enabled to do it. And that’s going to crush your average Wi-Fi network unless you build it like a stadium.

On the other hand, private 5G and private LTE deployments are built for this scale. And because they’re lightly regulated compared to full-on provider setups you can do them easily without causing interference. As long as someone that owns a license for your frequency isn’t nearby you can just set things up and get moving. And as soon as you order the devices that have SIM slots you can plug in your cards and off you go!

I wouldn’t be shocked to see Amazon start offering a “new” lineup of enterprise-ready IoT devices with pre-installed SIMs for Amazon Private 5G service. Just buy these infrastructure devices from us and click the button on your AWS dashboard and you can have on-prem 5G. Hell, call it Network Outpost or something. Just install it and pay us and we’ll take care of the rest for you. And as soon as they get you locked in to their services they’ve got you hooked. Because if you’re already using those devices with 5G, why would you want to go through the pain on configuring them for the Wi-Fi?

This isn’t a play for consumers. Configuring a consumer-grade Wi-Fi router from a big box store is one thing. Private 5G is beyond most people, even if it’s a managed service. It also offers no advantages for Amazon. Because private 5G in the consumer space is just like hardware sales. Customers aren’t going to buy features as much as they’re shopping for the lowest sticker price. In the enterprise, Amazon can attach private 5G service to existing cloud spend and make a fortune while at the same time ensuring their IoT devices are connected at all times and possibly even streaming telemetry and collecting anonymized data, depending on how the operations contracts are written. But that’s a whole different mess of data privacy.


Tom’s Take

I’ve said it before but I’ll repeat it until we finally get the picture: IoT and 5G are now joined at the hip and will continue to grow together in the enterprise. Anyone out there that sees IoT as a hobby for home automation or sees 5G as a mere mobile phone feature will be enjoying their Betamax movies along with web apps on their mobile phones. This is bigger than the consumer space. The number of companies that are jumping into the private 5G arena should prove the smoke is hiding a fire that can signal that Gondor is calling for aid. It’s time you get on board with IoT and 5G and see that. The future isn’t a thick client with a Wi-Fi stack that you need to configure. It’s a small sensor with a SIM slot running on a private network someone else fixes for you. Are you down with that?