Technical Debt or Underperforming Investment?

In this week’s issue of the Packet Pushers Human Infrastructure newsletter, there was an excellent blog post from Kam Lasater about how talking about technical debt makes us sound silly. I recommend you read the whole thing because he brings up some very valid points about how the way the other departments of the organization perceive our issues can vary. It also breaks down debt in a very simple format that takes it away from a negative connotation and shows how debt can be a leverage instrument.

To that end, I want to make a modest proposal to help the organization understand the challenges that IT faces with older systems and integration challenges. Except we need some new branding. So, I propose we start referring to technical debt as “underperforming technical investments”.

I’d Buy That For A Dollar

Technical debt is just a clever way to refer to the series of layered challenges we face from decisions that were made to accomplish tasks. It’s a burden we carry negatively throughout the execution of our job because it adds extra time to the process. We express it as debt because it’s a price that must be paid every time we need to log in to the old accounting system to make changes or we have to wake up on the third Sunday of the month to reboot the access points because of a software bug that we can’t fix.

As Lasater points out, the financial departments look at debt differently. It’s not always negative. Commercial paper allows for purchasing power beyond cash-on-hand. Mortgages and loans allow us to buy houses and cars that enable us to do much more with our lives. Reasonable debt can be good so long as what we get out of it is a net positive. Debt doesn’t become an issue until there’s too much of it for the return we get, which eventually leads to bankruptcy.

Let’s point out some of the challenges we face with technical things that we refer to as debt. We usually use it in reference to outdated systems or slower equipment that doesn’t allow us to do our jobs as effectively as possible. Unlike a house which still provides the value of a dwelling at any age given reasonable maintenance or a car that still operates with the same maintenance, technology changes more rapidly. The original iPhone is unable to operate in a modern environment because of outdated software or inability to connect to the mobile network.

In consumer tech, the idea of technical debt doesn’t exist. Because when things are “slow” or “broken” they get discarded and people spend money to get something new. Someone that is willing to live with a shattered phone screen because they don’t want to pay to fix it might be willing to wait another two months to get a new device because it supports the latest OS updates or has a better camera. The chase for new features is usually enough of a driver to get people to upgrade.

In the enterprise, the infrastructure equipment is expensive enough that we need to recognize the value of what we installed. We can’t just swap out all the switches in three years if we spent hundreds of thousands of dollars on them. Likewise, enterprise tech vendors are more likely to provide patches and updates for this gear knowing that customers are going to hold on to it for longer periods of time. And even then, just because something is EoL doesn’t mean it’s End of Use.

However, the disconnect between these two things comes when we talk about how equipment that is functional is also causing issues with our job. Maybe it’s an older, slower switch that drops more packets than the rest of the newer gear. It’s a problem but not big enough to justify buying a new one. It could be an older wireless controller that doesn’t have the capability to run the newest code because of an older CPU. We have gear that works but it doesn’t work as well as it could. But we also don’t have enough justification to get new stuff because the old things haven’t broken completely yet.

Putting Your Money Where Their Mouth Is

As Lasater says in the above article, talking about how it’s hard to do your job because everything is old doesn’t resonate with management. No one really wants to hear a department head whining about how their job is tough because they don’t have the latest toys. Admit it, you’d be upset if your CEO told you they couldn’t answer their email because they don’t have the newest MacBook Pro. Capable means functional in every enterprise.

However, I think the challenge we face and the solution to the problem is how we frame the discussion of the challenge itself. We talk about it as a debt that we feel saddled with. It’s something we did because we had to or something someone else chose to do that we have to live with. It feels like we’re paying a tax every day over something that we don’t like. But choices need to be made and we have to work with what we have. Unless we can provide for a way to get people to understand the tradeoffs.

Hence, my proposal above. First, we stop referring to IT infrastructure gear as “technical debt”. It’s not something we bought and now have to deal with. Our accounting departments are amortizing the acquisition costs of the equipment over a period of time and writing it off of taxes against recognized revenue. There is a real dollar amount attached to every month of the lifetime of a switch or a server. That’s how finance sees your equipment. It’s not debt.

It’s a technical investment.

That’s your key to helping them understand the challenges you face. You’re not complaining about doing your job more slowly. Instead, you’re pointing out that you are investing a resource (your time) into a process that could have reduced resource requirements (again, your time and job execution time) if they would just invest in a different solution.

You’re not telling them they bought a lemon. Instead, you’re just telling them that the thing they invested in years ago isn’t performing as well as it could based on new data. If you need to break it down even further for your executive team you can just equate it to a home loan refinancing. If interest rates have dropped since you bought your home, doesn’t it make a ton of sense to refinance and take advantage of the savings? Likewise, if the speed or power of a system has increased in the past four years doesn’t it make sense to invest in something new?

When you change the discussion to focus on investment of resources instead of dealing with debt you change the way that the people will look at the return on that investment. If you just talk about debt the executives are going to easily be able to dismiss you because there is no upside to the conversation. They have to pay money to replace things so why spend it? If you talk about how much they could save on labor or how much more efficiently things could run if they made some changes or bought something new then that is harder to ignore. If their existing investment is underperforming some baseline on the stock market they’re going to want to move it right away. If the company’s stock is underperforming you can better believe some of the shareholders are going to want to move their investment too.

Here Comes the Money

One word of warning here. You can’t go in to this conversation with vague assurances that spending more money will make money. You have to come with numbers that support your case. Some of them are easy. You know what the new system will cost. You know how much time it will take to implement it. What you need to bring is the cost of what you are paying now to do things inefficiently.

If your team of 5 has to spend two extra hours a week doing something because the system is old or doesn’t work well with others then you have ten hours of lost time. You need to figure out what the value of your time is. Big note here: Don’t just divide your yearly salary by 2000 and take the averages. That’s what you’re getting paid. That’s the debt that the company incurs to keep you on staff. Instead, find out what your rate is. If you are billed at a higher rate than your earnings, which is almost a guarantee, use that number instead. If you’re an engineer for a reseller your hourly billing rate is easy to find. If you work in corporate IT the rate is harder to figure out but if you need a rule of thumb just figure out what your hourly pay rate is (yearly salary divided by 2000 hours per year) and double it. That’s the kind of return that any company would love to have on an investment. If it’s costing the company that much to do something inefficiently then you can make your case to get it fixed or replaced.

Lastly, make sure to record the changes and the results if you manage to make it happen. It’s easy to chart when you take more time to do something to prove a point. But it’s easy to forget to do it when you have what you want. Eventually, someone that manages investments needs to know if it paid off. If there’s a simple number like a stock price it’s cut-and-dried. If there’s more to the calculation you need to do the work to prove why it’s better now. And you want to keep a running total so you can provide it on demand when asked. Just tell them you have more time to track those kinds of things now that they made a wiser investment.


Tom’s Take

No one likes debt. Even the best debt is still an obligation to pay back to someone. However, investments are something positive. Yes, they still require resources to be exchanged to receive a good or service. However, changing the terminology changes the perception of the result. Debts are incurred and paid back begrudgingly. Investments grow and provide additional resources throughout their lives. And, importantly, when an investment isn’t paying off the way to fix it is to reinvest. It’s a clever trick but one that should work much better than just whining about drowning in debt.

Who Wants to Be Supported Forever?

I saw an interesting thread today on Reddit talking about using networking equipment past the End of Life. It’s a fun read that talks about why someone would want to do something like this and how you might find yourself in some trouble depending on your company policies and such. But I wanted to touch on something that I think we skip over when we get here. What does the life of the equipment really mean?

It’s a Kind of Magic

As someone that uses equipment of all kinds, the lifetime of that equipment means something different for me than it does for vendors. When I think of how long something lasts I think of it in terms of how long I can use it until it is unable to be repaired any further. A great example of this is a car. All of my life I have driven older used cars that I continue to fix over and over until they have a very high mileage or my needs change and I must buy something different.

My vehicles don’t have a warranty or any kind of support, necessarily. If I need something fixed I either fix it myself or I take it to a mechanic to have it fixed and pay the costs associated with that. I’m not the kind of person that will get rid of a car just because the warranty has run out and it’s no longer under support.

The market of replacement parts and mechanics is made for people like me. Why buy something new when you can buy the parts to repair what you have? Sadly, that market exists for automobiles but not for IT gear. It’s hard to pull a switch apart and resolder capacitors when you feel like it. Most of the time you want to replace something that has failed and is end-of-life, you need to have a spare of the same kind of equipment sitting on the shelf. You can put the configuration files you need on it and put it in place and just keep going. The old equipment is tossed aside or recycled in some way.

Infrastructure gear will work well past the End of Support (EoS) or End of Life (EoL) dates. It will keep passing packets until something breaks, either the hardware in the ports or the CPU that drives it. Maybe it’s even a power supply. Usually it’s not the end of the life of the hardware that causes the issues. Instead, it’s the software concerns.

Supported Rhapsody

I’d venture a guess that most people reading this blog have some kind of mobile device sitting in their junk drawer or equipment box that works perfectly fine yet isn’t used because the current version of software doesn’t support that model. Software is a bigger determining factor the end of life of IT equipment than anything else.

Consumer gear needs to be fast to keep users happy. Phones, laptops, and other end user devices have to keep up with the desires of the people running applications and playing games and recording video. These devices iterate quickly and fall out of support just as fast. Your old iPhone or Galaxy probably won’t run the latest code which has a feature you really want to use so you move on to something newer.

Infrastructure gear doesn’t have software go out of fashion quite as quickly as a mobile phone but it does have another issue that causes grief: security patches. You may not be chasing a new feature in a software release but you’re either looking to patch a bug that is affecting performance or patch a security hole that has the potential to cause issues for your organization.

Security patches for old hardware are a constant game of cat-and-mouse. Whether it’s an old operating system running important hardware terminals or the need to keep an old terminal running because the application software doesn’t work on anything newer IT departments are often saddled with equipment that’s out of date and insecure. Do you keep running it, hoping that there will still be patches for it? Or do you try to move to something newer and hope that you can make things work? Anyone that is trying to use software that requires versions of Microsoft Internet Explorer knows that pain can last for years.

The real issue here doesn’t have anything to do with support of the equipment or the operating system. Instead, it has everything to do with the support that you can provide for it. A mechanic isn’t magical. They learn how to get parts for cars and fix the problems as they arise. They become the support structure instead of the dealership that offers the warranty. Likewise, as an IT pro you become the support structure for an old switch or old operating system. If you can’t download patches for it you either need to provide a workaround for issues or find a way to create solutions to address it. It’s not end of support as much as it’s end of someone else’s support.


Tom’s Take

I use things until they don’t work. Then I fix them and use them more. It’s the way I was raised. I have a hard time getting new things simply because they’re supported or a little bit faster. I might upgrade my phone to get a new feature but you can bet I’m going to find a way to use my old one longer, either by giving it to one of my kids or using it in a novel way. I hate getting rid of technology that has some use left in it. End of Life for me really does mean the moment when it stops sending bits or storing data. As long as there is no other conflict in the system you can count on me extending the life of a device long past the date that someone says they don’t want to deal with it any more.

The Demise of G-Suite

In case you missed it this week, Google is killing off the free edition of Google Apps/G-Suite/Workspace. The short version is that you need to convert to a paid plan by May 1, 2022. If you don’t you’re going to lose everything in July. The initial offering of the free tier was back in 2006 and the free plan hasn’t been available since 2012. I suppose a decade is a long time to enjoy custom email but I’m still a bit miffed at the decision.

Value Added, Value Lost

It’s pretty easy to see that the free version of Workspace was designed to encourage people to use it and then upgrade to a paid account to gain more features. As time wore on Google realized that people were taking advantage of having a full suite of 50 accounts and never moving, which is why 2012 was the original cutoff date. Now there has been some other change that has forced their hand into dropping the plan entirely.

I won’t speculate about what’s happening because I’m sure it’s complex and tied to ad revenue and privacy restrictions that people are implementing that is reducing the value of the data Google has been mining for years. However, I can tell you that the value of what they’re offering with their entry-level business plan isn’t as valuable as they might think.

The cheapest Google Workspace plan available is $6 per user per month. It covers the email and custom domain that was the biggest attraction. It also has a whole host of other features:

  • Google Meet, which I never use when Zoom/Webex/Teams exist
  • Google Drive, which is somewhat appealing except for when no one wants to use Docs or Sheets and Dropbox is practically a standard and still free
  • Chat, which makes me laugh because it’s probably the next thing to get killed off in favor of some other messaging platform that will get abandoned in six months

Essentially, Google is hoping to convince me to pay for their service that has been free this entire time by giving me things I don’t use now and probably won’t use in the future? Not exactly a good selling model.

Model Citizens

I’ve heard there is a plan for trying to give loyal customers a discount for the first year of service to ease the transition but that’s not going to cut it for most. Based on the comments I’ve seen most people are upset that they have purchases from Google tied to an account they can’t transfer away from as well as the possibility that whatever happens next is going to be shut down anyway. I mean, Killed by Google is starting to look like a massive graveyard at this point.

I’m willing to concede at this point that the free tier of Google Workspace is gone and won’t be coming back. What I’m not ready to give in on is the model that you’re forcing me to pay for and use other services because you have a target revenue number to hit and you keep throwing useless stuff in to make it seem valuable. You’re not transitioning us to a new model. You’re ramming the existing one down our throats because you need users for those other services that are paying.

Want to extend some goodwill to the community? Offer us a solution that gives us what we want for a reasonable pricing model? How about email without video chat and Drive for $3 per month per user? How about allowing me to cut out the junk and reduce my spend. How about giving me something with other value based on how I use your service and not how you think I should be using it?


Tom’s Take

I realize I’m just tilting at windmills in this whole mess but It’s frustrating. I’m totally prepared to never see a resolution to this issue because Google has decided it’s time to kill it off. Yes, I got a decade of free email hosting out of the deal. I got a lot of value for what I invested. I even realize that Google can’t keep things free forever. I just wish there was a way for me to pay them for what I want and not have to pay more for things that are useless to me. Technology marches on and new models always supplant old ones. The only constant is change. But change is something we should be able to process and accept. Not have it forced upon us to ensure someone is using Google Meet.

Holiday Networking Thoughts from 2021

It’s the Christmas break for 2021, which means lots of time spent doing very little work-related stuff. I’m currently putting together a Lego set, playing Metroid Dread and working on beating Ocarina of Time again.

As I waited for updates to download on Christmas morning I remembered how many packets must be flying across the wire to update software and operating systems for consoles. Even having done a few of the updates the night before I could see the traffic to those servers started to get a bit congested. It’s like Black Friday but for the latest patches to keep your games running. Add in the content that needs to be installed now in order to make that game disc work, or the download-only consoles for sale, and you can see that network engineers aren’t going to be a dying profession any time soon.

I’m a bit jaded because I come from a time when you didn’t need to be constantly connected to use software or need to download an update every few days. Heck, some of the bugs in Ocarina of Time have been there for over twenty years because those cartridges are not designed to be patched, having been created before a time when you could barely get online with a modem, let alone wirelessly connect a console.

I also am happy that upgrading devices in the house means fewer and fewer older units performing poorly on the wireless network. As more devices require me to connect them to the network for updates or app connectivity, I’m reminded that things like the Xbox 360 need low data rates enabled to work properly and that makes me sad. But I also can’t turn them off for fear that nothing will work and my children will scream. I don’t think spending a ton of money to get rid of an 802.11b client is really that big of a deal but I’m happy to see them go when I get the chance.

Likewise, I’m going to need to upgrade my APs a bit now that I have clients that can actually use 802.11ax (Wi-Fi 6). Even the older clients will get a performance boost. So It’s a matter of catching a good AP on sale and getting it done. Since I don’t use big box APs I just have to look a bit harder.


Tom’s Take

Make sure you give a shoutout to your friendly neighborhood network engineer for all their hard work making sure the services we’re currently consuming stayed up while the skeleton crew was carrying the pager this weekend. We’ve seen a lot of services crash on Christmas morning in recent years because of unexpected load. Also, give yourselves a hand for keeping your own network up long enough to download the latest DLC for a game or ensure that your new smart appliance can talk to the fancy app you need to use to control it. Let’s make it through the rest of the year with the change freeze intact and start 2022 off on the right foot with no outages.

A Recipe for Presentation Success

When I was a kid, I loved to help my mother bake. My absolute favorite thing to make was a pecan pie. I made sure I was always the one that got to do the work to fix it during the holidays. When I was first starting out I made sure I followed the recipe to the letter. I mixed everything in the order that it was listed. One of the first times I made the pie I melted the butter and poured it into the mixture which also had an egg. To my horror I saw the egg starting to cook and scramble in the bowl due to the hot butter. When I asked my mom she chuckled and said, “Now you get to learn about why the recipe isn’t always right.”

Throughout my career in IT and in presentations, I’ve also had to learn about why even if the recipe for success is written down properly there are other things you need to take into account before you put everything together. Just like tempering a mixture or properly creaming butter and sugar together, you may find that you need to do some things in a different order to make it all work correctly.

Step by Out of Step

As above, sometimes you need to know how things are going to interact so you do them in the right order. If you pour hot liquid on eggs you’re going to cook them. If you do a demo of your product without providing context for what’s happening you’re likely going to lose your audience. You need to set things up in the proper order for it all to make sense.

Likewise if you spend all your time talking about a problem that needs to be solved without telling your listeners that you solve the problem you’re going to have them focused on what’s wrong, not on how you fix it. Do you want them thinking about how you get a flat tire when you run over a nail? Or do you want them to buy your tires that don’t go flat when you run over sharp objects? It’s important to sell your product, not the problem.

It’s also important to know when to do those things out of order. Does your demo do something magical or amazing with a common issue? It might be more impactful to have your audience witness what happens before explaining how it works behind the scenes. It’s almost like a magician revealing their trick. Wow them with the result before you pull back the curtain to show them how it’s done.

The feel for how to do this varies from presentation to presentation. Are you talking to an audience that doesn’t understand the topic at all? You need to start with a lead-in or some other kind of level setting so no one gets lost. Are they experienced and understand the basics? You should be able to jump in at a higher level and show off a few things before going into detail. You have to understand whether or not you’re taking to a group of neophytes or a crowd of wizened veterans.

A counterpoint to this is the crowd of people that might be funding your project or startup. If they’re a person that gets pitched daily about “the problem” or they have a keen understanding of the market, what exactly are you educating them about when you open with a discussion of the issues? Are you telling them that you know what they are? Or are you just trying to set a hook? Might be worth explaining what you do first and then showing how you attack the problem directly.

Weaving a Story

The other thing that I see being an issue in presentations is the lack of a story. A recipe tells a story if you listen. Things have relationships. Liquids should be mixed together. Dry ingredients should be combined beforehand. Certain pieces should be put on last. If you put the frosting on a cake before you put it in the oven you’re going to be disappointed. It’s all part of the story that links the parts together.

Likewise, your presentation or lesson should flow. There should be a theme. It should make sense if you watch it. You can have individual pieces but if you tie it all together you’re going to have a better time of helping people understand it.

When I was growing up, TV shows didn’t tell longer stories. Episodes of the Addams Family or Gilligan’s Island stood alone. What happened in the first season didn’t matter in the next. Later, the idea of a narrative arc in a story started appearing. If you watch Babylon 5 today you’ll see how earlier episodes introduce things that matter later. Characters have growth and plot threads are tied up before being drawn out into new tapestries. It’s very much a job of weaving them all together.

When you present, do your sections have a flow? Do they make sense to be together? Or does it all feel like an anthology that was thrown together? Even anthologies have framing devices. Maybe you’re brining in two different groups that have different technologies that need to be covered. Rather than just throwing them out there you could create an overview of why they are important or how they work together. It’s rare that two things are completely unrelated, especially if you’re presenting them together.


Tom’s Take

If all you ever did was list out ingredients for recipes you’d be missing the important parts. They need to be combined in a certain order. Things need to go together properly. Yes, you’re going to make mistakes when you do it for the first time and you don’t understand the importance of certain things. But that learning process should help you put them together the way they need to be arranged. Take notes. Ask for feedback. And most importantly, know when it’s time to change the recipe to help you make it better the next time.

Is Disaggregation Going to Be Cord Cutting for the Enterprise?

There’s a lot of talk in the networking industry around disaggregation. The basic premise is that by decoupling the operating system from the hardware you can gain the freedom to run the devices you want from any vendor with the software that does what you want it to do. You can standardize or mix-and-match as you see fit. You gain the ability to direct the way your network works and you control how things will be going forward.

To me it sounds an awful lot like the trend of “cutting the cord” or unsubscribing from cable TV service and picking and choosing how you want to consume your content. Ten years ago the idea of getting rid of your cable TV provider was somewhat crazy. In 2021 it seems almost a given that you no long need to rely on your cable provider for entertainment. However, just like with the landscape of the post-cable cutting world, I think disaggregation is going to lead to a vastly different outcome than expected.

TNSTAAFL

Let’s get one thing out of the way up front: This idea of “freedom” when it comes to disaggregation and cord cutting is almost always about money. Yes, you want the ability to decide what software runs on your system. You don’t want to have unnecessary features or channels in your lineup. But why? I think maybe 5% of the community is worried about code quality or attack surfaces. The rest? They want to pay less for the software or hardware by unbundling the two. Instead of getting better code for their switches they’re really just chasing a lower cost per unit to run things. If that weren’t the case, why do so many of these NOS vendors run on Linux?

Yes, that feels like a bit of shot but reality speaks volumes over the pleasantries we often spout. The value of disaggregation is a smaller bottom line. Code quality can be improved over time with the proper controls in place. Hell, you could even write your own NOS given the right platform and development resources. However, people don’t want to build the perfect NOS or help vendors with the code issues. They want someone to build 90% of the perfect NOS and then sell it to them cheaply so they can run it on a cheap whitebox switch.

This is an issue that is faced by developers the world over. Look at the number of apps in the various mobile app stores that have a free entry point or are a “Freemium” business model. You don’t pay up front but as soon as you find a feature you really like it’s locked behind a subscription model. Why? Because one-time purchases don’t fund development. If everyone buys your app and then expects you to keep providing features for it and not just bug fixes, where does the investment for that development come from? Work requires resources – time or money. If you’re not getting paid for something you have to invest more time to make it work the way you want.

Vendors of disaggregated systems are finding themselves in a similar quandary. How do we charge enough for the various features we want to put into the system to be able to develop new features? The common way I see this done is to put in the most basic features that customers would want and then wait for someone to ask for something to be added. If the customer is asking for it the odds are good they’ll be willing to pay for it. You can even get them to buy your software now and sign an agreement that you’ll include the new feature in a few weeks in order to be sure your development resources aren’t wasted.

There are other ways, such as relying on single merchant silicon platforms or developing tight relationships with other vendors in the market, but ultimately it comes back to the question of resources. What are you willing to invest to make this happen? And what are you willing to accept as a cost that must be paid to get what you think you want?

The Buffet of Plenty…of Stuff You Don’t Want

The other aspect of this comparison is how the cable TV market responded to cord cutting. People started leaving cable TV for apps like Netflix and Hulu because they were cheaper than paying for a full cable subscription and had most of the content that people wanted. For the few pieces that weren’t available there were workarounds. By and large, you could find most of what you wanted in an auxiliary app when you occasionally wanted it.

So is this how things are today? Or did the market shift to the response of what customer behavior was? I think you’ll find that you’re not paying a single lump sum for content if you cut the cord for your cable provider. However, you are paying a large portion of that investment in separate apps that offer a portion of the content on-demand. And that’s why separating things is going to lead to new market dynamics.

The first behavior we saw was every media company coming up with their own app to host content. Instead of having a Disney channel on cable you now had a number of Disney apps that replicated the content channels. Later they merged into a single app with all the content. But was it all the content you wanted? Or was it all the content they owned? The drive for companies to create apps was not to offer customers a way to consume content along with their existing subscriptions. It was to provide a landing page for content you couldn’t find anywhere else.

That’s where phase two kicks in. Once you’ve created the destination, you need to make it the only place to be. That means removing content from other locations. Netflix started losing content when the creators started taking control of their own content. Soon it was necessary to create custom content to replace what was lost. Now, instead of buying a cable subscription and getting all the channels you had to sign up for five different apps, each comprising one or two of the channels you used to watch. Disney content is in the Disney app. NBC content is in another. The idea of channel surfing is gone. The back catalog of content added to the apps served more to entice people to keep their subscriptions during droughts of fresh new content.

How does this whole model break down in the enterprise? Well, going back to our earlier discussion about features being added to devices, what are you going to have to do to get new functions in your operating system? Are you going to require the vendor to write them on their schedule? Are you going to use a separate app or platform? Why should the vendor support some random feature that might not get much adoption and would take a significant amount of resources to build? Why not just make you do it yourself?

The idea is that you gain freedom and cheaper software. The hope is that you can build an enterprise network for half of what it would normally cost. The reality is that you’re going to gain less functionality and spend more time integrating things together on your own instead of just putting in a turnkey solution. And yes, there are people out there that are nodding their heads and saying they would love to do this. They want the perfect network with the perfect cheap NOS and whitebox hardware. But do you want this to be your only job for the rest of your career?

Once you build things the way you want them you become the only person that can work on them. You become the only source of support for your solution. If it’s a custom snowflake of a network you are the only person that can fix the snow issues. Traditional software and hardware may be unwieldy and difficult to troubleshoot but you can also call a support line where people have been paid to get training on how to implement and fix issues. If you built it yourself you’re the person that has to pick up the phone to fix it. Unless you want to train your team to support it too. Which takes time and money. So your savings between the two solutions are going to evaporate. And if you want the NOS vendor or the hardware supplier to support more functions to make it all easier you’re going to drive the price of the equipment up. So instead of writing one big check to the old guard you’re writing a bunch of little ones to every part of the new infrastructure you helped create.


Tom’s Take

I know it sounds like I’m not a fan of all this disaggregation stuff. In fact, I am a huge proponent of it. I just don’t buy the “freedom” excuse. My business background helps me understand the resource contention issues. My history of supporting snowflake implementations reminds me that you have to be able to turn your work over to someone else at some point in the future. Disaggregation has a lot of positive effects. You can mix and match your software and hardware and make it much easier to support for your own purposes. You no longer have to take a completed project and find workarounds to fit it to your needs. You get what you want. But don’t think you’re going to be able to get exactly what you need without some work of your own. Just like the cable cord cutting craze, you’re going to find out that you’re getting something totally different in the short term and a much different consumption model when the market shifts to the demands of the consumers. Don’t get complacent with your solutions and be ready to adapt when the suppliers force your hand.

You Down with IoT? You Better Be!

Did you see the big announcement from AWS re:Invent that Amazon has a preview of a Private 5G service? It probably got buried under the 200 other announcements that came out on so many other things so I’ll forgive you for missing it. Especially if you also managed to miss a few of the “hot takes” that mentioned how Amazon was trying to become a cellular provider. If I rolled my eyes any harder I might have caused permanent damage. Leave it to the professionals to screw up what seems to be the most cut-and-dried case of not reading the room.

Amazon doesn’t care about providing mobile service. How in the hell did we already forget about the Amazon (dumpster) Fire Phone? Amazon isn’t trying to supplant AT&T or Verizon. They are trying to provide additional connectivity for their IoT devices. It’s about as clear as it can get.

Remember all the flap about Amazon Sidewalk? How IoT devices were going to use 900 MHz to connect to each other if they had no other connectivity? Well, now it doesn’t matter because as long as one speaker or doorbell has a SIM slot for a private 5G or CBRS node then everything else can connect to it too. Who’s to say they aren’t going to start putting those slots in everything going forward? I’d be willing to bet the farm that they are. It’s cheap compared to upgrading everything to use 802.11ax radios or 6 GHz technology. And the benefits for Amazon are legion.

It’s Your Density

Have you ever designed a wireless network for a high-density deployment? Like a stadium or a lecture hall? The needs of your infrastructure look radically different compared to your home. You’re not planning for a couple of devices in a few dozen square feet. You’re thinking about dozens or even hundreds of devices in the most cramped space possible. To say that a stadium is one of the most hostile environments out there is underselling both the rabid loyalty of your average fan and the wireless airspace they’re using to post about how the other team sucks.

You know who does have a lot of experience designing high density deployments with hundreds of devices? Cellular and mobile providers. That’s because those devices were designed from the start to be more agreeable to hostile environments and have higher density deployments. Anyone that can think back to the halcyon days of 3G and how crazy it got when you went to Cisco Live and had no cell coverage in the hotel until you got to the wireless network in the convention center may disagree with me. But that exact scenario is why providers started focusing more on the number of deployed devices instead of the total throughput of the tower. It was more important in the long run to get devices connected at lower data rates than it was to pump up the wattage and get a few devices to shine at the expense of all the other ones that couldn’t get connected.

In today’s 5G landscape, it’s all about the clients. High density and good throughput. And that’s for devices with a human attached to them. Sure, we all carry a mobile phone and a laptop and maybe a tablet that are all connected to the Wi-Fi network. With IoT, the game changes significantly. Even in your consumer-focused IoT landscape you can probably think of ten devices around you right now that are connected to the network, from garage door openers to thermostats to light switches or light bulbs.

IoT at Work

In the enterprise it’s going to get crazy with industrial and operational IoT. Every building is going to have sensors packed all over the place. Temperature, humidity, occupancy, and more are going to be little tags on the walls sampling data and feeding it back to the system dashboard. Every piece of equipment you use on a factory floor is going to be connected, either by default with upgrade kits or with add-on networking gear that provides an interface to the control system. If it can talk to the Internet it’s going to be enabled to do it. And that’s going to crush your average Wi-Fi network unless you build it like a stadium.

On the other hand, private 5G and private LTE deployments are built for this scale. And because they’re lightly regulated compared to full-on provider setups you can do them easily without causing interference. As long as someone that owns a license for your frequency isn’t nearby you can just set things up and get moving. And as soon as you order the devices that have SIM slots you can plug in your cards and off you go!

I wouldn’t be shocked to see Amazon start offering a “new” lineup of enterprise-ready IoT devices with pre-installed SIMs for Amazon Private 5G service. Just buy these infrastructure devices from us and click the button on your AWS dashboard and you can have on-prem 5G. Hell, call it Network Outpost or something. Just install it and pay us and we’ll take care of the rest for you. And as soon as they get you locked in to their services they’ve got you hooked. Because if you’re already using those devices with 5G, why would you want to go through the pain on configuring them for the Wi-Fi?

This isn’t a play for consumers. Configuring a consumer-grade Wi-Fi router from a big box store is one thing. Private 5G is beyond most people, even if it’s a managed service. It also offers no advantages for Amazon. Because private 5G in the consumer space is just like hardware sales. Customers aren’t going to buy features as much as they’re shopping for the lowest sticker price. In the enterprise, Amazon can attach private 5G service to existing cloud spend and make a fortune while at the same time ensuring their IoT devices are connected at all times and possibly even streaming telemetry and collecting anonymized data, depending on how the operations contracts are written. But that’s a whole different mess of data privacy.


Tom’s Take

I’ve said it before but I’ll repeat it until we finally get the picture: IoT and 5G are now joined at the hip and will continue to grow together in the enterprise. Anyone out there that sees IoT as a hobby for home automation or sees 5G as a mere mobile phone feature will be enjoying their Betamax movies along with web apps on their mobile phones. This is bigger than the consumer space. The number of companies that are jumping into the private 5G arena should prove the smoke is hiding a fire that can signal that Gondor is calling for aid. It’s time you get on board with IoT and 5G and see that. The future isn’t a thick client with a Wi-Fi stack that you need to configure. It’s a small sensor with a SIM slot running on a private network someone else fixes for you. Are you down with that?

A Gift Guide for Sanity In Your Home IT Life

If you’re reading my blog you’re probably the designated IT person for your family or immediate friend group. Just like doctors that get called for every little scrape or plumbers that get the nod when something isn’t draining over the holidays, you are the one that gets an email or a text message when something pops up that isn’t “right” or has a weird error message. These kinds of engagements are hard because you can’t just walk away from them and you’re likely not getting paid. So how can you be the Designated Computer Friend and still keep your sanity this holiday season?

The answer, dear reader, is gifts. If you’re struggling to find something to give your friends that says “I like you but I also want to reduce the number of times that you call me about your computer problems” then you should definitely read on for more info! Note that I’m not going to fill this post will affiliate links or plug products that have sponsored anything. Instead, I’m going to just share the classes or types of devices that I think are the best way to get control of things.

Step 1: Infrastructure Upgrades

When you go visit your parents for Thanksgiving or some other holiday check in, are they still running the same wireless network they got when they got their high-speed Internet? Is their Wi-Fi SSID still the default with the password printed on the side of the router/modem combo? Then you’re going to want to upgrade their experience to keep your sanity for the next few holidays.

The first thing you need to do it get control of their wireless setup. You need to get some form of wireless access point that wasn’t manufactured in the early part of the century. Most of the models on the market have Wi-Fi 6 support now. You don’t need to go crazy with a Wi-Fi 6E model for your loved ones right now because none of their devices will support it. You just need something more modern with a user interface that wasn’t written to look like Windows 3.1.

You also need to see about an access point that is controlled via a cloud console. If you’re the IT person in the group you probably already use some form control for your home equipment. You don’t need a full Meraki or Juniper Mist setup to lighten your load. That is, unless you already have one of those dashboards set up and you have spare capacity. Otherwise you could look at something like Ubiquiti as a middle ground.

Why a cloud controller AP? Because then you can log in and fix things or diagnose issues without needing to spend time talking to less technical users. You can find out if they have an unstable Internet connection or change SSID passwords at the drop of a hat. You can even set up notifications for those remote devices to let you know when a problem happens so you can be ready and waiting for the call. And you can keep tabs on necessary upgrades and such so you aren’t fielding calls when the next major exploit comes out and your parents call you asking if they’re going to get infected by this virus. You can just tell them they’re up-to-date and good to go. The other advantage of this method is that when you upgrade your own equipment at home you can just waterfall the old functional gear down to them and give them a “new to you” upgrade that they’ll appreciate.

Step 2: Device Upgrades

My dad was notorious for using everything long past the point of needing to be retired. It’s the way he was raised. If there’s a hole you patch it. If it breaks you fix it. If that fix doesn’t work you wrap it in duct tape and use it until it crumbles to dust. While that works for the majority of things out there it does cause issues with technology far too often.

He had a iPad that he loved. He didn’t use it all day, every day but he did use it frequently enough to say that it was his primary computing device. It was a fourth-generation device, so it fell out of fashion a few years ago. When he would call me and ask me questions about why it was behaving a certain way or why he couldn’t download some new app from the App Store I would always remind him that he had an older device that wasn’t fast enough or new enough to run the latest programs or even operating software. This would usually elicit a grumble or two and then we would move on.

If you’re the Designated IT Person and you spend half your time trying to figure out what versions of OS and software are running on a device, do yourself a favor and invest in a new device for your users just to ease the headaches. If they use a tablet as their primary computing device, which many people today do, then just buy a new one and help them migrate all the data across to the new one while you’re eating turkey or opening presents.

Being on later hardware ensures that the operating system is the latest version with all the patches for security that are needed to keep your users safe. It also means you’re not trying to figure out what the last supported version of the software was that works with the rest of the things. I’ve played this game trying to get an Apple Watch to connect to an older phone with mismatched software as well as trying to get support for newer wireless security on older laptops with very little capability to do much more than WPA1. The amount of hours I burned trying to make the old junk work with the new stuff would have been better served just buying a new version of the same old thing and getting all their software moved over. Problems seem to just disappear when you are running on something that was manufactured within the last five years.

Step 3: Help Them Remember

This is probably my biggest request: Forgotten passwords. Either it’s the forgotten Apple ID or maybe the wireless network password. My parents and in-laws forget the passwords they need to log into things all the time. I finally broke down and taught them how to use a password management tool a few years ago and it made all the difference in the world. Now, instead of them having to remember what their password was for a shopping site they can just set it to automatically fill everything in. And since they only need to remember the master password for their app they don’t have to change it.

Better yet, most of these apps have a secure section for notes. So all those other important non-password things that seem to come up all the time are great to put in here. Social Security Numbers, bank account numbers, and so much more can be put in one central location and made easy to access. The best part? If you make it a shared vault you can request access to help them out when they forget how to get in. Or you can be designated as a trusted party that can access the account in the event of a tragedy. Getting your loved ones used to using password vaults now makes it much easier to have them storing important info there in case something happens down the road that requires you to jump in without their interaction. Trust me on this.


Tom’s Take

Your loved ones don’t need knick knacks and useless junk. If you want to show them you love them, give them the gift of not having to call you every couple of days because they can’t remember the wireless password or because they keep getting this error that says their app isn’t support on this device. Invest in your sanity and their happiness by giving them something that works and that has the ability for you to help manage it from the background. If you can make it stable and useful and magically work before they call you with a problem you’re going to find yourself a happier person in the years to come.

Getting In Front of Future Regret

Yesterday I sat in on the keynote from Commvault Connections21 and participated in a live blog of it on Gestalt IT. There was a lot of interesting info around security, especially related to how backup and disaster recovery companies are trying to add value to the growing ransomware issue in global commerce. One thing that I did take away from the conversation wasn’t specifically related to security though and I wanted to dive into a bit more.

Reza Morakabati, CIO for Commvault, was asked what he thought teams needed to do to advance their data strategy. And his response was very insightful:

Ask your team to imagine waking up to hear some major incident has happened. What would their biggest regret be? Now, go to work tomorrow and fix it.

It’s a short, sweet, and powerful sentence. Technology professionals are usually focused on implementing new things to improve productivity or introduce new features to users and customers. We focus on moving fast and making people happy. Security is often seen as running counter to this ideal. Security wants to keep people safe and secure. It’s not unlike the parents that hold on to their child’s bicycle after the training wheels come off just to make sure the kids are safe. The kids want to ride and be free. The parents aren’t quite sure how secure they’re going to be just yet.

Regret Storming

Thought exercises make for entertaining ways to scare yourself to death some days. If you spend too much time thinking about all the ways that things can go wrong you’re going to spend far too much energy focused on the negative aspects of your work. However, you do need to occasionally open yourself up to the likelihood that things are going to go wrong at some point.

For the thought exercise above, it’s not crucial to think about how they could go wrong. It’s more important to think about what could be the worst thing that could happen as a result of those bad things and how much you’ll regret it. You need to identify those areas and try to figure out how they can be mitigated.

Let me give you a specific example from my area. In May 2013 a massive tornado ripped through Moore, OK just north of where I live. It was a tragic event that had loss of life. People were displaced and homes and businesses were destroyed. One of the places that was damaged severely was the Moore Public Schools administration building. In the aftermath of trying to clean up the debris and find survivors, one of my friends that worked for an IT vendor told me he spent hours helping sift through the rubble of the building looking for hard disk drives for the district’s servers. Why? Because the tornado had struck just before the payroll for the district’s teachers and staff was due to be run. Without the drives they couldn’t run payroll or print paychecks for those employees. With an even greater need to have funds to pay for food or start repairs on their homes you can imagine that not getting paid was going to be a big deal for those educators and staff.

There are a lot of regrets that came out of the May 2013 tornado. Loss of life and loss of property are always at the top of the list. The psychological damage of enduring something like that is also a huge impact. But for the school district one of the biggest regrets they faced was not having a contingency plan for what to do about paying their employees to help them deal with the disaster. It sounds small in comparison to the millions of dollars of damage that happened but it also represents something important that can be controlled. The school system can’t upgrade the warning system or build businesses that can withstand the most powerful storms imaginable. But they can fix their systems to prevent teachers from going without resources in the event of an emergency.

In this case, the regret is not being able to pay teachers if the district data center goes down. How could we fix that regret today if we had imagined it beforehand? We could have migrated the data center to the cloud so no one weather event could take it out. Likewise, we could have moved to a service that provides payroll entry and check printing that could be accessed from anywhere. We could also have encouraged our teachers and employees to use direct deposit functions to reduce the need to physically print checks. Technology today provides us with a number of solutions to the regret we face. We can put together plans to implement any one of them quickly. We just need to identify the problem and build a resolution for it.

Building Your Future

It’s not easy to foresee every possible outcome. Nor should it be. But if you focus on the feelings those unknown outcomes could bring you’ll have a much better sense for what’s important to protect and how to go about doing it. Are you worried your customer data is going to be stolen and shared on the Internet? Then you need to focus your efforts on protecting it. Are you concerned your AWS bill is going to skyrocket if someone steals your credentials and starts borrowing your resource pool? Then you need to have governance in place to prevent unauthorized users from doing that thing.

You don’t have to have a solution for every possible regret. You may even find that some of the things you thought you might end up regretting are actually pretty mild. If you’re not concerned about what would happen to your testing environment because you can just clone it from a repository then you can put that to bed and not worry about it any longer. Likewise, you may discover some regrets you didn’t anticipate. For example, if you’re using Active Directory credentials to back up your server data, you need to make sure you’re backing up Active Directory as well. You’re going to find yourself infuriated if you have the data you need to get back to business but it’s locked behind cryptographic locks that you can’t open because someone forgot to back up a domain controller.


Tom’s Take

I’ve been told that I’m somewhat negative because I’m always worried about what could go wrong with a project or an event. It’s not that I’m a pessimist as much as I’ve got a track record for seeing how things can go off the rails. Thanks to Commvault I’m going to spend more time thinking of my regrets and trying to plan for them to be mitigated ahead of time so all the possible ways things could fail won’t consume my thoughts. I don’t have to have a plan for everything. I just need to get in front of the regrets before I feel them for real.

Chip Shortages Aren’t Sweet for Networking

Have you tried to order networking gear recently? You’re probably cursing because the lead times on most everything are getting long. It’s not uncommon to see lead times on wireless access points or switch gear reaching 180 days or more. Reports from the Internet say that some people are still waiting to get things they ordered this spring. The prospect of rapid delivery of equipment is fading like the summer sun.

Why are we here? What happened? And can we do anything about it?

Fewer Chips, More Air

The pandemic has obviously had the biggest impact for a number of reasons. When a fabrication facility shuts down it doesn’t just ramp back up. Even when all the workers are healthy and the city where it is located is open for business it takes weeks to bring everything back online to full capacity. Just like any manufacturing facility you can’t just snap your fingers and get back to churning out the widgets.

The pandemic has also strained supply chains around the world. Even if the fabs had stayed open this entire time you’d be looking at a shortage of materials to make the equipment. Global supply chains were running extremely lean in 2019 and exposing one aspect of them has created a cascade effect that has caused stress everywhere. The lack of toilet paper or lunchmeat in your grocery store shows that. Even when the supply is available the ability to deliver it is impacted.

The supply chain problem also belies the issue on the other side of the shipping container. Even if the fabs had enough chips to sell to anyone that wanted them it’s hard to get those parts delivered to the companies that make things. If this were simply an issue of a company not getting the materials it needed to make a widget in a reasonable time there wouldn’t be as much issue. But because these companies make things that other companies use to make things the hiccups in the chain are exacerbated. If TSMC is delayed by a month getting a run of chips out, that month-long delay only increases for those down the line.

We’ve got issues getting facilities back online. We’ve got supply chains causing problems all over the place. Simple economics says we should just build more facilities, right? The opportunity costs of not having enough production around means we have ample space to make more of the things we need and profit. You’re right. Companies like Intel are bringing new fabs online as fast as they can. Sadly, that is a process that is measured in months or even years. The capacity we need to offset the disruption to the chip market should have been built two years ago if we wanted it ready now.

All of these factors are mixed into one simple truth. Without the materials, manufacturing, or supply chain to deliver the equipment we’re going to be left out in the cold if we want something delivered today. Just in Time inventory is about to become Somewhere in Time inventory. We’re powerless to change the supply chain. Does that means we’re powerless to prevent disruption to our planning process?

Proactive Processes

We may not be able to assemble networking gear ourselves to speed up the process but we are far from helpless. The process and the planning around gear acquisition and deployment has to change to reflect the current state of the world. We can have an impact provided we’re ready to lead by example.

  • Procure NOW: Purchasing departments are notorious for waiting until the last minute to buy things. Part of that reasoning is that expenditures are worth less in the future than right now because those assets are more valuable today gaining interest or something. You need to go to the purchasing department and educate them about how things are working right now. Instead of them sitting on the project for another few months you need to tell them that the parts have to be ordered right now in order for them to be delivered in six or seven months. They’re going to fight you and tell you that they can just wait. However, we all know this isn’t going to clear up any time soon. If they persist in trying to tell you that you need to wait just have them try to go car shopping to illustrate the issue. If you want stuff by the end of Q1 2022, you need to get that order in NOW.
  • Preconfigure Things However You Can: If you’re stuck waiting six months to get switches and access points, are you going to be stuck waiting another month after they come in to configure them? I hope that answer is a resounding “NO”. There are resources available to make sure you can get things configured now so you’re not waiting when the equipment is sitting on a loading dock somewhere. You need to reach out to your VAR or your vendor and get some time on lab gear in the interim. If you ordered a wireless controller or a data center switch you can probably get some rack time on a very similar device or even the exact same one in a lab somewhere. That means you can work on a basic configuration or even provision things like VLANs or SSIDs so you’re not recreating the wheel when things come in. Even if all you have is a skeleton config you’re hours ahead of where you would be otherwise. And if the VAR or vendor gives you a hard time about lab gear you can always remind them that there are other options available for the next product refresh.
  • Minimum Viable Functionality: All this advice is great for a new pod or an addition to an existing network that isn’t critical. What if the gear you ordered is needed right now? What if this project can’t wait? How can you make things work today with nothing in hand? This is a bit trickier because it will require duplicate work. If you need to get things operational today you need to work with what you have today. That means you may have to salvage an old lab switch or pull something out of production and reduce available ports until the gear can arrive. It also means you’re going to have to backup the old configs, erase them completely (don’t forget about the VLAN database and VTP server configurations), and then put on the new info. When the new equipment comes in you’re going to have to do it all over again in reverse. It’s more work but it leads to things being operational today instead of constantly telling someone that it’s going to be a while. If you’re a VAR that’s doing this for a customer, you’d better make it very clear this is temporary and just a loan. Otherwise you might find your equipment being a permanent addition even after everything comes in.

Tom’s Take

The chip shortage is one of those things that’s going to linger under the best of circumstances. We’re going to be pressed to get gear in well into 2022. That means delayed projects and lots of arguing about what’s critical and what’s not. We can’t fix the semiconductor sector of the market but we can work to make sure that the impact felt there is the only one that impacts us right now. The more we do ahead of time to make things smooth the better it will be when it’s finally time to make things happen. Don’t let the lack of planning on the part of the supply chain sour your outlook on doing your role in networking.