Author Archives: networkingnerd

Unknown's avatar

About networkingnerd

Tom Hollingsworth, CCIE #29213, is a former network engineer and current organizer for Tech Field Day. Tom has been in the IT industry since 2002, and has been a nerd since he first drew breath.

A Handy Acronym for Troubleshooting

Posted on by
Reply

While I may be getting further from my days of being an active IT troubleshooter it doesn’t mean that I can’t keep refining my technique. As I spend time looking back on my formative years of doing troubleshooting either from a desktop perspective or from a larger enterprise role I find that there were always a few things that were critical to understand about the issues I was facing.

Sadly, getting that information out of people in the middle of a crisis wasn’t always super easy. I often ran into people that were very hard to communicate with during an outage or a big problem. Sometimes they were complicit because they made the mistake that caused it. They also bristled at the idea of someone else coming to fix something they couldn’t or wouldn’t. Just as often I ran into people that loved to give me lots of information that wasn’t relevant to the issue. Whether they were nervous talkers or just had a bad grasp on the situation it resulted in me having to sift through all that data to tease out the information I needed.

The Method

Today, as I look back on my career I would like to posit an idea of collecting the information that you need in order to effectively troubleshoot an issue.

  • Scope: How big is this problem? Is it just a single system or is it an entire building? Is it every site? If you don’t know how widespread the problem is you can’t really begin to figure out how to fix it. You need to properly understand the scope. That also includes understanding what the scope of the system for the business is. Taking down a reservation system for an airline is a bigger deal that guest Wi-Fi being down at a restaurant.
  • Timeline: When did this start happening? What occurred right before? Were there any issues that you think might have contributed here. It’s important to make the people you’re working with understand that a proper timeline is critical because it allows you to eliminate issues. You don’t want to spend hours trying to find the root cause in one system only to learn it wasn’t even powered on at the time and the real cause is in a switch that was just plugged in.
  • Frequency: Is this the first time this has happened? Does it happen randomly or seemingly on a schedule? This one helps you figure out if it’s systemic and regular or just cosmic rays. It also forces your team or customers to think about when it’s occurring and how far back the issue goes. If you come in thinking it’s a one-off that happened yesterday only to find out it’s actually been happening for weeks or months you’ll take a much different approach.
  • Urgency: Is this an emergency? Are we talking about a hospital ER being down or a typo in a documentation page? Do I need to roll out to spend the whole night fixing this or is it something that I can look at on a scheduled visit. Be sure to note the reasoning behind why they choose to make it a priority too. Some customers love to make everything a dire emergency just to ensure they get someone out right away. At least until it’s time to pay the emergency call rate.

A four step plan that’s easy to remember. Scope, Timeline, Frequency, Urgency. STFU.

Memory Aids

Okay, you can stop giggling now. I did that on purpose. In part to help you remember what the acronym was. In part to help you take a big of a relaxed approach to troubleshooting. In, in some ways, to help you learn to get those chatterboxes and pushy stakeholders off your back. If your methodology includes STFU they might figure out quickly that you need to be the one doing the talking and they need to be the one giving the answers, not the other way around.

And yes, each of these little steps would have saved me so much time in my old role. For example:

  • Scope – Was the whole network down? Or did one of the kids just unplug your Ethernet cable?
  • Frequency – Has this server seriously been beeping every 30 seconds for the last two years? Did you bother to look at the error message?
  • Timeline – Yes, I would assume that when you put that lab switch into your network was when the problem with VTP started.
  • Urgency – Do you really need me to drive three hours to press the F1 key on a keyboard?

I seriously have dozens of examples but these are four of the stories I tell all of the time to show just how some basic understanding can help people do more than they think.

Tom’s Take

People love mnemonic devices to remember things. Whether it’s My Very Eager Mother Just Served Us Nine (Pizzas) to remember the 8 planets and that one weird one or All People Seem To Need Data Processing to remember the seven layers of the OSI Model. I remember thinking through the important need-to-know information for doing some basic initial troubleshooting and how easily it fit into an acronym that could be handy for other things too when you’re in a stressful situation. Feel free to use it.

Painless Progress with My Ubiquiti Upgrade

Posted on by
3

I’m not a wireless engineer by trade. I don’t have a lab of access points that I’m using to test the latest and greatest solutions. I leave that to my friends. I fall more in the camp of having a working wireless network that meets my needs and keeps my family from yelling at me when the network is down.

Ubiquitous Usage

For the last five years my house has been running on Ubiquiti gear. You may recall I did a review back in 2018 after having it up and running for a few months. Since then I’ve had no issues. In fact, the only problem I had was not with the gear but with the machine I installed the controller software on. Turns out hard disk drives do eventually go bad and I needed to replace it and get everything up and running again. Which was my intention when it went down sometime in 2021. Of course, life being what it is I deprioritized the recovery of the system. I realized after more than a year that my wireless network hadn’t hiccuped once. Sure, I couldn’t make any changes to it but the joy of having a stable environment is that you don’t need to make constant changes. Still, I was impressed that I had no issues the necessitated my recovery of my controller software.

Flash forward to late 2023. I’m talking with some of the folks at Ubiquiti about a totally unrelated matter and I just happened to mention that I was impressed at how long the system had been running. They asked me what hardware I was working with and when I told them they laughed and said I needed to check out their new stuff. I was just about to ask them what I should look at when they told me they were going to ship me a package to install and try out.

Dreaming of Ease

Tom Hildebrand really did a great job because I got a UPS shipment at the beginning of December with a Ubiquiti Dream Machine SE, a new U6 Pro AP, and a G5 Flex Camera. As soon as I had the chance I unboxed the UDM SE and started looking over the installation process. The UDM SE is an all-in-one switch, firewall, and controller for the APs. I booted the system and started to do the setup process. I panicked for a moment because I realized that my computer was currently doing something connected to my old network and I didn’t want to have to dig through the pile to find a laptop to connect in via Ethernet to configure things.

That’s when my first surprise popped up. The UDM SE allows you to download the UniFi app to your phone and do the setup process from a mobile device. I was able to configure the UDM SE with my network settings and network names and get it staged and ready to do without the need for a laptop. That was a big win in my book. Lugging your laptop to a remote site for an installation isn’t always feasible. And counting on someone to have the right software isn’t either. How many times have you asked a junior admin or remote IT person what terminal program they’re using only to be met with a blank stare?

Once the UDM SE was up and running, getting the new U6 AP joined was easy. It joined the controller, downloaded the firmware updates and adopted my new (old) network settings. Since I didn’t have my old controller software handy I just recreated the old network settings from scratch. I took the opportunity to clean out some older compatibility issues that I was ready to be rid of thanks to an old Xbox 360 and some other ancient devices that were long ago retired. Clean implementations for the win. After the U6 was ready to go I installed it in my office and got ready to move my old AP to a different location to provide coverage.

The UDM SE detected that there were two APs that were running but part of a different controller. It asked me if I wanted to take them over and I happily responded in the affirmative. Sadly, when asked for the password to the old controller I drew a blank because that was two years ago and I can barely remember what I eat for breakfast. Ubiquiti has a solution for that and with some judicious use of the reset button I was able to reset the APs and join them to the UDM SE with no issues. Now everything is humming along smoothly. The camera is still waiting to be deployed once I figure out where I want to put it.

How is it all working? Zero complaints so far. Much like my previous deployment everything is humming right along and all my devices joined the new network without complaint. All the APs are running on new firmware and my new settings mean fewer questions about why something isn’t working because the kids are on a different network than the printer or one of the devices can’t download movies or something like that. Given how long I was running the old network without any form of control I’m glad it picked right up and kept going. Scheduling the right downtime at the beginning of the month may have had something to do with that but otherwise I’m trilled to see how it’s going.

Tom’s Take

Now that I’ve been running Ubiquiti for the last five years how would I rate it? I’d say for people that don’t want to rely on consumer APs from a big box store to run your home network you need to check Ubiquiti out. I know my friend Darrel Derosia is doing some amazing enterprise things with it in Memphis but I don’t need to run an entire arena. What I need is seamless connectivity for my devices without worry about what’s going to go down when I walk upstairs. My home network budget precludes enterprise gear. It fits nicely with Ubiquiti’s price point and functionality. Whether I’m trying to track down a lost Nintendo Switch or limit bandwidth so game updates aren’t choking out my cable modem I’m pleased with the performance and flexibility I have so far. I’m still putting the UDM SE through it’s paces and once I get the camera installed and working with it I’ll have more to say but rest assured I’m very thankful for Tom and his team for letting me kick the tires on some awesome hardware.

Disclaimer: The hardware mentioned in this post was provided by Ubiquiti at no charge to me. Ubiquiti did not ask for a review of their equipment and the opinions and perspectives represented in this post are mine and mine alone with not expectation of editorial review or compensation.

Back On Track in 2024

Posted on by
1

It’s time to look back at my year that was and figure out where this little train jumped off the rails. I’ll be the first to admit that I ran out of steam chugging along toward the end of the year. My writing output was way down for reasons I still can’t quite figure out. Everything has felt like a much bigger task to accomplish throughout the year. To that end, let’s look at what I wanted to do and how it came out:

  • Keeping Track of Things: I did a little bit better with this one, aside from my post schedule. I tried to track things much more and understand deadlines and such. I didn’t always succeed like I wanted to but at least I made the effort.
  • Creating Evergreen Content: This one was probably a miss. I didn’t create nearly as much content this year as I have in years past. What little I did create sometimes felt unfocused and less impactful. Part of that has to do with the overall move away from written content to something more video and audio focused. However, even my other content like Tomversations was significantly reduced this year. I will say that the one episode that I did record that dealt with InfiniBand was actually really good and I think it’s going to have some life in the future.
  • Insuring Intentionality: I tried to be more intentional with things in 2023 and we see how this turned out. I think I need to make sure to put that more at the front of my mind in 2024 as we look at the way that writing and other content creation is being transformed. In fact, the number of times that I’ve had to fight my AI-based autocomplete to make it stop finishing sentences for me reminds me how intentional I need to be in order to get the right things out there that I want to say. And before you say “just turn it off” I want to see how trainable it is to actually do what I want. So maybe part of the intentionality is making it intentional that I’m going to beat this thing.

Looking back at where I was makes me realize that content creation is always going to be a battle and so it making sure I have time for it. That means prioritizing the schedule for 2024, which isn’t going to be easy. Tech Field Day is now a part of the Futurum Group, which means I’m going to need to figure out how my role is going to be different in the coming months. I’m still going to be a part of Field Day but I also know I’m going to need to figure out how to navigate new coworkers and new goals. I have also been named a course director for my council’s Wood Badge course in the fall. That means doing some of the hardest leadership I’ve ever had to do, which I’m sure I’ll be documenting along the way here. As to what I want to specifically work on in 2024, what needs the most help?

  • Reaching Out For Help: Not surprisingly, this is something I have always needed help with (pun intended). I’ve never been one to ask for help with things until it’s almost to the point of disaster. So I need to be better in 2024 about asking for the help I need or think I’m going to need before it gets to be a huge problem. But that also means asking for assistance with things early on to help me get on the right track. Help isn’t always just doing things. It’s about making sure that you have the right ideas before you start down the track. So I’m going to make sure I’m ready to get the guidance and assistance I need when it’s needed and not when it’s an attempt to save the day.
  • Prioritizing Scheduling Intelligently: Part of the struggle in 2023 was making sure I was prioritizing things appropriately. Yes, work things always take priority as they should. But It’s also about other things that are part of my calendar that I need to get a handle on. I’ve done a good job of letting some of them go over the last year so the next phase is taming the ones that are left. Making sure the important meetings have their place and time but also making sure that those meetings have prep time and other pieces in the calendar so they don’t push anything else out of the way. It’s not enough to just block time and hope for the best. It’s about knowing what needs to be done and making it happen the right way at the right time.
  • Staying Consistent with Content: After the rise of GPT assistants and the flood of video content in 2023 I realize that I like writing more and more. Not having something complete my thoughts for me. Not jumping in front of a video camera to do stuff cold. I like to write. As much as I love the weekly Rundown show that we do I love writing the scripts almost as much. My Zen is in the keyboard, not the camera. I’ll still be creating video content but my focus will be in creating more of the writing that I like so much. I’ve already been experimenting with LinkedIn as a platform and I think I’ll be doing some more there too. Maybe not as much as I hope to do here but we will see how that goes.

Tom’s Take

We all have challenges we have to overcome. That’s the nature of life. As the industry has changed and evolved over time the way we communicate our ideas and perspectives to everyone has had to change as well. If you’d have told me ten years ago that Twitter would be a ghost town and Youtube would be everyone’s preferred learning tool I might have laughed. If you’d have told me five years ago I couldn’t have foreseen how things would turn out. The way we make it work is by staying on track and taking the challenges as they come. Switching social media platforms or embracing new content styles is all part of the game. But working with your strengths and making people smile and helping them to be informed is part of what this whole game is all about. 2024 is going to be another year of challenges and opportunities to shine. I hope the make the most of it and stay on track to success.

Production Reductions

Posted on by
1

You’ve probably noticed that I haven’t been writing quite as much this year as I have in years past. I finally hit the wall that comes for all content creators. A combination of my job and the state of the industry meant that I found myself slipping off my self-appointed weekly posting schedule more and more often in 2023. In fact, there were several times I skipped a whole week to get put something out every other week, especially in the latter half of the year.

I’ve always wanted to keep the content level high around here and give my audience things to think about. As the year wore on I found myself running out of those ideas as portions of the industry slowed down. If other people aren’t getting excited about tech why should I? Sure, I could probably write about Wi-Fi 7 or SD-WAN or any number of topics over and over again but it’s harder to repeat yourself for an audience that takes a more critical eye to your writing than it is for someone that just wants to churn out material.

My Bruce Wayne job kept me busy this year. I’m proud of all the content that we created through Tech Field Day and Gestalt IT, especially things like the weekly Rundown show. Writing a post every week is hard. Writing a snarky news show script is just as taxing. If I can find a way to do that I can find a way to write, right?

Moving Targets

Alas, in order to have a plan for content creation you have to make a plan and then stick to it. I did that last year with my Tomversations pieces and it succeeded. This year? I managed to make one. Granted, it was a good one but it was still only one. Is it because I didn’t plan ahead far enough? Or because I didn’t feel like I had much to say?

Part of the secret behind writing is to jot down your ideas right away , no matter how small they might be. You can develop an idea that has merit. You can’t develop a lack of an idea. I have a note where I add quotes and suggestions and random things that I overhear that give me inspiration. Sometimes those ideas pan out. Other times they don’t. I won’t know either way if I don’t write them down and do something about them. If you don’t create the ground for your ideas to flourish you’ll have nothing to reap when it’s time.

The other thing that causes falloffs in content creation is timing. I always knew that leaving my posts until Friday mornings was going to eventually bite me and this year was the year with teeth. Forcing myself to come up with something in a couple of hours time initially led to some pretty rushed ideas and that later pushed into the following Monday (or beyond). While creating a schedule for my thoughts has helped me stay consistent throughout the years the pressures on my schedule this year have meant letting some things slip when they weren’t critical. Hard to prioritize a personal post over a work video that needs to be edited or a paper that needs to be written first.

One other thing that I feel merits some mention is the idea of using tools to help the creative process. I am personally against using a GPT algorithm to write for me. It just doesn’t sound like me and I feel that having something approximating who I am doesn’t have the same feel. Likewise, one of the other things this year that I’m fighting with is word predictions in writing tools. Not as bad as full-on content creation but merely “suggestions” about what word I want to use next. I’ve disabled them for the most part because, while helpful in certain situations they annoy me more than anything when writing. Seeing a tool suggest a word for me while I’m in the flow of writing a post is like hearing a note a half step out of tune in a piece of music. It’s just jarring enough to take you out of the whole experience. Stop trying to anticipate what I’m going to say and let me say it!

Producing Ahead

Does all this mean I’m giving up on my writing? Not hardly. I still feel like writing is my best form of communication. Even a simple post about complaining about my ability to write this year is going to be wordy. I feel it’s because written words give us more opportunity to work at our own pace. When we watch videos we work at someone else’s idea of a learning pace. If you make a ten-minute video to get across a point that could have been read in three minutes you’re either doing a very good job of explaining everything or you’re padding out your work. I prefer to skim, condense, and study the parts that are important to me. I can’t really do that with a video.

I feel the written form of content is still going to be king for years to come. You can search words. You can rephrase words. You can get a sense for how dense a topic is by word count. There’s value in seeing the entire body of knowledge in front of you before you begin. Besides, the backspace key is a whole lot easier to edit than doing another take and remembering to edit out the bad one in the first place.

Tom’s Take

Writing is practically meditation for me at this point. I can find a topic I’m interested in and write. Empty my brain of thoughts and ideas and let them take shape here. AI can’t approximate that for me. Video has too many other variables to worry about. That’s why I’m a writer. I love the way the process works with just a keyboard, a couple of references, and my brain doing the heavy lifting. I’m not sure what my schedule for posting is going to look like in 2024 and beyond but trust me when I say it’s not going away any time soon.

Routing Through the Forest of Trees

Posted on by
Reply

Some friends shared a Reddit post the other day that made me both shake my head and ponder the state of the networking industry. Here is the locked post for your viewing pleasure. It was locked because the comments were going to devolve into a mess eventually. The person making the comment seems to be honest and sincere in their approach to “layer 3 going away”. The post generated a lot of amusement from the networking side of IT about how this person doesn’t understand the basics but I think there’s a deeper issue going on.

Trails To Nowhere

Our visibility of the state of the network below the application interface is very general in today’s world. That’s because things “just work” to borrow an overused phrase. Aside from the occasional troubleshooting exercise to find out why packets destined for Azure or AWS are failing along the way when is the last time you had to get really creative in finding a routing issue in someone else’s equipment? We spend more time now trying to figure out how to make our own networks operate efficiently and less time worrying about what happens to the packets when they leave our organization. Provided, of course, that the users don’t start complaining about latency or service outages.

That means that visibility of the network functions below the interface of the application doesn’t really exist. As pointed out in the post, applications have security infrastructure that communicates with other applications and everything is nicely taken care of. Kind of like ordering packages from your favorite online store. The app places the order with a storefront and things arrive at your house. You don’t have to worry about picking the best shipping method or trying to find a storefront with availability or any of the older ways that we had to deal with weirdness.

That doesn’t mean that the processes that enable that kind of service are going away though. Optimizing transport networks is a skill that is highly specialized but isn’t a solved issue. You’ve probably heard by now that UPS trucks avoid left turns whenever possible to optimize safety and efficiency. The kind of route planning that needs to be done in order to eliminate as many left turns as possible from the route is massive. It’s on the order of a very highly specialized routing protocol. What OSPF and BGP are doing is akin to removing the “left turns” from the network. They find the best path for packets and keep up-to-date as the information changes. That doesn’t mean the network is going away. It means we’re finding the most efficient route through it for a given set of circumstances. If a shipping company decides tomorrow that they can no longer guarantee overnight delivery or even two-day shipping that would change the nature of the applications and services that offer that kind of service drastically. The network still matters.

OSI Has to Die

The other thing that jumped out at me about the post was the title. Referring to Layer 3 of the OSI model as a routing function. The timing was fortuitous because I had just finished reading Robert Graham’s excellent treatise on getting rid of the OSI model and I couldn’t agree more with him. Containing routing and addressing functions to a single layer of an obsolete model gives people the wrong ideas. At the very least is encourages them to form bad opinions about those ideas.

Let’s look at the post as an example. Taking a stance like “we don’t need layer three because applications will connect to each other” is bad. So is “We don’t need layer two because all devices can just broadcast for the destination”. It’s wrong to say those things but if you don’t know why it’s wrong then it doesn’t sound so bad. Why spend time standing up routing protocols if applications can just find their endpoints? Why bother putting higher order addresses on devices when the nature of Ethernet means things can just be found easily with a broadcast or neighbor discovery transmission? Except you know that’s wrong if you understand how remote networks operate and why having a broadcast domain of millions of devices would be chaos.

Graham has some very compelling points about relegating the OSI model to history and teaching how networks really operate. It helps people understand that there are multiple networks that exist at one time to get traffic to where it belongs. While we may see the Internet and Ethernet LAN as a single network they have different purposes. One is for local traffic delivery and the other is for remote traffic delivery. The closest analog for certain generations is the phone system. There was a time when you have local calls and long distance calls that required different dialing instructions. You still have it today but it’s less noticeable thanks to mobile devices not requiring long distance dialing instructions.

It might be more appropriate to think of the local/remote dichotomy like a private branch exchange (PBX) phone network. Phones inside the PBX have locally significant extensions that have no meaning outside of the system. Likewise, remote traffic can only enter the system through entry points created by administrators, like a main dial-in number that terminates on an extension or direct inward dial (DID) numbers that have significance outside the system. Extensions only matter for the local users and have no way to communicate outside without addressing rules. Outside addresses have no way of communicating into the local system without creating rules that allow it to happen. It’s a much better metaphor than the OSI model.

Tom’s Take

I don’t blame our intrepid poster for misunderstanding the way network addresses operate. I blame IT for obfuscating it because it doesn’t matter anymore to application developers. Sure, we’ve finally hit the point where the network has merged into a single entity with almost no distinction from remote WAN and local LAN. But we’ve also created a system where people forget the dependencies of the system at lower levels. You can’t encode signals without a destination and you can’t determine the right destination without knowing where it’s supposed to be. That’s true if you’re running a simple app in an RFC 1918 private space or the public cloud. Forgetting that little detail means you could end up lost in a forest not being able to route yourself out of it again.

Asking The Right Question About The Wireless Future

Posted on by
2

It wasn’t that long ago that I wrote a piece about how Wi-Fi 6E isn’t going to move the needle very much in terms of connectivity. I stand by my convictions that the technology is just too new and doesn’t provide a great impetus to force users to upgrade or augment systems that are already deployed. Thankfully, someone at the recent Mobility Field Day 10 went and did a great job of summarizing some of my objections in a much simpler way. Thanks to Nick Swiatecki for this amazing presentation:

He captured so many of my hesitations as he discussed the future of wireless connectivity. And he managed to expand on them perfectly!

New Isn’t Automatically Better

Any time I see someone telling me that Wi-Fi 7 is right around the corner and that we need to see what it brings I can’t help but laugh. There may be devices that have support for it right now, but as Nick points out in the above video, that’s only one part of the puzzle. We still have to wait for the clients and the regulatory bodies to catch up to the infrastructure technology. Could you imagine if we did the same thing with wired networks? If we deployed amazing new cables that ran four times the speed but didn’t interface with the existing Ethernet connections at the client? We’d be laughed out of the building.

Likewise, deploying pre-standard Wi-Fi 7 devices today doesn’t gain you much unless you have a way to access them with a client adapter. Yes, they do exist. Yes, they’re final. However, they’re more final than the Draft 802.11n cards that I deployed years and years ago. That doesn’t mean that we’re going to see a lot of benefit from them however. Because the value of the first generation of a technology is rarely leaps and bounds above what came before it.

A couple of years ago I asked if the M1 MacBook wireless was really slower than the predecessor laptop. Spoiler alert, it is but not so much you’d really notice. Since then we’ve gained two more generations of that hardware and the wireless has gotten faster. Not because the specs have changed in the standard. It’s because the manufacturers have gotten better about building the devices. We’ve squeezed more performance out of them instead of just slapping a label on the box and saying it’s a version number higher or it’s got more of the MHz things so it must be better.

Nick, in the above video, points this out perfectly. People keep asking about Wi-Fi 7 and they miss out on the fact that there’s a lot of technology that needs to run very smoothly in order to give us significant gains in speed over Wi-Fi 6 and Wi-Fi 6E. And those technologies probably aren’t going to be implemented well (if at all) in the first cards and APs that come off the line. In fact, given the history of 802.11 specifications those important features are probably going to be marked as optional anyway to ensure the specifications get passed on time to allow the shipping hardware to be standardized.

In a perfect world you’re going to miss a lot of the advances in the first revision of the hardware. I remember a time when you had to be right under the AP to see the speed increases promised by the “next generation” of wireless. Adding more and more advanced technology to the AP and hoping the client adapters catch up quickly isn’t going to help sell your devices any faster either. Everything has to work together to ensure it all runs smoothly for the users. If you think for a minutes that they aren’t going to call you to tell you that the wireless is running slow then you’re very mistaken. They’re upset they didn’t get the promised speeds on the box or that something along the line is making their experience difficult. That’s the nature of the beast.

Asking the Right Questions

The other part of this discussion is how to ensure that everyone has realistic ideas about what new technology brings. For that, we recorded a great roundtable discussion about Wi-Fi 7 promises and reality:

I think the biggest takeaway from this discussion is that, despite the hype, we’re not ready for Wi-Fi 7 just yet. The key to having this talk with your stakeholders is to remind them that spending the money on the new devices isn’t going to automatically mean increased speeds or enhanced performance. In fact, you’re going to do a great job of talking them out of deploying cutting edge hardware simply by reminding them they aren’t going to see anywhere near the promises from the vendors without investing even more in client hardware or understanding that those amazing fast multi-spectrum speeds aren’t going to be possible on an iPhone.

We’re not even really touching on the reality that some of the best parts of 6GHz aren’t even available yet because of FCC restrictions. Or that we just assume that Wi-Fi 7 will include 6GHz when it doesn’t even have to. That’s especially true of IoT devices. Lower cost devices will likely have lower cost radios for components which means the best speed increases are going to be for the most expensive pieces of the puzzle. Are you ready to upgrade your brand new laptop in six months because a new version of the standard came out that’s just slightly faster?

Those are the questions you have to ask and answer from your stakeholders before you ever decide how the next part of the project is going to proceed. Because there is always going to be faster hardware or newer revisions of the specification for you to understand. And if the goalposts keep moving every time something new comes along you’re either going to be broke or extremely disappointed.

Tom’s Take

I’m glad that Nick from Cisco was able to present at Mobility Field Day. Not only did he confirm what a lot of professionals are thinking but he did it in a way that helped other viewers understand where the challenges with new wireless technologies lie. We may be a bit jaded in the wired world because Ethernet is such a bedrock standard. In the wireless world I promise that clients are always going to be getting more impressive and the amount of time between those leaps is going to shrink even more than it already has. The real question should be whether or not we need to chase that advantage.

AI Is Making Data Cost Too Much

Posted on by
Reply

You may recall that I wrote a piece almost six years ago comparing big data to nuclear power. Part of the purpose of that piece was to knock the wind out of the “data is oil” comparisons that were so popular. Today’s landscape is totally different now thanks to the shifts that the IT industry has undergone in the past few years. I now believe that AI is going to cause a massive amount of wealth transfer away from the AI companies and cause startup economics to shift.

Can AI Really Work for Enterprises?

In this episode of Packet Pushers, Greg Ferro and Brad Casemore debate a lot of topics around the future of networking. One of the things that Brad brought up that Greg pointed out is that data being used for AI algorithm training is being stored in the cloud. That massive amount of data is sitting there waiting to be used between training runs and it’s costing some AI startups a fortune in cloud costs.

AI algorithms need to be trained to be useful. When someone uses ChatGPT to write a term paper or ask nonsensical questions you’re using the output of the GPT training run. The real work happens when OpenAI is crunching data and feeding their monster. They have to give it a set of parameters and data to analyze in order to come up with the magic that you see in the prompt window. That data doesn’t just come out of nowhere. It has to be compiled and analyzed.

There are a lot of creators of content that are angry that their words are being fed into the GPT algorithm runs and then being used in the results without giving proper credit. That means that OpenAI is scraping the content from the web and feeding it into the algorithm without care for what they’re looking it. It also creates issues where the validity and the accuracy of the data isn’t verified ahead of time.

Now, this focuses on OpenAI and GPT specifically because everyone seems to think that’s AI right now. Much like every solution in the history of IT, GPT-based large language models (LLMs) are just a stepping stone along the way to greater understanding of what AI can do. The real value for organizations, as Greg pointed out in the podcast, can be something as simple as analyzing the trouble ticket a user has submitted and then offering directed questions to help clarify the ticket for the help desk so they spend less time chasing false leads.

No Free Lunchboxes

Where are organizations going to store that data? In the old days it was going to be collected in on-prem storage arrays that weren’t being used for anything else. The opportunity cost of using something you already owned was minimal. After all, you bought the capacity so why not use it? Organizations that took this approach decided to just save every data point they could find in an effort to “mine” for insights later. Hence the references to oil and other natural resources.

Today’s world is different. LLMs need massive resources to run. Unless you’re willing to drop several million dollars to build out your own cluster resources and hire engineers to keep the running at peak performance you’re probably going to be using a hosted cloud solution. That’s easy enough to set up and run. And you’re only paying for what you use. CPU and GPU times are important so you want the job to complete as fast as possible in order to keep your costs low.

What about the data that you need to feed to the algorithm? Are you going to feed it from your on-prem storage? That’s way too slow, even with super fast WAN links. You need to get the data as close to the processors as possible. That means you need to migrate it into the cloud. You need to keep it there while the magic AI building machine does the work. Are you going to keep that valuable data in the cloud, incurring costs every hour it’s stored there? Or are you going to pay to have it moved back to your enterprise? Either way the sound of a cash register is deafening to your finance department and music to the ears of cloud providers and storage vendors selling them exabytes of data storage.

All those hopes of making tons of money from your AI insights are going to evaporate in a pile of cloud bills. The operations costs of keeping that data are now more than minimal. If you want to have good data to operate on you’re going to need to keep it. And if you can’t keep it locally in your organization you’re going to have to pay someone to keep it for you. That means writing big checks to the cloud providers that have effectively infinite storage, bounded only by the limit on your credit card or purchase order. That kind of wealth transfer makes investors seem a bit hesitant when they aren’t going to get the casino-like payouts they’d been hoping for.

The shift will cause AI startups to be very frugal in what they keep. They will either amass data only when they think their algorithm is ready for a run or keep only critical data that they know they’re going to need to feed the monster. That means they’re going to be playing a game with the accuracy of the resulting software as well as giving up chances that some insignificant piece of data ends up being the key to a huge shift. In essence, the software will all start looking and sounding the same after a while and there won’t be enough differentiation to make they competitive because no one will be able to afford it.

Tom’s Take

The relative ease with which data could be stored turned companies into data hoarders. They kept it forever hoping they could get some value out of it and create a return curve that soared to the moon. Instead, the application for that data mining finally came along and everyone realized that getting the value out of the data meant investing even more capital into refining it. That kind of investment makes those curves much flatter and makes investors more reluctant. That kind of shift means more work and less astronomical payout. All because your resources were more costly than you first thought.

Does Automation Require Reengineering?

Posted on by
2

During Networking Field Day 33 this week we had a great presentation from Graphiant around their solution. While the presentation was great you should definitely check out the videos linked above, Ali Shaikh said something in one of the sessions that resonated with me quite a bit:

Automation of an existing system doesn’t change the system.

Seems simple, right? It belies a major issue we’re seeing with automation. Making the existing stuff run faster doesn’t actually fix our issues. It just makes them less visible.

Rapid Rattletraps

Most systems don’t work according to plan. They’re an accumulation of years of work that doesn’t always fit well together. For instance, the classic XKCD comic:

When it comes to automation, the idea is that we want to make things run faster and reduce the likelihood of error. What we don’t talk about is how each individual system has its own quirks and may not even be a good candidate for automation at any point. Automation is all about making things work without intervention. It’s also dependent on making sure the process you’re trying to automate is well-documented and repeatable in the first place.

How many times have you seen or heard of someone spending hours trying to script a process that takes about five minutes to do once or even twice a year? The return on time investment in automating something like that doesn’t really make sense, does it? Sure, it’s cool to automate everything but it’s not really useful, especially if the task has changes every time it’s run that requires you to change in the inputs. It’s like building a default query for data that needs to be rewritten every time the query is run.

You’re probably laughing right now but you also have at least one or two things that would fit this bill. Rather than asking if you should be automating this task you should instead be asking why we’re doing it in the first place. Why are we looking to accomplish this goal if it only needs to be done on occasion? Is it something critical like a configuration backup? Or maybe just a sanity check to see that unused switch ports have been disabled or tagged with some kind of security configuration. Are you trying to do the task for safety or security? Or are you doing it for busy work purposes?

Streamlining the System

In all of those cases we have to ask why the existing system exists. That’s because investing time and resources into automating a system can result in a big overrun in budget when you run into unintended side effects or issues that weren’t documented in the first place. Nothing defeats an automation project faster than hitting roadblocks out of nowhere.

If you shouldn’t invest time in automating something that is already there, what should you do instead? How about reengineering the whole process instead? If you occasionally run configuration backups to make sure you have good copies of the devices why not institute change controls or rolling automatic backups? Instead of solving an existing problem with a script why shouldn’t you change the way you do things that might have other hidden benefits? If you’re scripting changes to ports to verify security status why not have a system in place that creates configuration on those ports when they’re configured and require change controls to enable them?

It feels like extra work. It always seems easier to jump in from the bottom up with both feet and work on a problem until you solve it. Top down means you’re changing the way the system does things instead so the problems either disappear or change to something more manageable. The important question to ask is “where are my resources best spent?” If you see your time as a resource to invest in projects are you better served making something existing work slightly faster? Or would it be better for you to take the time to do something in a different, potentially better way?

If you believe your process is optimized as much as possible and just needs to run on its own that makes for an easy conversation. But if you’re thinking you need to change the way you do things this is a great time to make those changes and use your time investment to do things properly this time around. You may have to knock down a few walls to get there but it’s way better than building a house of cards that is just going to collapse faster.

Tom’s Take

I’m a fan of automation. Batch files and scripting and orchestration systems have a big place in the network to reduce error and multiply the capabilities of teams. Automation isn’t a magic solution. It requires investment of time and effort and a return for the stakeholders to see value. That means you may need to approach the problem from a different perspective to understand what really should be done instead of just doing the same old things a little faster. Future you will thank you for reengineering today.

Victims of Success

Posted on by
Reply

It feels like the cybersecurity space is getting more and more crowded with breaches in the modern era. I joke that on our weekly Gestalt IT Rundown news show that we could include a breach story every week and still not cover them all. Even Risky Business can’t keep up. However, the defenders seem to be gaining on the attackers and that means the battle lines are shifting again.

Don’t Dwell

A recent article from The Register noted that dwell times for detection of ransomware and malware hav dropped almost a full day in the last year. Dwell time is especially important because detecting the ransomware early means you can take preventative measures before it can be deployed. I’ve seen all manner of early detection systems, such as data protection companies measuring the entropy of data-at-rest to determine when it is no longer able to be compressed, meaning it likely has been encrypted and should be restored.

Likewise, XDR companies are starting to reduce the time it takes to catch behaviors on the network that are out of the ordinary. When a user starts scanning for open file shares and doing recon on the network you can almost guarantee they’ve been compromised somehow. You can start limiting access and begin cleanup right away to ensure that they aren’t going to get much further. This is an area where zero trust network architecture (ZTNA) is shining. The less a particular user has access to without additional authentication, the less they can give up before the controls in place in the system catch them doing something out of the ordinary. This holds true even if the user hasn’t been tricked into giving up their credentials but instead is working with the attackers through monetary compensation or misguided ire toward the company.

Thanks to the advent of technologies like AI, machine learning, and automation we can now put controls in place quickly to prevent the spread of disaster. You might be forgiven for thinking that kind of response will eradicate this vector of attack. After all, killing off the nasty things floating in our systems means we’re healthier overall, right? It’s not like we’re breeding a stronger strain of disease?

Breeding Grounds

Ask a frightening question and get a frightening answer, right? In the same linked Register article the researchers point out that while dwell times have been reduced the time it takes attackers to capitalize on their efforts has also been accelerated. In addition, attackers are looking at multiple vectors of persistence in order to accomplish their ultimate goal of getting paid.

Let’s assume for the moment that you are an attacker that knows the company you’re going after is going to notice your intrusion much more quickly than before. Do you try to sneak in and avoid detection for an extra day? Or do you crash in through the front door and cause as much chaos as possible before anyone notices? Rather than taking the sophisticated approach of persistence and massive system disruption, attackers are instead taking a more low-tech approach to grabbing whatever they can before they get spotted and neutralized.

If you look at the most successful attacks so far in 2023 you might notice they’ve gone for a “quantity over quality” approach. Sure, a heist like Oceans 11 is pretty impressive. But so is smashing the display case and running out with the jewels. Maybe it’s not as lucrative but when you hit twenty jewelry stores a week you’re going to make up the low per capita take with volume.

Half of all the intrusion attempts are coming at the expense of stolen or compromised credentials. There are a number of impressive tools out there that can search for weak points in the system and expose bugs you never even dreamed could exist. There are also much easier ways to phish knowledge workers for their passwords or just bribe them to gain access to restricted resources. Think of it like the crowbar approach to the heist scenario above.

Lock It Down

Luckily, even the fastest attackers still have to gain access to the system to do damage. I know we all harp on it constantly but the best way to prevent attacks is to minimize the ways that attack vectors get exploited in the first place. Rotate credentials frequently. Have knowledge workers use generated passwords in place of ones that can be tied back to them. Invest in password management systems or, more broadly, identity management solutions in the enterprise. You can’t leak what you don’t know or can’t figure out quickly.

After that, look at how attackers capitalize on leaks or collusion. I know it’s a tale as old as time but you shouldn’t be running anything with admin access that doesn’t absolutely need it. Yes, even YOUR account. You can’t be the vector for a breach if you are just as unimportant as everyone else. Have a separate account with a completely different password for doing those kinds of tasks. Regularly audit accounts that have system-level privilege and make sure they’re being rotated too. Another great reason for having an identity solution is that the passwords can be rotated quickly without disruption. Oh, and make sure the logins to the identity system are as protected as anything else.

Lastly, don’t make the mistake of thinking you’re an unappealing target. Just because you don’t deal with customer data or have personally identifiable information (PII) stored in your system doesn’t mean you’re not going to get swept up in the next major attack. With the quantity approach the attackers don’t care what they grab as long as they can get out with something. They can spend time analyzing it later to figure out how to best take advantage of what they’ve stolen. Don’t give them the chance. Security through obscurity doesn’t work well in an age where you can be targeted and exposed before you realize what’s going on.

Tom’s Take

Building a better mousetrap means you catch more mice. However, the ones that you don’t catch just get smarter and figure out how to avoid the bait. That’s the eternal game in security. You stamp out the low-level threats quickly but that means the ones that aren’t ensnared become more resistant to your efforts. You can’t assume every attack is going to be a sophisticated nation state attempt to steal classified info. You may just be the unlucky target of a smash-and-grab with stolen passwords. Don’t become a victim of your own success. Keep tightening the defenses and make sure you don’t wind up missing the likely while looking for the impossible.

The Essence of Cisco and Splunk

Posted on by
1

You no doubt noticed that Cisco bought Splunk last week for $28 billion. It was a deal that had been rumored for at least a year if not longer. The purchase makes a lot of sense from a number of angles. I’m going to focus on a couple of them here with some alliteration to help you understand why this may be one of the biggest signals of a shift in the way that Cisco does business.

The S Stands for Security

Cisco is now a premier security company now. The addition of the most power SIEM on the market means that Cisco’s security strategy now has a completeness of vision. SecureX has been a very big part of the sales cycle for Cisco as of late and having all the parts to make it work top to bottom is a big win. XDR is a great thing for organizations but it doesn’t work without massive amounts of data to analyze. Guess where Splunk comes in?

Aside from some very specialized plays, Cisco now has an answer for just about everything a modern enterprise could want in a security vendor. They may not be number one in every market but they’re making a play for number two in as many places as possible. More importantly it’s a stack that is nominally integrated together to serve as a single source for customers. I’ll be the first person to say that the integration of Cisco software acquisitions isn’t seamless. However, when the SKUs all appear together on a bill of materials most organizations won’t look beyond that. Especially if there are professional services available to just make it work.

Cisco is building out their security presence in a big way. All thanks to a big investment in Splunk.

The S Stands for Software

When the pundits said that Cisco could never really transform themselves from a hardware vendor to a software company there was a lot of agreement. Looking back at the transition that Chuck Robbins has led since then what would you say now? Cisco has aggressively gone after software companies across the board to build up a portfolio of recurring revenue that isn’t dependent on refresh cycles or silicon innovations.

Software is the future for Cisco. Iteration on their core value products is going to increase their profit far beyond what they could hope to realize through continuing to push switches and routers. That doesn’t mean that Cisco is going to abandon the hardware market. It just means that Cisco is going to spend more time investing in things with better margins. The current market for software subscriptions and recurring licensing revenue is hot and investors want to see those periodic returns instead of a cycle-based push for companies to adopt new technologies.

What makes more sense to you? Betting on a model where customers need to pay per gigabyte of data stored or a technology which may be dead on the vine?. Taking the nerd hat off for a moment means you need to see the value that companies want to realize, not the hope that something is going to be big in the future. Hardware will come along when the software is ready to support it. Blu-Ray didn’t win over HD-DVD because it was technically superior. It won because Sony supported it and convinced Disney to move their media onto it exclusively.

Software is the way forward for Cisco. Software that provides value for enterprises and drives upgrades and expansion. The hardware itself isn’t going to pull more software onto the bottom line.

The S Stands for Synergy

The word synergy is overused in business vernacular. Jack Welch burned us out on the idea that we can get more out of things together by finding hidden gems that aren’t readily apparent. I think that the real value in the synergy between Cisco and Splunk can be found in the value of creating better code and better programmers.

A bad example of synergy is Cisco’s purchase of Flip Video. When it became clear that the market for consumer video gear wasn’t going to blow up quite like Cisco had hoped they pivoted to talk about using the optics inside the cameras to improve video quality for their video collaboration products. Which never struck me as a good argument. They bet on something that didn’t pay off and had to salvage it the best they could. How many people went on to use Cisco cameras outside of the big telepresence rooms? How many are using them today instead of phone cameras or cheap webcams?

Real synergy comes when the underlying processes are examined and improved or augmented. Cisco is gaining a group of developers and product people that succeed based on the quality of their code. If Splunk didn’t work it wouldn’t be a market leader. The value of what that team can deliver to Cisco across the organization is important. The ongoing critiques of Cisco’s code quality has created a group of industry professionals that are guarded when it comes to using Cisco software on their devices or for their enterprises. I think adding the expertise of the Splunk teams to that group will go a long way to helping Cisco write more stable code in the long term.

Tom’s Take

Cisco needed Splunk. They needed a SIEM to compete in the wider security market and rather than investing in R&D to build one they decided to pick up the best that was available. The long courtship between the two companies finally paid off for the investors. Now the key is to properly integrate Splunk into the wider Cisco Security strategy and also figure out how to get additional benefits from the development teams to offset the big purchase price. The essence of those S’s above is that Cisco is continuing their transformation away from a networking hardware company and becoming a more diversified software firm. It will take time for the market to see if that is the best course of action.