Author Archives: networkingnerd

Unknown's avatar

About networkingnerd

Tom Hollingsworth, CCIE #29213, is a former network engineer and current organizer for Tech Field Day. Tom has been in the IT industry since 2002, and has been a nerd since he first drew breath.

Wi-Fi 6E Won’t Make a Difference

Posted on by
3

It’s finally here. The vaunted day when the newest iPhone model has Wi-Fi 6E. You’d be forgiven for missing it. It wasn’t mentioned as a flagship feature in the keynote. I had to unearth it in the tech specs page linked above. The trumpets didn’t sound heralding the coming of a new paradigm shift. In fact, you’d be hard pressed to find anyone that even cares in the long run. Even the rumor mill had moved on before the iPhone 15 was even released. If this is the technological innovation we’ve all been waiting for, why does it sound like no one cares?

Newer Is Better

I might be overselling the importance of Wi-Fi 6E just a bit, but that’s because I talk to a lot of wireless engineers. More than a couple of them had said they weren’t even going to bother upgrading to the new USB-C wonder phone unless it had Wi-Fi 6E. Of course, I didn’t do a survey to find out how many of them had 6E-capable access points at home, either. I’d bet the number was 100%. I’d be willing to be the survey of people outside of that sphere looking to buy an iPhone 15 Pro that can tell me if they have a 6E-capable chipset at home is much, much lower.

The newest flagship device has cool stuff. Better cameras, faster processor, more RAM, and even titanium! The reasons to upgrade are legion depending on how old your device is. Are you really ready to sink it all because of a wireless chipset design? There are already a number of folks saying they won’t upgrade their amazing watch because Apple didn’t make it black this year. Are the minor technical achievements really deal breakers in the long run?

The fact of the matter is that the community of IT pros outside of the wireless space don’t actually care about the wireless chipset in their phone. Maybe it’s faster. Maybe it’s cooler. It could even be more about bragging rights than anything else. However, just like the M1 MacBook Wi-Fi, the real-world results are going to be a big pile of “it depends”. That’s because organizations don’t make buying decisions based on consumer tech.

Sure, the enterprise may have been pushed in certain directions in the past due to the adoption of smart phones. Go into any big box store and see how the employees are using phones instead of traditional scanners for inventory management. Go into your average bank or hospital and ask the CIO what their plans are to upgrade the wireless infrastructure to support Wi-Fi 6E now that Apple supports it across the board on their newest devices. I bet you get a very terse answer.

Gen Minus One

The buying patterns for enterprise IT don’t support bleeding edge technology. That’s because most enterprises don’t run on the bleeding edge. Their buying decisions are informed by the installation base of their users, not on their projected purchases. Enterprises aren’t going to take a risk on buying something that isn’t going to provide benefit for the investment. Trying to provide that benefit for a small number of users is even more suspect. Why spend big bucks for a new access point that a tenth of my workforce can properly use?

Buying decisions and deployment methodology follow a timeline that was decided upon months ago, even for projects that come up out of the blue. If you interview your average CIO with a good support team they can tell you how old their devices are, what order they are planned to be replaced, and roughly how much that will cost today. They have a plan ready to plug in when the executive team decides there is budget to spend. Strike while the funding iron is hot!

To upend the whole plan because some new device came out is not an easy sell to the team. Especially if it means reducing the number of devices that can be purchased because the newer ones cost more. If anything it will encourage the teams to hold on to that particular budget until the prices of those cutting edge devices falls to a point where they are more cost effective for a user base that has refreshed devices and has a need for faster connectivity.

Wi-Fi 6E suffers from a problem common to IT across the board. It’s not exciting enough to be important. The current generation of devices can utilize the connectivity it provides efficiently. The airspace in an enterprise is certainly crowded enough to need new bands for high performance devices to move into. But does the performance of Wi-Fi 6E create such a gap as to make it a “must have” in the budget? What would you be willing to sacrifice to get it? And would your average user notice the difference? If you can’t say for certain that incremental improvement will make that much of a difference for the non-wireless savvy person then you’re going to find yourself waiting for the next revision of the standard. Which, sadly, as the benefit of having a higher number. Which means it’s obviously better, right?

Tom’s Take

I like shiny new things. I didn’t upgrade my phone this year because my older one is good enough for my use case. If I were to rank all the reasons why I wanted to upgrade I’d put Wi-Fi 6E near the bottom of the list. It’s neat. I like the technology behind it. For the average CIO it doesn’t move the needle. It doesn’t have an impressive pie chart or cost savings associated with it. If you upgraded everyone to Wi-Fi 6E overnight no one would notice. And even if they did they’d be asking when Wi-Fi 7 was coming out because that one is really cool, even if they know zero about what it does. Wi-Fi 6E on a mobile device won’t matter in the long run because the technology isn’t cool enough to be noticed by people that aren’t looking for it.

Overcoming the Wall

Posted on by
Reply

I was watching a Youtube video this week that had a great quote. The creator was talking about sanding a woodworking project and said something about how much it needed to be sanded.

Whenever you think you’re done, that’s when you’ve just started.

That statement really resonated with me. I’ve found that it’s far too easy to think you’re finished with something right about the time you really need to hunker down and put in extra effort. In running they call it “hitting the wall” and it usually marks the point when your body is out of energy. There’s often another wall you hit mentally before you get there, though, and that’s the one that needs to be overcome with some tenacity.

The Looming Rise

If your brain is like mine you don’t like belaboring something. The mind craves completion and resolution. Once you’ve solved a problem it’s done and finished. No need to continue on with it once you’ve reached a point where it’s good enough. Time to move on to something else that’s new and exciting and a source of dopamine.

However, that feeling of being done with something early on is often a false sense of completion. I learned that the hard way when I was studying for my CCIE. Every question has an answer. Some questions have a couple of different answers. However, knowing the correct answer isn’t the same as knowing all the incorrect answers. Why would I want to take the time to learn all the wrong things instead of just learning what’s right and moving on to the next topic?

The reason to keep going even after you know what’s right is to recognize what the wrong thing looks like. When studying you’re often confronted with suboptimal situations or, especially with the CCIE, put into positions where you can make mistakes that will lead to disaster if you don’t recognize the pitfalls early. Maybe it’s creating a routing loop. It could be a choice between two methods of configuration that really only has one correct answer if you know why the other one will cause problems.

Persevering through that mental wall that says “you’ve done enough” is important because the extra value you gain when you do is critical to understand the myriad ways that something can be broken. It’s not enough to know it’s not right. You have to recognize what isn’t right about it. That kind of understanding can come from practice experience, like making the mistake, or through careful study in controlled situations like learning all the wrong ways to work the problem.

The Challenging Ascent

Getting over that wall isn’t easy. Your brain doesn’t want to struggle past the right way to do things. It craves challenge and novelty. You’re going to have to work against your better nature to get to a point where you’re past the wall. Don’t be afraid to lie to yourself to get where you need to be.

When running I will trick myself when I hit my mental wall by saying “one more song” or “one more block” when I’m ready to give up. The idea that I can make it a short distance or short amount of time is comforting to my brain when it wants to stop. And by tricking it I can often push a little harder to another song or two more blocks before I get completely over the wall and have the mental toughness to continue.

Likewise, when you’re studying and you’ve found the correct answer you need to push yourself to find one incorrect way at first. Maybe a second. If it’s something that has configurable settings you should investigate a few wrong values to figure out what happens when things are outside of bounds or when they’re just a little bit off. Maybe convince yourself to figure out two or three and write down the results. If one of them ends up being really interesting it could spark you to do more investigation to find out what caused that particular outcome.

You’ll find that you can get past your mental blocks much easier with tricks like that. More importantly, you’ll also find that you can get them to pop up faster and be overcome with less effort as you understand when they happen. If you’ve ever sat down to study something and your brain immediately wants to give up you know that the wall is right in front of you. How you overcome it can mean the difference between truly understanding a topic and just knowing enough about the answer to regurgitate it later.

Tom’s Take

As always, your mileage may vary with skills like these. I’d wager that most people do hit a wall whether it’s running or doing math or studying the intricacies of how OSPF works over non-broadcast networks. Don’t settle for your brain telling you that you’re done. Seek to really put in the work and understand what’s going on. Write everything down so you know what you’ve discovered. And when that wall seems like it’s too high to climb just whisper to yourself you’re going to climb another foot. And then another. And pretty soon you’ll be over and in the clear.

Networking Is Fast Enough

Posted on by
2

Without looking up the specs, can you tell me the PHY differences between Gigabit Ethernet and 10GbE? How about 40GbE and 800GbE? Other than the numbers being different do you know how things change? Do you honestly care? Likewise for Wi-Fi 6, 6E, and 7. Can you tell me how the spectrum changes affect you or why the QAM changes are so important? Or do you want those technologies simply because the numbers are bigger?

The more time I spend in the networking space the more I realize that we’ve come to a comfortable point with our technology. You could call it a wall but that provides negative connotations to things. Most of our end-user Ethernet connectivity is gigabit. Sure, there are the occasional 10GbE cards for desktop workstations that do lots of heavy lifting for video editing or more specialized workflows like medical imaging. The rest of the world has old fashioned 1000Mb connections based on 802.3z ratified in 1998.

Wireless is similar. You’re probably running on a Wi-Fi 5 (802.11ac) or Wi-Fi 6 (802.11ax) access point right now. If you’re running on 11ac you might even be connected using Wi-Fi 4 (802.11n) if you’re running in 2.4GHz. Those technologies, while not quite as old as GigE, are still prevalent. Wi-Fi 6E isn’t really shipping in quantity right now due to FCC restrictions on outdoor use and Wi-Fi 7 is a twinkle in hardware manufacturers’ eye right now. Why aren’t we clamoring for more, faster, better, stronger all the time?

Speedometers

How fast can your car go? You might say you’ve had it up to 100 mph or above. You might take a look at your speedometer and say that it can go as high as 150 mph. But do you know for sure? Have you really driven it that fast? Or are you guessing? Would you be shocked to learn that even in Germany, where the Autobahn has an effectively unlimited speed limit, that cars are often limited to 155 mph?. Even though the speedometer may go higher the cars are limited through an agreement for safety reasons. Many US vehicles are also speed limited between 110 and 140 mph.

Why are we restricting the speeds for these vehicles? Safety is almost always the primary concern, driven by the desire for insurance companies to limit claims and reduce accidents. However, another good reason is also why the Autobahn has a higher effective speed limit: road conditions. My car may go 100 mph but there are very few roads in my part of the US that I would feel comfortable going that fast on. The Autobahn is a much better road surface for driving fast compared to some of the two-lane highways around here. Even if the limit was higher I would probably drive slower for safety reasons. The roads aren’t built for screaming speeds.

That same analogy applies to networking. Sure, you may have a 10GbE connection to your Mac Mini and you may be moving gigs of files back and forth between machines in your local network. What happens if you need to upload it to Youtube or back it up to cloud storage? Are you going to see those 10GbE speeds? Or are you going to be limited to your ISP’s data rates? The fastest engine can only go as fast the pathways will permit. In essence, that hot little car is speed limited because of the pathway the data takes to the destination.

There’s been a lot of discussion in the space about ever-increasing connectivity from 400GbE to 800GbE and soon even into the terabit range. But most of it is specialized for AI workloads or other massive elephant flows that are delivered via a fabric. I doubt an ISP is going to put in an 800GbE cross connect to increase bandwidth for consumers any time soon. They won’t do it because they don’t need to. No consumer is going to be running quite that fast.

Likewise, increasing speeds on wireless APs to more than gigabit speeds is silly unless you want to run multiple cables or install expensive 10GbE cards that will require new expensive switches. Forgetting Multigig stuff for now you’re not going to be able to plug in a 10GbE AP to an older switch and get the same performance levels. And most companies aren’t making 10GbE campus switches. They’re still making 1GbE devices. Clients aren’t topping out their transfer rates over wireless. And even if they did they aren’t going to be going faster than the cable that plugs the AP into the rest of the network.

Innovation Idling

It’s silly, right? Why can’t we make things go faster?!? We need to use these super fast connections to make everything better. Yet somehow our world works just fine today. We’ve learned to work with the system we have. Streaming movies wouldn’t work on a dial-up connection but adding 10GbE connections to the home won’t make Netflix work any faster than it does today. That’s because the system is optimized to deliver content just fast enough to keep your attention. If the caching servers or the network degrades to the point where you have to buffer your experience is poor. But so long as the client is getting streaming data ahead of you consuming it you never know the difference, right?

Our networks are optimized to deliver data to clients running on 1GbE. Without a massive change in the way that workloads are done in the coming years we’re never going to be faster than that. Our software programs might be more optimized to deliver content within that framework but I wouldn’t expect to see 10GbE become a huge demand in client devices. Frankly, we don’t need that much speed. We don’t need to run flat out all the time. Just like a car engine we’re more comfortable running at a certain safe speed that preserves our safety and the life of the equipment.

Tom’s Take

Be honest with yourself. Do you want 10GbE or Wi-Fi 7 because you actually need the performance? Or do you just want to say you have the latest and greatest? Would you pay extra for a v12 engine in a sports car that you never drive over 80 mph? Just to say you have it? Ironically enough, this is the same issue that cloud migrations face today. We buy more than we need and never use it because we don’t know what our workloads require. Instead, we buy the fastest biggest thing we can afford and complain that something is holding it back. Rather than rushing out to upgrade your Wi-Fi or Ethernet, ask yourself what you need, not what you want. I think you’ll realize the network is fast enough for the foreseeable future.

Argument Farming

Posted on by
Reply

The old standard.

I’m no stranger to disagreement with people on the Internet. Most of my popular posts grew from my disagreement with others around things like being called an engineer, being a 10x engineer, and something about IPv6 and NAT. I’ve always tried to explain my reasoning for my positions and discuss the relevant points with people that want to have a debate. I tend to avoid commenting on people that just accuse me of being wrong and tell me I need to grow up or work in the real world.

Buying the Farm

However, I’ve noticed recently that there have been some people in the realm of social media and influencing that have taken to posting so-called hot takes on things solely for the purpose of engagement. It’s less of a discussion and more of a post that outlines all the reasons why a particular thing that people might like is wrong.

For example, it would be like me posting something about how an apple is the dumbest fruit because it’s not perfectly round or orange or how the peel is ridiculous because you can eat it. While there are some opinions and points to be made, the goal isn’t to discuss the merits of the fruit hierarchy. Instead, it’s designed to draw in people that disagree to generate comments about how apples are, in fact, good fruits and maybe if I tried one some time I would understand. In this example, I would reply to the comment with something along the lines of “thanks for your perspective” or maybe even a flippant question about why you think that way to keep the chain going.

I’ve found that this is very prevalent on platforms that reward engagement over content. Facebook and LinkedIn chiefly spring to mind. The content of the message isn’t as important as how people react to it. The reward isn’t a well-reasoned discussion. It’s people sharing your post and telling you how stupid you are for making it. Or trying to change your mind.

Except I know what I’m doing. I may not even have strongly held beliefs on my post. I may even prefer apples to oranges. The point is to get you all in an uproar and make you drive my post to the top of someone’s feed. A contrarian way to look at things for sure. But it works. Because we’ve rewarded people for making a splash instead of making a case.

Crop Rotation

In the 10x engineer post I linked above, I had no intention of it blowing up. I noticed some things that irked me about the culture we’ve created around the people that do a lot and how we worship their aura without examining the downsides. Naturally, that meant that it got picked up on Hacker News and there were a raft of comments about how I was an idiot and how I’d get fired if I worked for a “real” company because I wasn’t pulling my weight.

I was horrified, to say the least. I didn’t want that kind of engagement. I wanted a reasoned discussion. I wanted people to see my points and engage in debate. I certainly wasn’t trying to specifically craft a post with a contrarian viewpoint explicitly designed to incense the community to drive them to my page or blog. Yet that is exactly how I’m seeing some members of the wider community acting today. The clicks are more important than the words. And if you end up being proven wrong? So be it. Whoops. On to the next hot take!

I wish I had a better method for dealing with this new angle other than just ignoring it. If it’s someone with a legitimate bad viewpoint that could use some guidance or education I am happy to chip in and provide a different viewpoint. However the difference between the occasional post and constant engagement farming for arguments in the comments to drive your view counts higher is disingenuous. Disagreeing with something is one thing. Writing 400 words about how it’s the “worst mistake you can make” or “you should think about what that will mean for your career” are a bit heavy handed. And yes, I’ve seen both of those statements in recent months about something as innocuous as a training class.

Tom’s Take

Healthy disagreement and debate makes us improve. Honest mistakes happen and can be corrected. I have no issue with either of these, even if both sides will never agree. What I take issue with is people being deliberately disingenuous to manipulate algorithms or manufacture outrage for their own ends. I always come back to a simple question: Are you doing this to solve a problem? Or become popular? If the answer is the latter it might be time to put down the plow and ask yourself if the crop you’re sewing is worth it.

Changing Diapers, Not Lives

Posted on by
Reply

When was the last time you heard a product pitch that included words like paradigm shift or disruptive or even game changing? Odds are good that covers the majority of them. Marketing teams love to sell people on the idea of radically shifting the way that they do something or revolutionizing an industry. How often do you feel that companies make something that accomplishes the goal of their marketing hype? Once a year? Once a decade? Of the things that really have changed the world, did they do it with a big splash? Or was it more of a gradual change?

Repetition and Routine

When children are small they are practically helpless. They need to be fed and held and have their diapers changed. Until they are old enough to move and have the motor functions to feed themselves they require constant care. In fact, potty training is usually one of the last things on that list above. Kids can feed themselves and walk places and still be wearing diapers. It’s just one of those things that we do as parents.

Yet, changing diapers represents a task that we usually have no issue with. Sure it’s not the most glamorous work. But it’s necessary. Children can’t do it themselves. Maybe they can take off a wet or soiled diaper on their own (my kids did on occasion), but they can’t quite put one on. We encourage them to conform to the societal norm of using a bathroom instead of using a disposable diaper.

I use changing diapers as a metaphor for something we do regularly that is thankless but necessary. Kids never thank you for changing their diapers when they get older but it needs to be done. You may not think it’s a life-changing experience at the time but you know it’s one small part of what needs to happen to make them better as people later on. As a company that is trying to change people’s lives with the products you’re selling you often aim toward the sky. You want a utopia of flying cars and automated homes and AI-driven everything. But do your customers want that?

Your customers don’t want self-driving cars. They want to not have to spend their time driving. They don’t want AI-powered dinner ordering. They want to not have to make dinner decisions. Your customers don’t want a magical dashboard that makes automatic configuration changes for them. They want to operate their systems without constant attention to every little detail to keep them from falling apart. They don’t want revolutionary. They want relief.

Aim Small, Miss Small

If your first thought when building a product is “we’re going to change the world!” then you need to stop back because you missed the target. One of smartest things I overheard regarding startups was “Don’t solve a problem. Solve a problem someone has every day.” People are so focused on making an impact a revolutionizing the world they often miss the opportunity to do something that really does change things by simply solving common problems that happen all the time.

When you go back to your vision, think about changing diapers, not lives. Think about solving the problems people have every day. Take network automation, for example. You’re not going to create a paradigm shifting organizational restructuring in a day or a week or even a year. What you can do is automate things like password changes or switch deployments. You can solve that everyday problem so there is more time to work on other things. You can remove errors and create responsiveness where it didn’t exist before. Sure, your Ansible script that provisions a switch isn’t going to get your name etched in stone in Silicon Valley. But it can lead to changes in the organization that create efficiency and make your team happier and more focused on solving other hard problems.

Likewise, if you tell someone your product is going to change their life they will probably laugh at you or shake their head in disbelief. After all, everything promises to change their lives. However, if you tell them your product will solve a specific issue they have then they are very likely to take you up on it. Your target market will identify what you do and respond positively. Rather than trying to boil an ocean with hype you’re providing clear messaging on what you can do and how it can help. People want that clarity over hype.

Tom’s Take

If you try to promise me a life-changing experience with an app or a piece of hardware I’m going to make sure you understand what that means and what it takes. On the other hand, if you come to me with a proposal to change something I dislike doing every day or simplifying it in some way I’m more likely to listen to your pitch. Changing lives is hard. Changing diapers is not fun but it is necessary and repetitive. Focus on the small things and make those easier to do before you take on the rest of the world. Your customers will be happier and you will too.

Don’t Let the Cybersecurity Trust Mark Become Like Food Labeling

Posted on by
Reply

I got several press releases this week talking about the newest program from the US Federal government for cybersecurity labeling. This program is something designed to help consumers understand how secure IoT devices are and the challenges that can be faced trying to keep your network secure from the large number of smart devices that are being implemented today. Consumer Reports has been pushing for something like this for a while and lauded the move with some caution. I’m going to take it a little further. We need to be very careful about this so it doesn’t become as worthless as the nutrition labels mandated by the government.

Absolute Units

Having labels is certainly better than not having them. Knowing how much sugar a sports drink has is way more helpful than when I was growing up and we had to guess. Knowing where to find that info on a package means I’m not having to go find it somewhere on the Internet1. However, all is not sunshine and roses. That’s because of the way that companies choose to fudge their numbers.

Food companies spent a lot of time trying to work the numbers on those nutrition labels for years. The most common way to do it is to adjust the serving size listed on the box. For example, a 20-ounce soda bottle isn’t a single serving of liquid. It’s 2.5 servings at 8 ounces each. In order to find the true nutritional value of the whole bottle you need to read close enough to do the math and find out it’s more sugar and calories than you were expecting. The whole game was so bad the FDA forced companies to change labeling in 2022.

One of the other ways that labeling guidelines have allowed companies to get away with misinformation is through clever interpretation. Did you know that TicTacs are sugar free? If you look at the nutritional label information they contain zero sugar despite being made of nothing but sugar. How can they accurately say that? Because the serving size is so small it rounds down to zero. You’re probably groaning now but this is what has happened for years unless some group steps in to fix the issue.

The Fine Print

Now let’s look at how this could be adapted to go horribly wrong with IoT devices. One of the simple ways that I could see it being an issue is with something like a baby monitor. These devices are usually low-cost and don’t have much security built in. If you know the address of the device you can often connect to it and watch the video feed. Adding more software controls on top of the hardware is going to increase the price significantly. So are the manufacturers going to add pricey software to meet labeling guidelines? Or are they going to pull a TicTac? Say, for example, labeling the device as secure against remote access with an asterisk saying it’s only secure if you turn off the Wi-Fi and only look at it in the same room?

The label is going to be a valuable thing to add to the box to differentiate the product from competitors. Given the choice between a box without a label and one with a label, which one would you pick Tommy boy? That being said, how far do you think someone would go to put the label on the box? The program is voluntary but it still has requirements that need to be met. Someone could potentially create specific scenarios that allow them to meet the guidelines under specific circumstances and include the label despite not being the most secure device.

If the government wants to ensure that users aren’t getting attacked and have their data stolen, they need to put explicit guidelines in place to specify how the labels need to be created. No creative interpretation. No asterisks or fine print. It needs to be a table that has simple answers. If you don’t meet the guidelines you don’t get the check mark. Don’t let the manufacturers interpret your rules in their favor. It’s a bit more of a pain for those administering the program but a little sweat equity up front is going to be more comforting than the news articles after the fact.

Tom’s Take

I want this program to work. I really do. I also know how capitalism works. Companies are going to work this label as much as possible in their favor, including some creative thoughts on the requirements. I’d rather have some fusing now that leads to proper implementation in the future than lots of bad press about how the labels are worthless. If the industry is going to take steps to make things better for consumers let’s make sure it’s really better and not some sugar-free version.

  1. Provided the packaging is big enough for it to be printed, that is. ↩︎

Cross Training for Career Completeness

Posted on by
1

Are you good at your job? Have you spent thousands of hours training to be the best at a particular discipline? Can you configure things with your eyes closed and are finally on top of the world? What happens next? Where do you go if things change?

It sounds like an age-old career question. You’ve mastered a role. You’ve learned all there is to learn. What more can you do? It’s not something specific to technology either. One of my favorite stories about this struggle comes from the iconic martial artist Bruce Lee. He spent his formative years becoming an expert at Wing Chun and no one would argue he wasn’t one of the best. As the story goes, in 1967 he engaged in a sparring match with a practitioner of a different art and, although he won, he was exhausted and thought things had gone on far too long. This is what encouraged him to develop Jeet Kun Do as a way to incorporate new styles together for more efficiency and eventually led to the development of mixed martial arts (MMA).

What does Bruce Lee have to do with tech? The value of cross training with different tech disciplines is critical for your ability to continue to exist as a technology practitioner.

Time Marches On

A great example of this came up during Mobility Field Day back in May. During the Fortinet presentation there was a discussion about wireless and SASE. I’m sure a couple of the delegates were shrugging their shoulders in puzzlement about this inclusion. After all, what does SASE have to do with SNR or Wi-Fi 6E? Why should they care about software running on an AP when the real action is in the spectrum?

To me, as someone who sees the bigger picture, the value of talking about SASE is crucial. Access points are no longer radio bridges. They are edge computing devices that run a variety of software programs. In the old days it took everything the CPU had to process the connection requests and forward the frames to the right location. Today there is a whole suite of security being done at the edge to keep users safe and reduce the amount of traffic being forwarded into the network.

Does that mean that every wireless engineer needs to become a security expert? No. Far from it. There is specialized knowledge in both areas that people will spend years perfecting. Does that mean that wireless people need to ignore the bigger security picture? That’s also a negative. APs are going to be running more and more software in the modern IT world because it makes sense to put it there and not in the middle of the enterprise or the cloud. Why process traffic if you don’t have to?

It also means that people need to look outside of their specific skillset to understand the value of cross training. There are some areas that have easy crossover potential. Networking and wireless have a lot of commonality. So do storage and cloud, as well as virtualization and storage and cloud. We constantly talk about the importance of including security in the discussion everywhere, from implementation to development. Yet when we talk about the need to understand these technologies at a basic level we often face resistance from operations teams that just want to focus on their area and not the bigger picture.

New Approaches

Jeet Kune Do is a great example of why cross training has valuable lessons for us to learn about disruption. In a traditional martial arts fight, you attack your opponent. The philosophy of Jeet Kun Do is to attack your opponent’s attacks. You spend time defending by keeping them from attacking you. That’s a pretty different approach.

Likewise, in IT we need to examine how to we secure users and operate networks. Fortinet believes security needs to happen at the edge. Their philosophy is informed by their expertise in developing edge hardware to do this role. Other companies would say this is best performed in the cloud using their software, which is often their strength. Which approach is better? There is no right answer. I will say that I am personally a proponent of doing the security stuff as close the edge as possible to reduce the need for more complexity in the core. It might be a remnant of my old “three tier” network training but I feel the edge is the best place to do the hard work, especially given the power of the modern edge compute node CPU.

That doesn’t mean it’s always going to be the best way to do things. That’s why you have to continuously learn and train on new ways of doing things. SASE itself came from SD-WAN which came from SDN. Ten years ago most of this was theoretical or in the very early deployment stage. Today we have practical applications and real-world examples. Where will it go in five years? You only know if you learn how it works now.

Tom’s Take

I’ve always been a voracious learner and training myself on different aspects of technology has given me the visibility to understand the importance of how it all works together. Like Bruce Lee I always look for what’s important and incorporate it into my knowledge base and discard the rest. I know that learning about multiple kinds of technology is the key to having a long career in the industry. You just have to want to see the bigger picture for cross training to be effective.

Disclaimer: This post mentions Fortinet, a presenter at Mobility Field Day 9. The opinions expressed in this post reflect my own perspective and were not influenced by consideration from any companies mentioned.

My Belated Review of Cisco Live 2023

Posted on by
Reply

It’s been a couple of weeks since Cisco Live US 2023 and I’m just now getting around to writing about it. I was thrilled to attend my 18th Cisco Live and it was just the thing I needed to reconnect with the community. The landscape of Cisco Live looks a little different than it has in years past. There are some challenges that are rising that need to be studied and understood before they become bigger than the event itself.

Showstopping Reveals? Or Consistent Improvement?

What was the big announcement from Cisco this year? What was the thing that was said on stage that stopped the presses and got people chattering? Was it a switch? A firewall? Was it a revolutionary new AI platform? Or a stable IP connection to Mars? Do you even know? Or was it more of a discussion of general topics with some technologies brought up alongside them?

In the last few years you may have noticed that the number of huge big announcements coinciding with the big yearly conferences has come down a bit. Rather than having some big news drop the morning of the keynote the big reveals are being given their own time to shine instead. Rather than piling up tons of news of acquisitions or new product releases and watching them all get lost in the shuffle of fanfare they’re now being spaced out or bunched up at the end of quarters instead.

The big keynotes are instead being used to push initiatives. Rather than talking products the companies are talking strategies. Things like sustainability and outreach replace speeds and feeds. The goal isn’t to show off something shiny but instead to show off what the goal is to utilize the new products. Those kinds of announcements tend to play better with the press and analysts as well as the investors.

Does that mean that we’re never going to see another big announcement during an event keynote? No. What it does mean is that you shouldn’t expect to see groundbreaking shifts happening during those discussions. Steady and predictable is what the investors like. And during those keynotes that’s what you’re going to see for the most part.

Community Marches On

Social media sure has been fun for the past few months wouldn’t you say? The decline of Twitter, the rise of Mastodon and BlueSky, and even more craziness all over the place. Proof? Check out my badge from Cisco Live this year:

Yes, I needed all of those flags to show people where I was posting things to social media. And keeping track of all of the communities can be tiring. Some people still use Twitter because it’s there. Some people have embraced the Fediverse and deleted Twitter altogether. Others are trying out BlueSky and finding their groove again. And that doesn’t even discuss the number of people that are embracing video platforms or other means of posting. It is a certainty that the former king of the hill is rolling down very quickly in the face of so many other options.

One thing that I loved is that the community around Cisco Live has endured through so much upheaval. As soon as we arrived on site it was just like old times. People coordinated hangouts and invited friends all over. Parties were held. Introductions were made. And people caught up as if they hadn’t seen each other in forever. It made me happy to see that the impending collapse of a social platform didn’t affect the people that used it to build a great group.

Another thing that I realized when I got to the event was that this was the tenth anniversary of the Cisco Live Social Media Hub. I can still vividly remember when I walked into the convention center in Orlando in 2013 to find this brand new area dedicated for us to hang out and enjoy a little spotlight. Over the years the hub has grown from just a few tables and some laptops to an entire control center that serves as a central meeting location for folks as well as a set for some creative content to be made. I remember on more than one occasion seeing folks running around staging shots for a TikTok video and seeing lots of extra content being posted from everywhere. It’s good when you don’t have to make your own little space.

Tom’s Take

What does the future of Cisco Live look like? Is it going to continue to be a huge draw for people to come and enjoy the community? Is Cisco going to keep releasing new products and making this a destination for networking professionals? Given the number of attendees increased again this year I’d say that there is definitely a desire for people to attend conferences in person again. Given that the community has continued to persevere through all manner of challenges I’d say they’re also here to stay as well. All in all, I’m glad to see Cisco Live has continued to see success. As long as we temper our expectations for what the conference will be in the future and continue to keep the community alive then I don’t see any challenges that can’t be overcome.

Using AI for Attack Attribution

Posted on by
Reply

While I was hanging out at Cisco Live last week, I had a fun conversation with someone about the use of AI in security. We’ve seen a lot of companies jump in to add AI-enabled services to their platforms and offerings. I’m not going to spend time debating the merits of it or trying to argue for AI versus machine learning (ML). What I do want to talk about is something that I feel might be a little overlooked when it comes to using AI in security research.

Whodunnit?

After a big breach notification or a report that something has been exposed there are two separate races that start. The most visible is the one to patch the exploit and contain the damage. Figure out what’s broken and fix it so there’s no more threat of attack. The other race involves figuring out who is responsible for causing the issue.

Attribution is something that security researchers value highly in the post-mortem of an attack. If the attack is the first of its kind the researchers want to know who caused it. They want to see if the attackers are someone new on the scene that have developed new tools and skills or if it is an existing person or group that has expanded their target list or repertoire. If you think of a more traditional definition of crime from legal dramas and police procedurals you are wondering if this is a one-off crime or if this is a group expanding their reach.

Attribution requires analysis. You need to look for the digital fingerprints of a group in the attack patterns. Did they favor a particular entry point? Are they looking for the same kinds of accounts to do privilege escalation? Did they deface the web servers with the same digital graffiti? For attackers looking to make a name for themselves, attribution is pretty easy to figure out. They want to make a splash. However, for state-sponsored crews or organizations looking to keep a low profile it is much more likely they’re going to obfuscate their methods to avoid detection as long as possible. They might even throw out a few red herrings to make people attribute the attack to a different group.

Picking Out Patterns

If the methodology of doing attribution requires pattern matching and research, why not use AI to assist? We already use AI and ML to help us detect the breaches. Why not apply it to figuring out who is doing the breaching? We already know that AI can help us identify people based on a variety of characteristics. Just look up any kind of market research done by advertising agencies and you can see how scary they can predict buyer behavior based on all kinds of pattern recognition.

Let’s apply that same methodology to attack attribution. AI and ML are great at not only sifting through the noise when it comes to pattern recognition but they can also build a profile of the patterns to confirm those suspicions. Imagine profiling an attacker by seeing that they use one or two methods for gaining entry, such as spearphishing, to gain access to start privilege escalation. They always go after the same service accounts and move laterally to the same servers after gaining it. This is all great information for predicting attacks and stopping them. But it’s super valuable for tracking down who is doing it.

Assuming that crews bring new attackers on board frequently to keep their crime pipeline full you can also see how much of the attack profile is innate talent versus training. One could assume that these organizations aren’t terribly different from your average IT shop when it comes to training. It’s just the result of that training that differs. If you start seeing a large influx of attacks that use repetition of similar techniques from different locations it could be assumed that there is some kind of training going on somewhere in the loop.

The other thing that provides value is determining when someone is trying to masquerade as a different group using techniques to obfuscate or misattribute breaches. Building a profile of an attacker means you know how long it takes them to move to new targets or how likely they are to take certain actions within a specific window. If you work out the details of an attack you can see quickly if someone is following a script or if they’re doing something in a specific way to make it look like someone else is trying to get in. This especially applies at the level of nation-state sponsored groups, since creating doubt in the attribution can prevent your detection or even cause diplomatic sanctions against the wrong country.

Of course, the real challenges is that AI and ML aren’t foolproof. They aren’t the ultimate arbiter of attack recognition and attribution. Instead, they are tools that should be introduced into the kit to help speed identification and provide assurances that you’ve got the right group before you publicize what you’ve found.

Tom’s Take

There’s a good chance that some security companies out there are already looking at or using AI to do attribution. I think it’s important to broaden our toolkits and use of models in all areas of cybersecurity. It also provides a baseline for creating normalized investigation. There have been too many cases where a researcher has rushed to pin attribution on a given group only to find out it wasn’t them at all. Using tools to confirm your suspicions not only reduces the likelihood you will name the wrong attacker but it also reduces the need to publicize quickly to claim credit for the identification. This should be about protection, no publicity.

Time Is Not On Your Side

Posted on by
Reply

It’s been almost five years since I wrote about the challenges of project management and timing your work as an engineer. While most of that information is still very true even today I’ve recently had my own challenges with my son’s Eagle Scout project. He is of a mind that you can throw together a plan and just do a whole week of work in just a couple of days. I, having worked in the IT industry for years, have assured him that it absolutely doesn’t work like that. Why is there a disconnect between us? And how does that disconnect look to the rest of the world?

Time Taking You

The first problem that I often see when working with people that aren’t familiar with projects is that they vastly underestimate the amount of time it takes to get something done. You may recall from my last post that my project managers at my old VAR job had built in something they called Tom Time to every quote. That provided a way for my estimate to reflect reality once I arrived on site and found the things didn’t go according to plan.

Part of the reason why my estimates didn’t reflect reality was because there are a lot of things that go into a project that can’t quite be explained or calculated into the final estimate. For example, how long does it take for a switch to reboot? Some of them can be ready to pass traffic in a couple of minutes. Larger devices that need to test modules may take up to ten minutes to be ready to go. If you have to reboot that switch multiple times during your project how do you account for that time? Is there a line item for a hour’s worth of switch reboots? What about the project closeout meetings a paperwork? How do you build that into a project timeline?

People that underestimate the timeline of a project are almost always only focused on the work. They see that it should take them about five minutes to copy the config the switch and ten minutes to put it in the rack. Did they think about the time to unbox it? Cable it? Do a final test to ensure all configuration is correct and saved to the startup config? Each of these things sound trivial but they add time. Maybe you don’t do the final config test and hope for the best. But you can’t shave time on unboxing unless you have someone helping you do that. Which, of course, just adds time to the project in a different way.

The Price of Time

Does this mean that you just need to increase the amount of time that you put on a project? No, it doesn’t. One of the connectivity providers I worked with in the past had what they called a “foolproof method” of getting the right time estimate for a circuit. They doubled the number and increased to the next time unit. So two hours became four days. Three days became six weeks. And I became infuriated when I realized how much time something like this would take.

Part of the reasoning behind that thinking was that the project management overhead always took longer than expected. But the other thinking was that quoting much longer timelines gave them more room to cram in too much work for a single team. They could juggle deployments because they had enough hours in the quote that they could be more interrupt driven. Work on something until someone complains then move to that project and work on it until the complaining stops. You can see why providers like that quickly get a reputation for padding their projects.

Time costs money. Either someone is paying you to do the job or you’re paying for that resource to be unavailable for doing the job. You have to learn how to allocate your resources effectively. If you need to help your teams or your contractors understand the additional time that it takes to do a project you need to either package that time as a line item or educate them about what additional tasks you see. Accounting for that extra time is a better way to show value than just adding lots of extra wiggle room to a project so you don’t go over budget. The education aspect is especially important for talent that isn’t familiar with things from the outset. Teaching them how to look for those time sinks and making sure they’re tracked means their estimates will be much more accurate in the future.

Tom’s Take

My son is going to complete his project but he’s going to learn a lot about the way the world works in the process. Paint doesn’t dry overnight. It takes time to load and unload lumber. People need more than 24 hours notice to show up to work on something. These are all lessons I’ve learned over the years that I’m happy to teach. Time is important to us all because we don’t get any more of it. Every minute that goes by is a minute we can’t get back. Make the most of your time by tracking it appropriately and building those hidden things into your project estimates. That’s how you get time to be on your side for once.