Trust Will Do You In

If you’re a fan of the Gestalt IT Rundown that I do every week on the Gestalt IT YouTube channel, you have probably heard about the recent hacks of NVIDIA and Samsung. The original investigation into those hacks talked about using MDM platforms and other vectors to gain access to the information that was obtained by the hacking groups. An interesting tweet popped up on my feed yesterday that helped me reframe the attacks:

It would appear that the group behind these attacks are going after their targets the old fashioned way. With people. For illustration, see XKCD from 2009:

The Weakest Links

People are always the weakest link in any security situation. They choose to make something insecure through bad policy or by trying to evade the policy. Perhaps they are trying to do harm to the organization or even try to shine a light on corrupt practices. Whatever the reason, people are the weak link. Because you can change hardware or software to eliminate failures and bugs. You can’t reprogram people.

We’ve struggled for years to keep people out of our systems. Perimeter security and bastion hosts were designed to make sure that the bad actors stayed off our stage. Alas, as we’ve gotten more creative about stopping them we’ve started to realize that more and more of the attacks aren’t coming from outside but instead from inside.

There are whole categories of solutions dedicated to stopping internal attackers now. Data Loss Prevention (DLP) can catch data being exfiltrated by sophisticated attackers but it is more often used to prevent people from leaking sensitive data either accidentally or purposefully. There are solutions to monitor access to systems and replay logs to find out how internal systems folks were able to get privileges they should have.

To me, this is the crux of the issue. As much as we want to create policies that prevent people from exceeding their authority we seem to have a hard time putting them into practice. For every well-meaning solution or rule that is designed to prevent someone from gaining access to something or keep them secure you will have someone making a shortcut around it. I’ve done it myself so I know it’s pretty common. For every rule that’s supposed to keep me safe I have an example of a way I went around it because it got in my way.

Who Do YOU Trust?

This is one of the reasons why a Zero Trust Network Architecture (ZTNA) appeals to me. At its core it makes a very basic assumption that I learned from the X-Files: Trust No One. If you can’t verify who you are you can’t gain access. And you only gain access to the bare minimum you need and have no capability for moving laterally.

We have systems that operate like this today. Call your doctor and ask them a question. They will first need to verify that you are who you say you are with information like your birthdate or some other key piece of Personally Identifiable Information (PII). Why? Because if they make a recommendation for a medication and you’re not the person they think they’re talking to you can create a big problem.

Computer systems should work the same way, shouldn’t they? We should need to verify who we are before we can access important data or change a configuration setting. Yet we constantly see blank passwords or people logging in to a server as a super user to “just make it easier”. And when someone gains access to that system through clever use of a wrench, as above, you should be able to see the potential for disaster.

ZTNA just says you can’t do that. Period. If the policy says no super user logins from remote terminals they mean it. If the policy says no sensitive data access from public networks then that is the law. And no amount of work to short circuit the system is going to work.

This is where I think the value of ZTNA is really going to help modern enterprises. It’s not the nefarious actor that is looking to sell their customer lists that creates security issues. It does happen but not nearly as often as an executive that wants a special exception for the proxy server because one of their things doesn’t work properly. Or maybe it’s a developer that created a connection from a development server into production because it was easier to copy data back and forth that way. Whatever the reasons the inadvertent security issues cause chaos because they are configured and forgotten. At least until someone hacks you and you end up on the news.

ZTNA forces you to look at your organization and justify why things are the way they are. Think of it like a posture audit with immediate rule creation. If development servers should never talk to production units then that is the rule. If you want that to happen for some strange reason you have to explicitly configure it. And your name is attached to that configuration so you know who did it. Hopefully something like this either requires sign off from multiple teams or triggers a notification for the SOC that then comes in to figure out why policy was violated. At best it’s an accident or a training situation. At worst you may have just caught a bad actor before they step into the limelight.


Tom’s Take

Security isn’t perfect. We can always improve. Every time we build a good lock someone can build a better way to pick it. The real end goal is to make things sufficiently secure that we don’t have to worry about them being compromised with no effort while at the same time keeping it easy enough for people to get their jobs done. ZTNA is an important step because it creates rules and puts teeth behind them to prevent easy compromise of the rules by people that are trying to err on the side of easy. If you don’t already have plans to include ZTNA in your enterprise now you really should start looking at them. I’d tell you to trust me, but not trusting me is the point.

AI is a Promotion

When I worked at IBM as an intern, part of my job was writing a deployment script to help make our lives easier when installing new ThinkPads. In order to change an MTU setting on the token ring PCMCIA cards (long story), I had to write a script that iterated through all the possible combinations of adapters in the registry to find the one I was looking for and change the value.

Now, I was 22 at the time and green behind the ears, especially when it came to programming. I finally figured out that the most efficient way to do this in the language that I was using was a very deep nested if statement. It wasn’t my best work but it operated properly. I mentioned this to my mentors on my team with a remark of how hard it was to understand the logic at first. My comment was “You know, if it’s hard to read for anyone else then I never have to worry about gettin fired.”

To which the response was, “Yes, but you can never be promoted either.”

That sage wisdom brings me to the modern world and how AI can fix that issue.

Eternal Support

One of the first things you learn in IT is not to volunteer. If you know how to implement a technology or troubleshoot a specific issue then you become the go-to person for that problem. If you add a skill to the team like debugging Rust or configuring voice mail dial peers then you are the new owner for that particular skill.

This becomes annoying when you finally move past that skillset and you’re on to new and better things. If you are the person that knows how to extend database schemas or change auto attendant functions you are always going to be called upon to do those things. Rather than moving on to new challenges you will be called back to work on the things that no one else bothered to learn or couldn’t pawn off on someone else.

When you think about it, that’s part of the reason why so many entry-level positions start with some basics like VLAN programming or LUN configuration. Sure, you’re learning the basics and understanding how systems operate. But you’re also repeating the things that the junior person before you did. Why? Because they now have someone to do that work for them. And when the new person gets hired as a junior-level technician, you’re going to pass along those tasks to someone so they can learn and so you can move on. It’s the Circle of IT Life.

The reinforcement of knowledge is only so important to the organization. Yes, you need someone to keep the lights on and do the tasks regarded as menial by the senior teams. However, miring people in those tasks makes them feel undervalued after a while because no one wants to take them on. And, as above, if it’s something that no one else can be bothered to learn or take over you’re going to be stuck doing it until you leave or until the system in question is eventually replaced. And even then you’ll probably be asked to learn how to do the new thing because “this is your system”.

Learning, Not Thinking

The reason why I’m excited for applied AI in infrastructure and operations is because it will learn these menial tasks and do them as instructed every time without complaint or mistake. Yes, that means you need to train the algorithm correctly. Yes, that means you need to be on the lookout for changes and update accordingly when hardware changes or when procedure changes. Given those constraints AI will work for you, not against you.

When you move on to a new role you expect to face new challenges. If you move from a junior role to a senior role you want to work on harder things or solve new problems. You don’t want to get a new title with the same old work. If you change teams you want to experience new systems and implement new technology. You don’t want to be the cloud person that still deploys campus switches in wiring closets.

In a way, having AI take over the basic tasks of your job functions doesn’t remove your job. It promotes you to a new role. It could be that the first promotion is running the system that you implemented to do the work. A “train the trainer” kind of role, if you will. As soon as you can ensure that it is working correctly and that the rest of the team is aware of how to keep the engine running you can decide where you want to go next. You essentially have the freedom to learn and do and not have to be concerned about going back to doing the old things.

Is AI the Enemy?

AI doesn’t take away jobs. It takes away tasks. It streamlines processes and provides consistent, repeatable outcomes. If your job is a collection of tasks that need to be done then it is worth asking why it’s so easy for it to be replaced by an AI system. I’ve said on a number of occasions that there will always been room for jobs that AI can’t do, such as those that involve physical labor, as well as advanced thinking and creative problem solving.

Where AI does succeed is in the repeatable task department. If a role is comprised of repeatable tasks then the question becomes “why does this role exist?” If an algorithm can determine the best way to implement VLANs or offload clients to other devices then the job that was doing those roles needs to be examined. Can that person be encourage to do something else for the company? Can that role be reduced and given to someone with less experience? If those questions make you or someone on your team bristle, is that the fault of the AI? Or the fact of the job being less than ideal for the business?


Tom’s Take

AI can take away some of the unimportant and forgettable parts of your job and give you the freedom to do things that are more important to the business, possibly in a different role. Isn’t that the textbook definition of a promotion? Change makes people uncomfortable. It makes sense that no one wants to have their job taken away. But if you ask anyone in IT right now if they could get rid of 10-20% of the tasks in their job that they don’t like to do they’d agree in a heartbeat. AI isn’t going to remove your job. It’s going to give you the freedom to do something different. It’s also going to ensure you don’t have to keep getting called back to do that thing over and over again because no one else wants to learn it. I think it’s time we let new technologies like AI give everyone on the team the promotion they deserve.

The Value of Old Ideas

I had a fun exchange on Twitter this week that bears some additional thinking. Emirage (@Emirage6) tweeted a fun meme about learning BGP:

I retweeted it and a few people jumped in the fun, including a couple that said it was better to configure BGP for reasons. This led to a blog post about routing protocols with even more great memes and a good dose of reality for anyone that isn’t a multi-CCIE.

Explain It Like I’m Five

I want you to call your mom and explain BGP to her. Go on and do that now because I’m curious to see how you’d open that conversation. Unless your mom is in networking already I’m willing to bet you’re going to have to start really, really basic. In fact, given the number of news organizations that don’t even know what the letters in the acronym stand for I’d guess you are going to have a hard time talking about the path selection process or leak maps or how sessions are established.

Now, try that same conversation with RIP. I bet it goes a lot smoother. Why? Because RIP is a simple protocol. We don’t worry about prefixes or AS Path prepending or other things when we talk about RIP. Because RIPv1 is the most basic routing protocol you can get. It learns about routes and sends the information on. It’s so simple that you can usually get all of the info about it out of the way in a couple of hours of instruction in a networking class.

So why do we still talk about RIP? It’s so old. Even the second version of the protocol is better. There’s no intelligence. No link state. You can’t even pick a successor route! We should never talk about RIPv1 again in any form, right? So what would you suggest we start with? OSPF? BGP?

The value of RIP is not in the protocol. Instead, the value is in the simplicity of the discussion. If you’ve never heard of a routing protocol before you don’t want to hear about all the complexity of something that runs half the Internet. At least you don’t want to hear about it at first. What you need to hear is how a routing protocol works. That’s why RIP is a great introduction. It does the very minimal basic functions that a routing protocol should do. It learns routes and tells other routers about them.

RIP is also a great way to explain why other routing protocols were developed. We use OPSF and EIGRP and other IGRPs now because RIP doesn’t perform well outside of small networks. There are limitations around IP subnets and network diameter and “routing by rumor” that only make sense when you know how routing protocols are supposed to operate and how they can fall down. If you start with OSPF and learn about link-state first then you don’t understand how a router could ever have knowledge of a route that it doesn’t know about directly or learn about from a trusted source. In essence, RIP is a great first lesson because it is both bad and good.

The Expert’s Conundrum

The other issue at hand is that experts tend to feel like complicated subjects they understand are easy to explain. If that’s the case, then the Explain Like I’m Five Reddit shouldn’t exist. It turns out that trying to explain a complex topic to people without expert knowledge is super hard because they lack the frame or reference you need to help them understand. If you are a network engineer that doesn’t believe me then go ask a friend in the medical profession to explain how the endocrine system works. Don’t make they do a simple explanation. Make them tell you like they’d teach a medical student.

We lose sight of the fact that complex topics can’t be mastered quickly and certain amounts of introductory knowledge need to be introduced to bring people along on the journey. We teach about older models and retired protocols because the knowledge they contain can help us understand why we moved away from them. It also helps us to have a baseline level of knowledge about things that permeate the way we do our jobs.

If we removed things that we never use from the teaching curriculum we’d never have the OSI model since it is never implemented in the pure form in any protocol. We’d teach the TCP/IP model and just tell people that the other things don’t really matter. In fact, they do matter because the pure OSI model takes other things into account that aren’t just focused on the network protocol stack. It may seem silly to us as experts to say that we teach one thing but reality looks much different but that’s how learning works.

We still teach older methods of network topologies like bus and ring even though we don’t use them much any more. We do this because entry-level people need to know why we arrived at the method we use now. Even with the move toward new setups like 3-tier network design and leaf/spine architecture you need to know where Ethernet started to understand why we are where we are today.


Tom’s Take

It’s always important to get a reminder that people are just starting out in the learning process. While it’s easy to be an expert and sit back and claim that iBGP is the best way to approach a routing protocol problem you also have to remember that learners sometimes just need a quick-and-dirty lab setup to test another function. If they’re going to spend hours configuring BGP neighbor relationships instead of just enabling RIP on all router interfaces and moving on to the next part then they’re not learning the right things. Knowledge is important, even if it’s outdated. We still teach RIP and Frame Relay and Token Ring in networking because people need to understand how they operate and why we’ve moved on. They may not ever configure them in practice but they may also never configure BGP either. The value of information doesn’t decrease because it’s old.

Technical Debt or Underperforming Investment?

In this week’s issue of the Packet Pushers Human Infrastructure newsletter, there was an excellent blog post from Kam Lasater about how talking about technical debt makes us sound silly. I recommend you read the whole thing because he brings up some very valid points about how the way the other departments of the organization perceive our issues can vary. It also breaks down debt in a very simple format that takes it away from a negative connotation and shows how debt can be a leverage instrument.

To that end, I want to make a modest proposal to help the organization understand the challenges that IT faces with older systems and integration challenges. Except we need some new branding. So, I propose we start referring to technical debt as “underperforming technical investments”.

I’d Buy That For A Dollar

Technical debt is just a clever way to refer to the series of layered challenges we face from decisions that were made to accomplish tasks. It’s a burden we carry negatively throughout the execution of our job because it adds extra time to the process. We express it as debt because it’s a price that must be paid every time we need to log in to the old accounting system to make changes or we have to wake up on the third Sunday of the month to reboot the access points because of a software bug that we can’t fix.

As Lasater points out, the financial departments look at debt differently. It’s not always negative. Commercial paper allows for purchasing power beyond cash-on-hand. Mortgages and loans allow us to buy houses and cars that enable us to do much more with our lives. Reasonable debt can be good so long as what we get out of it is a net positive. Debt doesn’t become an issue until there’s too much of it for the return we get, which eventually leads to bankruptcy.

Let’s point out some of the challenges we face with technical things that we refer to as debt. We usually use it in reference to outdated systems or slower equipment that doesn’t allow us to do our jobs as effectively as possible. Unlike a house which still provides the value of a dwelling at any age given reasonable maintenance or a car that still operates with the same maintenance, technology changes more rapidly. The original iPhone is unable to operate in a modern environment because of outdated software or inability to connect to the mobile network.

In consumer tech, the idea of technical debt doesn’t exist. Because when things are “slow” or “broken” they get discarded and people spend money to get something new. Someone that is willing to live with a shattered phone screen because they don’t want to pay to fix it might be willing to wait another two months to get a new device because it supports the latest OS updates or has a better camera. The chase for new features is usually enough of a driver to get people to upgrade.

In the enterprise, the infrastructure equipment is expensive enough that we need to recognize the value of what we installed. We can’t just swap out all the switches in three years if we spent hundreds of thousands of dollars on them. Likewise, enterprise tech vendors are more likely to provide patches and updates for this gear knowing that customers are going to hold on to it for longer periods of time. And even then, just because something is EoL doesn’t mean it’s End of Use.

However, the disconnect between these two things comes when we talk about how equipment that is functional is also causing issues with our job. Maybe it’s an older, slower switch that drops more packets than the rest of the newer gear. It’s a problem but not big enough to justify buying a new one. It could be an older wireless controller that doesn’t have the capability to run the newest code because of an older CPU. We have gear that works but it doesn’t work as well as it could. But we also don’t have enough justification to get new stuff because the old things haven’t broken completely yet.

Putting Your Money Where Their Mouth Is

As Lasater says in the above article, talking about how it’s hard to do your job because everything is old doesn’t resonate with management. No one really wants to hear a department head whining about how their job is tough because they don’t have the latest toys. Admit it, you’d be upset if your CEO told you they couldn’t answer their email because they don’t have the newest MacBook Pro. Capable means functional in every enterprise.

However, I think the challenge we face and the solution to the problem is how we frame the discussion of the challenge itself. We talk about it as a debt that we feel saddled with. It’s something we did because we had to or something someone else chose to do that we have to live with. It feels like we’re paying a tax every day over something that we don’t like. But choices need to be made and we have to work with what we have. Unless we can provide for a way to get people to understand the tradeoffs.

Hence, my proposal above. First, we stop referring to IT infrastructure gear as “technical debt”. It’s not something we bought and now have to deal with. Our accounting departments are amortizing the acquisition costs of the equipment over a period of time and writing it off of taxes against recognized revenue. There is a real dollar amount attached to every month of the lifetime of a switch or a server. That’s how finance sees your equipment. It’s not debt.

It’s a technical investment.

That’s your key to helping them understand the challenges you face. You’re not complaining about doing your job more slowly. Instead, you’re pointing out that you are investing a resource (your time) into a process that could have reduced resource requirements (again, your time and job execution time) if they would just invest in a different solution.

You’re not telling them they bought a lemon. Instead, you’re just telling them that the thing they invested in years ago isn’t performing as well as it could based on new data. If you need to break it down even further for your executive team you can just equate it to a home loan refinancing. If interest rates have dropped since you bought your home, doesn’t it make a ton of sense to refinance and take advantage of the savings? Likewise, if the speed or power of a system has increased in the past four years doesn’t it make sense to invest in something new?

When you change the discussion to focus on investment of resources instead of dealing with debt you change the way that the people will look at the return on that investment. If you just talk about debt the executives are going to easily be able to dismiss you because there is no upside to the conversation. They have to pay money to replace things so why spend it? If you talk about how much they could save on labor or how much more efficiently things could run if they made some changes or bought something new then that is harder to ignore. If their existing investment is underperforming some baseline on the stock market they’re going to want to move it right away. If the company’s stock is underperforming you can better believe some of the shareholders are going to want to move their investment too.

Here Comes the Money

One word of warning here. You can’t go in to this conversation with vague assurances that spending more money will make money. You have to come with numbers that support your case. Some of them are easy. You know what the new system will cost. You know how much time it will take to implement it. What you need to bring is the cost of what you are paying now to do things inefficiently.

If your team of 5 has to spend two extra hours a week doing something because the system is old or doesn’t work well with others then you have ten hours of lost time. You need to figure out what the value of your time is. Big note here: Don’t just divide your yearly salary by 2000 and take the averages. That’s what you’re getting paid. That’s the debt that the company incurs to keep you on staff. Instead, find out what your rate is. If you are billed at a higher rate than your earnings, which is almost a guarantee, use that number instead. If you’re an engineer for a reseller your hourly billing rate is easy to find. If you work in corporate IT the rate is harder to figure out but if you need a rule of thumb just figure out what your hourly pay rate is (yearly salary divided by 2000 hours per year) and double it. That’s the kind of return that any company would love to have on an investment. If it’s costing the company that much to do something inefficiently then you can make your case to get it fixed or replaced.

Lastly, make sure to record the changes and the results if you manage to make it happen. It’s easy to chart when you take more time to do something to prove a point. But it’s easy to forget to do it when you have what you want. Eventually, someone that manages investments needs to know if it paid off. If there’s a simple number like a stock price it’s cut-and-dried. If there’s more to the calculation you need to do the work to prove why it’s better now. And you want to keep a running total so you can provide it on demand when asked. Just tell them you have more time to track those kinds of things now that they made a wiser investment.


Tom’s Take

No one likes debt. Even the best debt is still an obligation to pay back to someone. However, investments are something positive. Yes, they still require resources to be exchanged to receive a good or service. However, changing the terminology changes the perception of the result. Debts are incurred and paid back begrudgingly. Investments grow and provide additional resources throughout their lives. And, importantly, when an investment isn’t paying off the way to fix it is to reinvest. It’s a clever trick but one that should work much better than just whining about drowning in debt.

Who Wants to Be Supported Forever?

I saw an interesting thread today on Reddit talking about using networking equipment past the End of Life. It’s a fun read that talks about why someone would want to do something like this and how you might find yourself in some trouble depending on your company policies and such. But I wanted to touch on something that I think we skip over when we get here. What does the life of the equipment really mean?

It’s a Kind of Magic

As someone that uses equipment of all kinds, the lifetime of that equipment means something different for me than it does for vendors. When I think of how long something lasts I think of it in terms of how long I can use it until it is unable to be repaired any further. A great example of this is a car. All of my life I have driven older used cars that I continue to fix over and over until they have a very high mileage or my needs change and I must buy something different.

My vehicles don’t have a warranty or any kind of support, necessarily. If I need something fixed I either fix it myself or I take it to a mechanic to have it fixed and pay the costs associated with that. I’m not the kind of person that will get rid of a car just because the warranty has run out and it’s no longer under support.

The market of replacement parts and mechanics is made for people like me. Why buy something new when you can buy the parts to repair what you have? Sadly, that market exists for automobiles but not for IT gear. It’s hard to pull a switch apart and resolder capacitors when you feel like it. Most of the time you want to replace something that has failed and is end-of-life, you need to have a spare of the same kind of equipment sitting on the shelf. You can put the configuration files you need on it and put it in place and just keep going. The old equipment is tossed aside or recycled in some way.

Infrastructure gear will work well past the End of Support (EoS) or End of Life (EoL) dates. It will keep passing packets until something breaks, either the hardware in the ports or the CPU that drives it. Maybe it’s even a power supply. Usually it’s not the end of the life of the hardware that causes the issues. Instead, it’s the software concerns.

Supported Rhapsody

I’d venture a guess that most people reading this blog have some kind of mobile device sitting in their junk drawer or equipment box that works perfectly fine yet isn’t used because the current version of software doesn’t support that model. Software is a bigger determining factor the end of life of IT equipment than anything else.

Consumer gear needs to be fast to keep users happy. Phones, laptops, and other end user devices have to keep up with the desires of the people running applications and playing games and recording video. These devices iterate quickly and fall out of support just as fast. Your old iPhone or Galaxy probably won’t run the latest code which has a feature you really want to use so you move on to something newer.

Infrastructure gear doesn’t have software go out of fashion quite as quickly as a mobile phone but it does have another issue that causes grief: security patches. You may not be chasing a new feature in a software release but you’re either looking to patch a bug that is affecting performance or patch a security hole that has the potential to cause issues for your organization.

Security patches for old hardware are a constant game of cat-and-mouse. Whether it’s an old operating system running important hardware terminals or the need to keep an old terminal running because the application software doesn’t work on anything newer IT departments are often saddled with equipment that’s out of date and insecure. Do you keep running it, hoping that there will still be patches for it? Or do you try to move to something newer and hope that you can make things work? Anyone that is trying to use software that requires versions of Microsoft Internet Explorer knows that pain can last for years.

The real issue here doesn’t have anything to do with support of the equipment or the operating system. Instead, it has everything to do with the support that you can provide for it. A mechanic isn’t magical. They learn how to get parts for cars and fix the problems as they arise. They become the support structure instead of the dealership that offers the warranty. Likewise, as an IT pro you become the support structure for an old switch or old operating system. If you can’t download patches for it you either need to provide a workaround for issues or find a way to create solutions to address it. It’s not end of support as much as it’s end of someone else’s support.


Tom’s Take

I use things until they don’t work. Then I fix them and use them more. It’s the way I was raised. I have a hard time getting new things simply because they’re supported or a little bit faster. I might upgrade my phone to get a new feature but you can bet I’m going to find a way to use my old one longer, either by giving it to one of my kids or using it in a novel way. I hate getting rid of technology that has some use left in it. End of Life for me really does mean the moment when it stops sending bits or storing data. As long as there is no other conflict in the system you can count on me extending the life of a device long past the date that someone says they don’t want to deal with it any more.

Backing Up the Dump Truck

Hello Ellen,

 

I have received a number of these spam messages over the past few weeks and I had hoped they would eventually taper off. However, it doesn’t appear that is the case. So I’ll take the direct approach.

 

I’m a member of the CCIE Advisory Council. Which means I am obligated to report any and all attempts to infringe upon the integrity of the exam. As you have seen fit to continue to email me to link to your site to promote your test dumps I think you should be aware that I will be reporting you to the CCIE team.

 

Good luck in your future endeavors after they shut you down for violating their exam terms and conditions. And do not email me again.

That’s an actual email that I sent TODAY to someone (who probably isn’t really named Ellen) that has been spamming me to link to their CCIE dump site. The spam is all the same. They really enjoy reading a random page on my site, usually some index page picked up by a crawler. They want me to insure a link to their site which is a brain dump site for CCIE materials, judging by the URL I refuse to click on. They say that if I am not interested that I should just ignore it, which I have been doing for the past two months. And that brings us to today.

Setting the Record Straight

Obviously, the company above is just spamming any and all people with reputable blogs to help build link credibility. It’s not a new scam but one that is pervasive in the industry. It’s one of the reasons why I try to be careful about which links I include in my posts. And I never accept money or sponsorship to link to something. Where appropriate I include information about disclosures and such.

What makes this especially hilarious is that I’m a pretty public member of the CCIE Advisory Council. I’ve been a part of it for almost three years at this point. You would think someone would have a little bit of logic in their system to figure this out. That’s like sending a pirated copy of an ebook to the author. Maybe revenue is down and they need to expand. Maybe they’re looking for popular networking bloggers. Who knows? Maybe they really like poking bears.

What is certain is that I wanted everyone to know that this goes on. And that I’m going to do something about it at the very least. I will report this person’s site, which I will not link to since it won’t be up much longer, and ensure that this crap stops. It’s not just the annoying spam. It’s the fact that they can be this brazen about looking for link karma for a dump site from someone that has the most investment in not having dumps out there.

Don’t buy dumps. You’re not doing yourself any favors. Learn the material. Learn the process. Learn why things work. When you do this you learn how to handle situations and all their permutations. You don’t just think that the answer to a routing protocol redistribution problem is just “B”. You should check out any reputable CCIE training vendor out there first. It’s going to cost you more than the dumps but you’re getting more for your money. Trust me on that.

Moreover, if you get these kinds of emails as a writer or podcaster, don’t accept them. By linking back to these sites you’re adding a portion of your clout and goodwill to them. When (and it’s always when) they get shut down, you take a hit from being associated with them. Don’t even give them the time of day. I had been ignoring this spam for quite a while in the hopes that this group would get the picture, especially based on their text that says ignoring it would make it go away. Alas for them, they pushed one time too many and found themselves on the wrong side of a poked bear.


Tom’s Take

Okay, rant over. This is stuff that just rubs me the wrong way. Not only because they don’t take silence for a hint but because they’re just trading on the good name of other networking bloggers in the hopes of making a few quick bucks before getting shut down and moving on to the next enterprise. I’m going to push back on this one. And the next one and the one after that. It may not amount to much in the long run but maybe it’s the start of something.

The Network Does Too Much

I’m at Networking Field Day this week and it’s good to be back in person around other brilliant engineers and companies. One of the other fun things that happens at Networking Field Day is that I get to chat with folks that help me think about things in new ways and come up with awesome ideas for networking blog posts.

One of the ones that was discussed quickly this week really got me thinking again about fragility and complexity. Thanks to Carl Fugate for reminding me about it. Essentially, networks are inherently unstable because they are doing far too much heavy lifting.

Swiss Army Design

Have you heard about the AxeSaw Reddit? It’s a page dedicated to finding silly tools that attempt to combine too many things together into one package that make the overall tool less useful. Like making a combination shovel and axe that isn’t easy to operate because you have to hold on to the shovel scoop as the handle for the axe and so on. It’s a goofy take on a trend of trying to make things too compact at the sake of usability.

Networking has this issue as well. I’ve talked about it before here but nothing has really changed in the intervening time since that post five years ago. The developers that build applications and systems still rely on the network to help solve challenges that shouldn’t be solved in the network. Things like first hop reachability protocols (FHRP) are perfect examples. Back in the dark ages, systems didn’t know how to handle what happened when a gateway disappeared. They knew they needed to find a new one eventually when the MAC address age timed out. However, for applications that couldn’t wait there needed to be a way to pick up the packets and keep them from timing out.

Great idea in theory, right? But what if the application knew how to handle that? Things like Cisco CallManager have been able to designate backup servers for years. Applications built for the cloud know how to fail over properly and work correctly when a peer disappears or a route to a resource fails. What happened? Why did we suddenly move from a model where you have to find a way to plan for failure with stupid switching tricks to a world where software just works as long as Amazon is online?

The answer is that we removed the ability for those stupid tricks to work without paying a high cost. You want to use Layer 2 tricks to fake NHRP? If it’s even available in the cloud you’re going to be paying a fortune for it. AWS wants you to use their tools that they optimize for. If you want to do things the way you’ve always done you can but you need to pay for that privileges.

With cost being the primary driver for all things, increased costs for stupid switching tricks have now given way to better software development. Instead of paying thousands of dollars a month for a layer 2 connection to run something like HSRP you can instead just make the application start searching for a new server when the old one goes away. You can write it to use DNS instead of IP so you can load balance or load share. You can do many, many exciting and wonderful things to provide a better experience that you wouldn’t have even considered before because you just relied on the networking team to keep you from shooting yourself in the foot.

Network Complexity Crunch

If the cloud forces people to use their solutions for reliability and such, that means the network is going to go away, right? It’s the apocalypse for engineer jobs. We’re all going to get replaced by a DevOps script. And the hyperbole continues on.

Reality is that networking engineers will still be as needed as highway engineers are even though cars are a hundred times safer now than in the middle of the 20th century. Just because you’ve accommodated for something that you used to be forced to do doesn’t mean you don’t need to build the pathway. You still need roads and networks to connect things that want to communicate.

What it means for the engineering team is an increased focus on providing optimal reliable communications. If we remove the need to deal with crazy ARP tricks and things like that we can focus on optimizing routing to provide multiple paths and ensure systems have better communications. We could even do crazy things like remove our reliability of legacy IP because the applications will survive a transition when they aren’t beholden to IP address or ARP to prevent failure.

Networking will become a utility. Like electricity or natural gas it won’t be visible unless it’s missing. Likewise, you don’t worry about the utility company solving issues about delivery to your home or business. You don’t ask them to provide backup pipelines or creative hacks to make it work. You are handed a pipeline that has bandwidth and the service is delivered. You don’t feel like the utility companies are outdated or useless because you’re getting what you pay for. And you don’t have to call them every time the heater doesn’t work or you flip a breaker. Because that infrastructure is on your side instead of theirs.


Tom’s Take

I’m ready for a brave new world where the network is featureless and boring. It’s an open highway with no airbags along the shoulder to prevent you from flying off the road. No drones designed to automatically pick you up and put you on the correct path to your destination if you missed your exit. The network is the pathway, not the system that provides all the connection. You need to rely on your systems and your applications to do the heavy lifting. Because we’ve finally found a solution that doesn’t allow the networking team to save the day we can absolutely build a world where the clients are responsible for their own behavior. The network needs to do what it was designed to do and no more. That’s how you solve complexity and fragility. Less features means less to break.

The Demise of G-Suite

In case you missed it this week, Google is killing off the free edition of Google Apps/G-Suite/Workspace. The short version is that you need to convert to a paid plan by May 1, 2022. If you don’t you’re going to lose everything in July. The initial offering of the free tier was back in 2006 and the free plan hasn’t been available since 2012. I suppose a decade is a long time to enjoy custom email but I’m still a bit miffed at the decision.

Value Added, Value Lost

It’s pretty easy to see that the free version of Workspace was designed to encourage people to use it and then upgrade to a paid account to gain more features. As time wore on Google realized that people were taking advantage of having a full suite of 50 accounts and never moving, which is why 2012 was the original cutoff date. Now there has been some other change that has forced their hand into dropping the plan entirely.

I won’t speculate about what’s happening because I’m sure it’s complex and tied to ad revenue and privacy restrictions that people are implementing that is reducing the value of the data Google has been mining for years. However, I can tell you that the value of what they’re offering with their entry-level business plan isn’t as valuable as they might think.

The cheapest Google Workspace plan available is $6 per user per month. It covers the email and custom domain that was the biggest attraction. It also has a whole host of other features:

  • Google Meet, which I never use when Zoom/Webex/Teams exist
  • Google Drive, which is somewhat appealing except for when no one wants to use Docs or Sheets and Dropbox is practically a standard and still free
  • Chat, which makes me laugh because it’s probably the next thing to get killed off in favor of some other messaging platform that will get abandoned in six months

Essentially, Google is hoping to convince me to pay for their service that has been free this entire time by giving me things I don’t use now and probably won’t use in the future? Not exactly a good selling model.

Model Citizens

I’ve heard there is a plan for trying to give loyal customers a discount for the first year of service to ease the transition but that’s not going to cut it for most. Based on the comments I’ve seen most people are upset that they have purchases from Google tied to an account they can’t transfer away from as well as the possibility that whatever happens next is going to be shut down anyway. I mean, Killed by Google is starting to look like a massive graveyard at this point.

I’m willing to concede at this point that the free tier of Google Workspace is gone and won’t be coming back. What I’m not ready to give in on is the model that you’re forcing me to pay for and use other services because you have a target revenue number to hit and you keep throwing useless stuff in to make it seem valuable. You’re not transitioning us to a new model. You’re ramming the existing one down our throats because you need users for those other services that are paying.

Want to extend some goodwill to the community? Offer us a solution that gives us what we want for a reasonable pricing model? How about email without video chat and Drive for $3 per month per user? How about allowing me to cut out the junk and reduce my spend. How about giving me something with other value based on how I use your service and not how you think I should be using it?


Tom’s Take

I realize I’m just tilting at windmills in this whole mess but It’s frustrating. I’m totally prepared to never see a resolution to this issue because Google has decided it’s time to kill it off. Yes, I got a decade of free email hosting out of the deal. I got a lot of value for what I invested. I even realize that Google can’t keep things free forever. I just wish there was a way for me to pay them for what I want and not have to pay more for things that are useless to me. Technology marches on and new models always supplant old ones. The only constant is change. But change is something we should be able to process and accept. Not have it forced upon us to ensure someone is using Google Meet.

Wi-Fi 6 Release 2, Or Why Naming Conventions Suck

I just noticed that the Wi-Fi Alliance announced a new spec for Wi-Fi 6 and Wi-Fi 6E. Long-time readers of this blog will know that I am a fan of referring to technology by the standard, not by a catch term that serves as a way to trademark something, like Pentium. Anyway, this updated new standard for wireless communications was announced on January 5th at CES and seems to be an entry in the long line of embarrassing companies that forget to think ahead when naming things.

Standards Bodies Suck

Let’s look at what’s included in the new release for Wi-Fi 6. The first and likely biggest thing to crow about is uplink multi-user MIMO. This technology is designed to enhance performance and reduce latency for things like video conferencing and uploading data. Essentially, it creates multi-user MIMO for data headed back the other direction. When the standard was first announced in 2018 who knew we would have spent two years using Zoom for everything? This adds functionality to help alleviate congestion for applications that upload lots of data.

The second new feature is power management. This one is aimed primarily at IoT devices. The combination of broadcast target wake time (TWT), extended sleep time, and multi-user spatial multiplexing power save (SMPS) are all aimed at battery powered devices. While the notes say that it’s an enterprise feature I’d argue this is aimed at the legion of new devices that need to be put into deep sleep mode and told to wake up at periodic intervals to transmit data. That’s not a laptop. That’s a sensor.

Okay, so why are we getting these features now? I’d be willing to bet that these were the sacrificial items that were holding up the release of the original spec of 802.11ax. Standards bodies often find themselves in a pickle because they need to get the specifications out the door so manufacturers can start making gear. However, if there are holdups in the process it can delay time-to-market and force manufacturers to wait or take a gamble on the supported feature set. And if there is a particular feature that is being hotly debated it’s often dropped because of the argument or because it’s too complex to implement.

These features are what has been added to the new specification, which doesn’t appear to change the 802.11ax protocol name. And, of course, these features must be added to new hardware in order to be available, both in radios and client devices. So don’t expect to have the hot new Release 2 stuff in your hands just yet.

A Marketing Term By Any Other Name Stinks

Here’s where I’m just shaking my head and giggling to myself. Wi-Fi 6 Release 2 includes improvements for all three supported bands of 802.11ax – 2.4GHz, 5GHz, and 6GHz. That means that Wi-Fi 6 Release 2 supersedes Wi-Fi 6 and Wi-Fi 6E, which were both designed to denote 802.11ax in the original supported spectrums of 2.4 and 5GHz and then to the 6GHz spectrum when it was ratified by the FCC in the US.

Let’s all step back and realize that the plan to simplify the naming convention of the Wi-Fi alliance for marketing has failed spectacularly. In an effort to avoid confusing consumers by creating a naming convention that just counts up the Wi-Fi Alliance has committed the third biggest blunder. They forgot to leave room for expansion!

If you’re old enough you probably remember Windows 3.1. It was the biggest version of Windows up to that time. It was the GUI I cut my teeth on. Later, there were new features that were added, which meant that Microsoft created Windows 3.11, a minor release. There was also a network-enabled version, Windows for Workgroups 3.11, which included still other features. Was Windows 3.11 just as good as Windows for Workgroups 3.11? Should I just wait for Windows 4.0?

Microsoft fixed this issue by naming the next version Windows 95, which created a bigger mess. Anyone that knows about Windows 95 releases know that the later ones had huge new improvements that made PCs easier to use. What was that version? No, not Windows 97 or whatever the year was. No, it was Windows 95 OEM Service Release 2 (Win95OSR2). That was a mouthful for any tech support person at the time. And it showed why creating naming conventions around years was a dumb idea.

Now we find ourselves in the mess of having a naming convention that shows major releases of the protocol. Except what happens when we have a minor release? We can’t call it by the old name because people won’t be impressed that it contains new features. Can we add a decimal to the name? No, because that will mess up the clean marketing icons that have already been created. We can’t call it Wi-Fi 7 because that’s already been reserved for the next protocol version. Let’s just stick “release 2” on the end!

Just like with 802.11ac Wave 2, the Wi-Fi Alliance is backed into a corner. They can’t change what they’ve done to make things easier without making it more complicated. They can’t call it Wi-Fi 7 because there isn’t enough difference between Wi-Fi 6 and 6E to really make it matter. So they’re just adding Release 2 and hoping for the best. Which will be even more complicated when people have to start denoting support for 6GHz, which isn’t universal, with monikers like Wi-Fi 6E Release 2 or Wi-Fi 6 Release 2 Plus 6E Support. This can of worms is going to wiggle for a long time to come.


Tom’s Take

I sincerely hope that someone that advised the Wi-Fi Alliance back in 2018 told them that trying to simplify the naming convention was going to bite them in the ass. Trying to be cool and hip comes with the cost of not being able to differentiate between minor version releases. You trade simplicity for precision. And you mess up all those neat icons you built. Because no one is going to legitimately spend hours at Best Buy comparing the feature sets of Wi-Fi 6, Wi-Fi 6E, and Wi-Fi 6 Release 2. They’re going to buy what’s on sale or what looks the coolest and be done with it. All that hard work for nothing. Maybe the Wi-Fi Alliance will have it figured out by the time Wi-Fi 7.5 Release Brown comes out in 2025.

Make Sure You Juggle The Right Way in IT

When my eldest son was just a baby, he had toys that looked like little baseballs. Long story short, I decided to teach myself to juggle with them. I’d always wanted to learn and thought to myself “How hard can it be?” Well, the answer was harder than I thought and it took me more time that I realized to finally get the hang of it.

One of the things that I needed to learn is that adding in one more ball to track while I’m trying to manage the ones that I had wasn’t as simple as it sounded. You would think that adding in a fourth ball should only be about 25% harder than the three you had been working with before. Or, you might even believe the statistical fallacy that you’re only going to fail about a quarter of the time and be successful the rest. The truth is that adding in one more object makes your entire performance subpar until you learn to adjust for it.

Clogging Up the Pipe

I mention this example because the most obvious application for the juggling metaphor is in Quality of Service (QoS). If you’ve ever read any of the training material related to QoS over the years, you’ll know that an oversubscribed link doesn’t perform poorly for the packets that are added in at the end. When a link hits the point of saturation all of the data flowing down the pipe is impacted in some way, whether it’s delays or or dropped packets or even application timeouts.

We teach that you need to manage congestion on the link as a whole and not just the data that is added that takes you over the stated rate. This is why we have queuing methods that are specifically tuned for latency sensitive traffic like voice or video. You can’t assume that traffic that gets stuffed in at the start will be properly handled. You can’t assume that all data is just going to line up in an orderly fashion and wait its turn. Yes, the transmission queue on the device is going to process the packets in a serial manner, but you can’t know for sure what packets are going to be shoved in the queue without some form of management.

It’s important to understand that QoS is about the quality of the experience for all consumers of the link and not just a select group. That’s why texts will teach you about priority queuing methods and why they’re so inefficient. If the priority queues are the only ones getting served then the regular queues will fail to send traffic. If users get creative and try to mark their packets as priority then the priority queue becomes no better than the regular queue.

QoS for Your Brain

All of these lessons for juggling packets and prioritizing them within reason don’t just resonate with technology. The same principles apply to the work you do and the projects and tasks that you take on. I can’t tell you the number of times I’ve thought to myself “I can just handle this one little extra thing and it won’t make a big difference.” Except it does make a big difference in the long run. Because adding one more task to my list is just like adding one more ball to the juggling list. It’s not additive. It adds a whole new dimension to what you’re working on.

Just like with the bandwidth example, the one extra piece added to the end makes the whole experience worse overall. Now you’re juggling more than you can handle. Instead of processing what you have efficiently and getting things done on time you’re flipping back and forth trying to make sure that all the parts are getting worked on properly and, in the end, using too much time inefficiently. Add in the likelihood that this new task is “important” and gets placed near the top of the list and you can quickly see how the priority queue example above is fitting. When every task is critical, there are no critical tasks.

Prioritize Before The Piles Happen

As luck would have it, the best way to deal with these issues of juggling too many tasks is the same as dealing with oversubscription on a link. You need to understand what your ability to deal with tasks looks like. Maybe you can handle eight things a day. Are those eight complex things? Eight easy things? Four of each? You need to know what it takes to maximize your productivity. If you don’t know what you can handle then you’ll only find out you’re oversubscribed when you take on one thing too many. And it’s too late to turn back after that.

Next, you need to manage the tasks you have in some way. Maybe it’s a simple list. But it’s way easier if the list has a way to arrange priority and deal with complicated or less critical tasks after the important stuff is done first. Remember that something being complex and critical is going to be a challenge. Easy tasks can be knocked out and crossed off your list sooner. You can also make sure that tasks that need to happen in a certain order are arranged in that way.

Lastly, you need to model the QoS drop method. Which means saying “no” to things that are going to oversubscribe you. It seems inelegant and will lead to others getting frustrated that you can’t get the work done. However, they also need to understand that if you can’t get the work done because you’re tasked with too much you’re going to do a poor job anyway. It’s better to get things done in a timely manner and tell people to come back later than take on more than you can do and disappoint everyone. And if someone tried to get creative and tell you their task is too important to put off, remind them that every task is critical to someone and you decide how important things are.


Tom’s Take

This is absolutely a case of “do as I say, not as I do”. I’m the world’s worst for taking on more than I can handle to avoid making other people feel disappointed. No matter how many times I remind myself that I can’t take on too much I have been known to find myself in a situation where I’m oversubscribed and my performance is suffering because of it. Use this as an opportunity to get a better handle on juggling things on your side. I never got good enough to juggle more than four at once and I’m okay with that. Don’t feel like you have to take on more than you can or else you’ll end up working in a circus.