Author Archives: networkingnerd

Unknown's avatar

About networkingnerd

Tom Hollingsworth, CCIE #29213, is a former network engineer and current organizer for Tech Field Day. Tom has been in the IT industry since 2002, and has been a nerd since he first drew breath.

Ease of Use or Ease of Repair

Posted on by
1

HammerAndSaw

Have you tried to repair a mobile device recently? Like an iPad or an MacBook? The odds are good you’ve never even tried to take one apart, let alone put in replacement parts. Devices like these are notorious to try and repair because they aren’t designed to be fixed by a normal person.

I’ve recently wondered why it’s so hard to repair things like this. I can recall opening up my old Tandy Sensation computer to add a Sound Blaster card and some more RAM back in the 90s but there’s no way I could do that today, even if the devices were designed to allow that to happen. In my thinking, I realized why that might be.

Build to Rebuild

When you look at the way that car engine bays were designed in the 80s and 90s you might be surprised to see lots of wasted space. There’s room to practically crawl in beside the engine and take a nap. Why is that? Why waste all that space? Well, if you’re a mechanic that wants to get up close and personal with some part of the engine you want all the space you can find. You’d rather waste a little space and not have busted knuckles.

A modern engine isn’t designed to be repaired by amateurs. The engine may be sideways or have replaceable parts put in places that aren’t easily accessed without specialized equipment. The idea is that professionals would have everything they need to remove the engine if necessary. Moreover, by making it harder for amateurs to try and fix the issues you’re are preventing them from doing something potentially problematic that you’ll have to fix later.

The same kind of thing applies to computing equipment. Whereas before you could open the case of a computer and try to fix the issues inside with a reasonable level of IT or electrical engineering knowledge you now find yourself faced with a compact device that needs specialized tools to open and even more specialized knowledge to fix. The average electrical engineer is going to have a hard time understanding the modern system-on-a-chip (SoC) designs when you disassemble your phone.

Usability Over Uniqueness

The question of usability is always going to come into play when you talk about devices. The original IBM PC was a usable design for the time it was built compared to mainframes. The iPhone is a usable design for the time when it was built as well. But compared to each other, which is the more usable design today?

Increasing usability often comes at the cost of flexibility. I was talking to someone at a fabrication lab about printer drivers for a laser etching machine. He said that installing the drivers for the device on Windows is easy but almost impossible on a Mac. I chuckled because the difficulty of installing a non-standard printer on a Mac is part of the reason why it’s easy to use for other kinds of printers. The printing subsystem traded usability with things like Bonjour printer installation over flexibility, like installing strange printers on a laptop.

Those trade offs extend into the hardware. Anyone that worked on PCs prior to Windows 95 knows all about fun things like IRQ conflicts, DIP switch settings for COM port addresses, and even more esoteric problems. We managed to get rid of those in the modern computing era but along with that went the ability to customize the device. You no longer have to do the dance of finding an available IRQ but you also can’t install a device that would conflict with another through hardware trickery.

By creating devices for the average user focused on usability you have to sacrifice the flexibility that makes them easier to fix. If you knew you were constantly going to need to be tweaking the jets on a carburetor you’d make it easy to access, right? But if a modern fuel injection system never needs to be adjusted except by a specialized professional why would you make it accessible to anyone that doesn’t have tools? You can see this in systems that use proprietary screws to keep users out of the system or glue parts together to prevent easy access.

Tom’s Take

I miss the ability to open up my laptop and fix issues with the add-in cards. The tinkerer in me likes to learn about how a system works. But I don’t miss the necessity to go in and do it. The usability of a system that “just works” is way more useful to me. It reminds me of my dad in some ways. He may have loved the ability to open up the engine and play with the carburetor but he also had to do it frequently to make things “work right”. In some ways, removing our ability to repair things in the IT space has forced manufacturers to build devices that don’t need to be frequently repaired. It’s not the optimal solution for sure, but the trade off it more than worth it in my mind.

In Defense of Subscriptions

Posted on by
Reply

It’s not hard to see the world has moved away from discrete software releases to a model that favors recurring periodic revenue. Gone are the days of a bi-yearly office suite offering or a tentpole version of an operating system that might gain some features down the road. Instead we now pay a yearly fee to use the program or application and in return we get lots of new things on a somewhat stilted cadence.

There are a lot of things to decry about software subscription models. I’m not a huge fan of the way that popular features are put behind subscription tiers that practically force you to buy the highest, most expensive one because of one or two things you need that can only be found there. It’s a callback to the way that cable companies put their most popular channels together in separate packages to raise the amount you’re paying per month.

I’m also not a fan of the way that the subscription model is a huge driver for profits for investors. If your favorite software program doesn’t have a subscription model just yet you’d better hope they never take a big investment. Because those investors are hungry for profit. Profit that is easier to see on a monthly basis because people are paying a fee for the software monthly instead of having big numbers pop up every couple of years.

However, this post is about the positive aspects of software subscriptions. So let’s talk about a couple of them.

Feature Creatures

If you’ve ever complained that an application or a platform doesn’t have a feature that you really want to see you know that no software is perfect. Developers either put in the features they think you want to see or they wait for you to ask for them and then race to include them in the next release. This push-pull model dominates the way that traditional software has been written.

How do those features get added to software? They do have to be written and tested and validated before they’re published. That takes time and resources. You need a team to add the code and tools to test it. You need time to make it all work properly. In other words, a price must be paid to make it happen. In a large organization, adding those features can be done if there is sufficient proof that adding them will recoup the costs or increase sales, therefore justifying their inclusion.

How about if no one wants to pay? You don’t have to look far for this one. Most of the comments on a walled garden app store are constantly talking about how “this app would be great if it had this feature” or “needs to do “X” then it’s perfect!” but no one wants to pay to have those features developed. If the developer is already charging a minimum for the app or giving it away in hope of making up the difference through ad revenue or something, how can we hope to encourage them to write the feature we want if we’re not willing to pay for it?

This is also the reason why you’re seeing more features being released for software outside of what you might consider a regular schedule. Instead of waiting for a big version release to pack in lots of new features in the hopes of getting customers to pay for the upgrade developers are taking a different approach. They’re adding features as they are finished to show people the value of having the subscription. This is similar to the model of releasing the new episodes of a series on a streaming service weekly instead of all at once. You get people subscribed for longer periods instead of doing it only long enough to get the feature they’re looking for. Note that this doesn’t cover licensing that you must hold to use the feature.

Respect for the Classics

One other side effect of subscription licensing is support for older hardware. You may think this sounds counterintuitive. However, the likelihood that an older model of hardware is going to be supported later in life is due to the fact that there is no longer a need to force upgrades to justify hardware development.

Yes, there are always going to be releases that don’t support older hardware. If it’s something that needs to use a new chip or a new hardware feature it’s going to be very difficult to export that back to a platform that doesn’t have the hardware and never will. It’s also harder to code the feature in such as way as to run hardware that has varying degrees of support and implementation capabilities. The fact that an iPhone 7 and an iPhone 13 can run the same version of iOS is a huge example of this.

More to the point, the driver to force hardware upgrades is less impactful. If you’re writing a new app or new piece of software it’s tempting to make the new features run only on the latest version of the hardware. That practically forces the users to upgrade to take advantage of the new kit and new apps. It feels disingenuous because the old hardware may still have had a few more years of life in it.

Another factor here is that the added revenue from the subscription model allows developers to essentially “pay” to support those older platforms. If you’re getting money from customers to support the software you have the resources you need to port it to older version as well as continuing to support those older platforms with things like security updates. You may not get your favorite switch supported for a decade now but you also don’t have to feel like you’re being forced to upgrade every three years either.

Tom’s Take

I go back and forth with subscription models. Some tools and platforms need them. Those are the ones that I use every day and make lots of sense to me to keep up with. Others feel like they’re using the lure of more frequent feature releases to rope users into paying for things and making it difficult to leave. When you couple that with the drive of investors to have regular recurring revenue it puts companies into a tough spot. You need resources to build things and you need regular revenue to have resources. You can’t just hope that what you’re going to release is going to be popular and worry about the rest later. I don’t know that our current glut of subscription models is the answer to all our issues. However, it’s not the most evil thing out there.

Helpdesk Skills Fit the Bill

Posted on by
Reply

RedLEDKeyboard

I saw a great tweet yesterday from Swift on Security that talked about helpdesk work and how it’s nothing to be ashamed of:

I thought it was especially important to call this out for my readers. I’ve made no secret that my first “real” job in IT was on the national helpdesk for Gateway Computers through a contractor. I was there for about six months before I got a job doing more enterprise-type support work. And while my skills today are far above what I did when I started out having customers search for red floppy disks and removing helper apps in MSCONFIG, the basics that I learned there are things I still carry with me today.

Rocket Science

Most people have a negative outlook on helpdesk work. They see it as entry-level and not worth admitting to. They also don’t quite understand just how hard it is to do this kind of work. It may be different today compared to when I did it twenty years ago through the advances in technology but there is one group of people that know just how hard it is to do remote work on a system you can’t physically touch:

NASA Engineers

If you think it’s easy to just jump in front of a keyboard and fix an issue with Outlook or Chrome, try doing it without being able to see what you’re looking at. Now try and describe what you want to do without technical jargon and without using the term “thingie”. Now do it with the patience that it takes to tell someone more than once what they need to do and how to get them to read you the error message verbatim so you can figure out what’s actually going on.

Inbound phone support is a lot like fix a space probe. You have a lag time between commands being sent and confirmation. You can’t make it too complicated because the memory of the thing you’re working with isn’t unlimited. And you have to hope what you did actually fixes the issue because you may not get a second chance at it. It might sound a touch hyperbolic but I’d argue that phone support teaches the skills needed to communicate effectively and concisely to people less technical than you. You know, executives.

Joking aside, being able to distill the issue from incomplete information and formulate a response while simultaneously describing it in non-technical terms to ease implementation is the kind of skill you don’t read out of a book. Don’t believe me? Call your parents and help them update their DNS settings. Unless your parents are in IT I doubt it’s going to be a fun call. Tech is obtuse on purpose to keep people from playing with it. The people that can cut through the jargon and keep users going are magicians.

Visualize the Result

I have a gift for analogies. They aren’t always 100% accurate but I can usually relate some kind of a technology to some real-world non-technical thing to help users understand what’s going on. That gift was developed in the six months I spent trying to explain bad capacitors, startup TSRs, and exactly what an FDISK, Format, Reload process did to the computer. It’s a super valuable skill to have outside of tech too.

Go ask your doctor to explain how the endocrine system works in non-medical terms. I bet they can because they have a way to describe most of the body functions in terms that someone who has never taken human anatomy can understand. You can’t explain technical subjects with technical terms to just anyone. And even the technical people that know what you’re saying often have a hard time visualizing what you’re talking about. That’s how we ended up with the “Internet as a series of tubes” meme.

Where does this skill come in handy? How about teaching advanced tech to your junior staff? Or educating the executives in a board meeting? Or even just trying to tell the users in your organization why it’s not the network this time? If you can only talk in technical jargon without giving analogies or helping them visualize what you’re saying you will only end up with confused angry users that think you’re being patronizing.

The helpdesk forces you to get better at being descriptive with shorter words and more visual descriptions. I’d argue that my helpdesk time made my writing better because it reminds me not to be loquacious when I should be economical in my word choices and sentence lengths. If you can’t help your user figure out what you’re doing they’ll never learn how to be better at telling people what they need support for.

Documenting All the Things

The last skill that the helpdesk instilled in me is documentation. I’m a bad note taker. I get distracted (thanks ADHD) and often forget to write down important things. Helpdesk work is a place where you absolutely need to write everything down. Ask questions and record info. Dig into error messages and find out when they started happening. Listen to what people are telling you and don’t jump to conclusions until you’ve written it all down.

I once had a call where the user told me that they couldn’t get into their computer. I thought this was an easy fix. I spent ten minutes walking through the Windows XP login process. Nothing worked. I was getting frustrated and was about to reboot to safe mode and start erasing password hashes. By chance the user mentioned that they saw the pop up for the buzzer and it still wasn’t working. After I questioned them on this I realized that they could log in to WinXP just fine. The “buzzer popup” was AOL’s login screen. They had an issue with the modem, not the operating system. If I’d asked more questions about when the problem started and what they saw instead of just jumping to the wrong conclusion I could have saved a ton of pain and wasted effort.

Likewise, when you’re doing IT work you have to write it all down. Troubleshooting makes a lot of sense here but so does implementation or other kinds of design work. If you’re doing a wireless survey on site and forget to write down what the walls are made out of you may find yourself with an ineffective design or, worse having to spend extra time and money fixing it on the fly because those reinforced concrete walls are way better at signal attenuation than you recalled from a spotty memory.

Get in the habit of taking good notes from the start and summarizing them when needed to weed out the working things from the non-working things. Honestly, having a record of all the steps you took to arrive at your conclusion helps in the future if the issue happens again or you find yourself at a brick wall and need to retrace your steps to figure out where you took the wrong turn. If you record it all you won’t need to spend too much effort scratching your head to figure out what you were thinking.

Tom’s Take

Every job teaches you skills you can carry forward. Even the worst job in the world teaches you something you can take with you. Some jobs have a negative stigma and really shouldn’t. Like Swift said above you should accentuate the positive aspects of your journey. Don’t look at the helpdesk as a slog through the trenches. See yourself as a NASA rocket scientist that can talk to normal people and document like a champion. That’s a career anyone would be proud to have.

Fast Friday Thoughts on Leadership

Posted on by
1

I’m once more taking part in the BSA Wood Badge leadership course for my local council. I enjoy the opportunity to hone my skills when it comes to leading others and teaching them how to train their own leaders. A lot of my content around coaching, mentoring, and even imposter syndrome has come from the lessons I’ve learned during Wood Badge. It sounds crazy but I enjoy taking vacation time to staff something that looks like work because it feels amazing!

A few random thoughts from the week:

  • You need a sense of urgency in everything you do. You may not know exactly what’s coming or how to adjust for what needs to be done but you need to be moving with purpose to get it done. Not only does that help you with your vision to make things happen but it encourages others to do the same.
  • Team building happens when you’re not focused entirely on the goal. It doesn’t take much for your group to come together but it can only happen when they aren’t charging toward the finish line. Remember that taking a few moments here and there to reinforce the group dynamic can do a lot to build a cohesive unit. Stopping to smell the roses isn’t a bad thing when your team members chat about them too.
  • Make sure you appreciate your team when they succeed. The grindstone doesn’t feel quite so rough when you know that your effort is appreciated. It’s difficult some times to remember to recognize those that put in the work but if you do you’ll create happier people and more functional ones too.

Tom’s Take

I’m sure I’ll have more thoughts about leadership and team building as soon as I’m done with the practical application of it this weekend!

A Modest Proposal for Cisco Interface Naming

Posted on by
4

If you’re going to be configuring an interface in a switch, which one are you going to use? The interface has a name and a number based on where it is on the device. The numbering part is fairly easy to figure out. The module number comes first, followed by the slot, and finally the port. In the world of Cisco, which is the one I’m the most familiar with, that means a fixed configuration switch usually has interfaces labeled 0/24, with no module and the slot almost always being zero. With a modular switch the interface would be labeled 2/0/28 to indicate the 28th port on the second line card.

The issue arises when you factor in the first part of the interface naming convention. The nomenclature used in the Cisco world since the beginning of time has been the interface speed. If your interface is a 100Mbit Ethernet interface then the interface name is FastEthernet0/48. If you’re using a 1Gbit interface it’s GigabitEthernet0/48. If it’s a 10Gbit interface it becomes TenGigabitEthernet0/48. It’s a progression of interface speeds. Even if the port is capable of using 10/100/1000 the port is referred to at the highest speed. The 10Gbit ports don’t negotiate to a lower speed so they are always TenGigabit.

Now, with the new Catalyst 9000X Series switches you have the option of a port that can do 10/100/1000/2.5G/5G/10G. It’s a multi-gigabit capable port as outlined in this video from a recent Field Day event:

I don’t even want to think about the nightmare of trying to remember the name of the interface in a situation like that. It’s hard enough still to remember what the switches are capable of, whether it’s a gigabit port or a ten gigabit port. It’s hard enough for me but it’s impossible for anyone looking to make a script or automation that needs to be repeatable over a wide variety of switches.

Naming Convention Unity

The obvious solution here is actually present in another Cisco platform already. The Nexus series of switches running Cisco NX-OS already knows how to handle interface naming issues. It’s a problem that occurs when you could be running Gigabit Ethernet or Fibre Channel over Ethernet interfaces.

How does NX-OS fix this? The interface type is always Ethernet. It doesn’t matter what speed the interface is running at because it’s always Ethernet under the hood. So the every interface is labeled Ethernet(module/slot/port). It’s wonderful when you’re working on the device because you never have to make a typo based on the interface speed. It’s also consistent across every platform and module that could be installed.

The solution to the problem actually came from my friend Bruno Wollmann (@WollmanBruno) during the event. It’s well past time to adopt the NX-OS convention in every other version of Cisco IOS. You can alias the old interface names to the new nomenclature to ease the transition if you’d like. Otherwise it’s time to start making people use the basic convention of calling everything Ethernet.

This would ease new users into IOS by making all interfaces the same and focusing on the module/slot/port complexity as the primary place to learn about naming. It also helps with the development of automation and orchestration scripts. Rather than needing to detect device types or write different scripts for NX-OS or IOS-XE you can just use the same logic for both.

Tom’s Take

It’s time we move into the future of interface names. Bruno made an excellent point when he said that it’s time to adopt a standard across the whole line of switches. Cisco may not be ready to do a single OS but having unity between the features will go a long way to making it easier to work on the various platforms. You can do a lot with aliasing in the system to ease the transition but maybe by the time we get to IOS 20 we can use something that’s more manageable for the future.

No Is A Complete Sentence

Posted on by
5

Has someone asked you to do something recently that you know you don’t have time to do but felt like you needed to do anyway? Or has someone tried to get you to help with something and impressed upon you just how important it is? You probably told them “yes” out of guilt or obligation or some other kind of negative emotion. Sure, you could have declined but you thought about how bad you would feel if someone did the same to you.

Let me tell you clearly. “No” is a complete sentence. It requires no explanation or defense. It is the only thing you need to say when you know you won’t be able to do something no matter how much the other party tries to get you to agree.

Everything Sucking Equally

If you know anything about QoS, you know that once a given circuit reaches the limitation for bandwidth you can no longer send additional information. What’s counterintuitive about this is most people would assume that if you try to squeeze one more stream or packet into the mix that only that last packet would be affected and everything else would work perfectly fine, right? Only one of the many would suck.

Sadly, all of the packets or flows on the circuit are affected in this situation. Adding just one more thing to the mix means all things are affected because the system has to process the problems all at once. You suffer everywhere because the interface couldn’t say “no” to that last packet.

In fact, the whole mantra behind QoS is prioritizing information before it hits a given interface because the interface is just going to send whatever it is given. Those packets will queue up or eventually be dropped if they can’t be processed. Likewise, if you are trying to do too many things at once you will eventually miss something important and make a mistake because you couldn’t handle everything at once. You need to be able to prioritize all the things that are going on and know when you hit a limit you can’t get past.

Where the challenge comes into play is when others try to add more to the interface without knowing you’re at maximum capacity. It’s easy for someone to ask you to do just one more little thing. In their minds it’s not a huge ask, right? In your mind you’ve already begun trying to juggle all the things that need to be changed to make this one little thing work with all the other things you’ve spent so much time prioritizing. Unfortunately, it’s too easy to just say “yes” to avoid uncomfortable conversations and try to fix the problems as they come up rather than turning away something you know would put you over your limit.

Taking No For an Answer

It’s easy to tell people to say “no”, right? I mean it is possibly the shortest sentence in the English language. However, it’s not the person saying it that usually has an issue. The people doing the asking are the ones that almost invariably are the ones trying to convince, cajole, or outright impress upon the person how critical it is that whatever this is get done in addition to all the other things that are going on.

For those that are asking the question and expecting a positive answer, let me ask you a question of my own: Why does this person need to do the task? There usually are very good reasons for it. Perhaps it’s something this person specializes in. Maybe it’s something that only they have knowledge about. It could perhaps even be their job responsibility to get it done. But the next question you need to ask is even more important.

If they must do this task, what can I do to help them?

The calling card of bad management types is to show up, assign a bunch of tasks that only other people should do, and then fly away to measure their productivity and comment on how nothing seems to be getting done. Instead, management should be asking how they can help. Is this something that someone else could do? Do you have the bandwidth to do the task? Is there another task on your plate that I can do for you that would help you to take care of this?

People are often very fully tasked with all the things they have to take care of. Thinking that you can sneak one more thing onto their schedule without impacting all of it smacks of hubris. What if someone started adding tasks to your plate instead? I doubt you’d like that. Even if you said no, what if it was the CEO or the Chairman of the Board telling you that you had to do it? You’d probably be just as upset, especially if it was a task you couldn’t delegate.

For the people out there like me, try this next time someone asks you to do something and you’re fully tasked or unable to clear enough to accomplish it. Say “no”. Don’t explain. Don’t justify. Don’t defend. Just. Say. No. Watch the other person’s reaction. Are they getting frustrated because you’re not saying anything? Are they coming back in a helpful manner trying to understand why you declined? Or are they more upset that they now have to go find someone else to do it instead? If they come to you with respect and understanding you can tell them why you declined and are unable to help. They may offer suggestions or try to find a way to make it work. The ones that get upset are usually the ones that wouldn’t have taken “no” for an answer no matter what. They are the ones you don’t need to waste your breath on.

Tom’s Take

I will admit this post is as much for me as it is for others. I am the world’s worst for taking on additional things when I know I don’t have the bandwidth to do it all. I don’t like to disappoint. I really want to keep people happy. And instead I end up failing because I overestimate my ability to get work done and underestimate how much others could do if they knew I needed help. Next time you need to tell someone you can’t do something, don’t spend a ton of effort figuring out how to counter their eventual arguments. Just say “no” and move on. If it’s important or something that only you can do they will find a way to make it work. Otherwise, you will have saved so much more effort because you used a complete sentence instead of a needless paragraph.

Trust Will Do You In

Posted on by
1

If you’re a fan of the Gestalt IT Rundown that I do every week on the Gestalt IT YouTube channel, you have probably heard about the recent hacks of NVIDIA and Samsung. The original investigation into those hacks talked about using MDM platforms and other vectors to gain access to the information that was obtained by the hacking groups. An interesting tweet popped up on my feed yesterday that helped me reframe the attacks:

It would appear that the group behind these attacks are going after their targets the old fashioned way. With people. For illustration, see XKCD from 2009:

The Weakest Links

People are always the weakest link in any security situation. They choose to make something insecure through bad policy or by trying to evade the policy. Perhaps they are trying to do harm to the organization or even try to shine a light on corrupt practices. Whatever the reason, people are the weak link. Because you can change hardware or software to eliminate failures and bugs. You can’t reprogram people.

We’ve struggled for years to keep people out of our systems. Perimeter security and bastion hosts were designed to make sure that the bad actors stayed off our stage. Alas, as we’ve gotten more creative about stopping them we’ve started to realize that more and more of the attacks aren’t coming from outside but instead from inside.

There are whole categories of solutions dedicated to stopping internal attackers now. Data Loss Prevention (DLP) can catch data being exfiltrated by sophisticated attackers but it is more often used to prevent people from leaking sensitive data either accidentally or purposefully. There are solutions to monitor access to systems and replay logs to find out how internal systems folks were able to get privileges they should have.

To me, this is the crux of the issue. As much as we want to create policies that prevent people from exceeding their authority we seem to have a hard time putting them into practice. For every well-meaning solution or rule that is designed to prevent someone from gaining access to something or keep them secure you will have someone making a shortcut around it. I’ve done it myself so I know it’s pretty common. For every rule that’s supposed to keep me safe I have an example of a way I went around it because it got in my way.

Who Do YOU Trust?

This is one of the reasons why a Zero Trust Network Architecture (ZTNA) appeals to me. At its core it makes a very basic assumption that I learned from the X-Files: Trust No One. If you can’t verify who you are you can’t gain access. And you only gain access to the bare minimum you need and have no capability for moving laterally.

We have systems that operate like this today. Call your doctor and ask them a question. They will first need to verify that you are who you say you are with information like your birthdate or some other key piece of Personally Identifiable Information (PII). Why? Because if they make a recommendation for a medication and you’re not the person they think they’re talking to you can create a big problem.

Computer systems should work the same way, shouldn’t they? We should need to verify who we are before we can access important data or change a configuration setting. Yet we constantly see blank passwords or people logging in to a server as a super user to “just make it easier”. And when someone gains access to that system through clever use of a wrench, as above, you should be able to see the potential for disaster.

ZTNA just says you can’t do that. Period. If the policy says no super user logins from remote terminals they mean it. If the policy says no sensitive data access from public networks then that is the law. And no amount of work to short circuit the system is going to work.

This is where I think the value of ZTNA is really going to help modern enterprises. It’s not the nefarious actor that is looking to sell their customer lists that creates security issues. It does happen but not nearly as often as an executive that wants a special exception for the proxy server because one of their things doesn’t work properly. Or maybe it’s a developer that created a connection from a development server into production because it was easier to copy data back and forth that way. Whatever the reasons the inadvertent security issues cause chaos because they are configured and forgotten. At least until someone hacks you and you end up on the news.

ZTNA forces you to look at your organization and justify why things are the way they are. Think of it like a posture audit with immediate rule creation. If development servers should never talk to production units then that is the rule. If you want that to happen for some strange reason you have to explicitly configure it. And your name is attached to that configuration so you know who did it. Hopefully something like this either requires sign off from multiple teams or triggers a notification for the SOC that then comes in to figure out why policy was violated. At best it’s an accident or a training situation. At worst you may have just caught a bad actor before they step into the limelight.

Tom’s Take

Security isn’t perfect. We can always improve. Every time we build a good lock someone can build a better way to pick it. The real end goal is to make things sufficiently secure that we don’t have to worry about them being compromised with no effort while at the same time keeping it easy enough for people to get their jobs done. ZTNA is an important step because it creates rules and puts teeth behind them to prevent easy compromise of the rules by people that are trying to err on the side of easy. If you don’t already have plans to include ZTNA in your enterprise now you really should start looking at them. I’d tell you to trust me, but not trusting me is the point.

AI is a Promotion

Posted on by
1

When I worked at IBM as an intern, part of my job was writing a deployment script to help make our lives easier when installing new ThinkPads. In order to change an MTU setting on the token ring PCMCIA cards (long story), I had to write a script that iterated through all the possible combinations of adapters in the registry to find the one I was looking for and change the value.

Now, I was 22 at the time and green behind the ears, especially when it came to programming. I finally figured out that the most efficient way to do this in the language that I was using was a very deep nested if statement. It wasn’t my best work but it operated properly. I mentioned this to my mentors on my team with a remark of how hard it was to understand the logic at first. My comment was “You know, if it’s hard to read for anyone else then I never have to worry about gettin fired.”

To which the response was, “Yes, but you can never be promoted either.”

That sage wisdom brings me to the modern world and how AI can fix that issue.

Eternal Support

One of the first things you learn in IT is not to volunteer. If you know how to implement a technology or troubleshoot a specific issue then you become the go-to person for that problem. If you add a skill to the team like debugging Rust or configuring voice mail dial peers then you are the new owner for that particular skill.

This becomes annoying when you finally move past that skillset and you’re on to new and better things. If you are the person that knows how to extend database schemas or change auto attendant functions you are always going to be called upon to do those things. Rather than moving on to new challenges you will be called back to work on the things that no one else bothered to learn or couldn’t pawn off on someone else.

When you think about it, that’s part of the reason why so many entry-level positions start with some basics like VLAN programming or LUN configuration. Sure, you’re learning the basics and understanding how systems operate. But you’re also repeating the things that the junior person before you did. Why? Because they now have someone to do that work for them. And when the new person gets hired as a junior-level technician, you’re going to pass along those tasks to someone so they can learn and so you can move on. It’s the Circle of IT Life.

The reinforcement of knowledge is only so important to the organization. Yes, you need someone to keep the lights on and do the tasks regarded as menial by the senior teams. However, miring people in those tasks makes them feel undervalued after a while because no one wants to take them on. And, as above, if it’s something that no one else can be bothered to learn or take over you’re going to be stuck doing it until you leave or until the system in question is eventually replaced. And even then you’ll probably be asked to learn how to do the new thing because “this is your system”.

Learning, Not Thinking

The reason why I’m excited for applied AI in infrastructure and operations is because it will learn these menial tasks and do them as instructed every time without complaint or mistake. Yes, that means you need to train the algorithm correctly. Yes, that means you need to be on the lookout for changes and update accordingly when hardware changes or when procedure changes. Given those constraints AI will work for you, not against you.

When you move on to a new role you expect to face new challenges. If you move from a junior role to a senior role you want to work on harder things or solve new problems. You don’t want to get a new title with the same old work. If you change teams you want to experience new systems and implement new technology. You don’t want to be the cloud person that still deploys campus switches in wiring closets.

In a way, having AI take over the basic tasks of your job functions doesn’t remove your job. It promotes you to a new role. It could be that the first promotion is running the system that you implemented to do the work. A “train the trainer” kind of role, if you will. As soon as you can ensure that it is working correctly and that the rest of the team is aware of how to keep the engine running you can decide where you want to go next. You essentially have the freedom to learn and do and not have to be concerned about going back to doing the old things.

Is AI the Enemy?

AI doesn’t take away jobs. It takes away tasks. It streamlines processes and provides consistent, repeatable outcomes. If your job is a collection of tasks that need to be done then it is worth asking why it’s so easy for it to be replaced by an AI system. I’ve said on a number of occasions that there will always been room for jobs that AI can’t do, such as those that involve physical labor, as well as advanced thinking and creative problem solving.

Where AI does succeed is in the repeatable task department. If a role is comprised of repeatable tasks then the question becomes “why does this role exist?” If an algorithm can determine the best way to implement VLANs or offload clients to other devices then the job that was doing those roles needs to be examined. Can that person be encourage to do something else for the company? Can that role be reduced and given to someone with less experience? If those questions make you or someone on your team bristle, is that the fault of the AI? Or the fact of the job being less than ideal for the business?

Tom’s Take

AI can take away some of the unimportant and forgettable parts of your job and give you the freedom to do things that are more important to the business, possibly in a different role. Isn’t that the textbook definition of a promotion? Change makes people uncomfortable. It makes sense that no one wants to have their job taken away. But if you ask anyone in IT right now if they could get rid of 10-20% of the tasks in their job that they don’t like to do they’d agree in a heartbeat. AI isn’t going to remove your job. It’s going to give you the freedom to do something different. It’s also going to ensure you don’t have to keep getting called back to do that thing over and over again because no one else wants to learn it. I think it’s time we let new technologies like AI give everyone on the team the promotion they deserve.

The Value of Old Ideas

Posted on by
1

I had a fun exchange on Twitter this week that bears some additional thinking. Emirage (@Emirage6) tweeted a fun meme about learning BGP:

I retweeted it and a few people jumped in the fun, including a couple that said it was better to configure BGP for reasons. This led to a blog post about routing protocols with even more great memes and a good dose of reality for anyone that isn’t a multi-CCIE.

Explain It Like I’m Five

I want you to call your mom and explain BGP to her. Go on and do that now because I’m curious to see how you’d open that conversation. Unless your mom is in networking already I’m willing to bet you’re going to have to start really, really basic. In fact, given the number of news organizations that don’t even know what the letters in the acronym stand for I’d guess you are going to have a hard time talking about the path selection process or leak maps or how sessions are established.

Now, try that same conversation with RIP. I bet it goes a lot smoother. Why? Because RIP is a simple protocol. We don’t worry about prefixes or AS Path prepending or other things when we talk about RIP. Because RIPv1 is the most basic routing protocol you can get. It learns about routes and sends the information on. It’s so simple that you can usually get all of the info about it out of the way in a couple of hours of instruction in a networking class.

So why do we still talk about RIP? It’s so old. Even the second version of the protocol is better. There’s no intelligence. No link state. You can’t even pick a successor route! We should never talk about RIPv1 again in any form, right? So what would you suggest we start with? OSPF? BGP?

The value of RIP is not in the protocol. Instead, the value is in the simplicity of the discussion. If you’ve never heard of a routing protocol before you don’t want to hear about all the complexity of something that runs half the Internet. At least you don’t want to hear about it at first. What you need to hear is how a routing protocol works. That’s why RIP is a great introduction. It does the very minimal basic functions that a routing protocol should do. It learns routes and tells other routers about them.

RIP is also a great way to explain why other routing protocols were developed. We use OPSF and EIGRP and other IGRPs now because RIP doesn’t perform well outside of small networks. There are limitations around IP subnets and network diameter and “routing by rumor” that only make sense when you know how routing protocols are supposed to operate and how they can fall down. If you start with OSPF and learn about link-state first then you don’t understand how a router could ever have knowledge of a route that it doesn’t know about directly or learn about from a trusted source. In essence, RIP is a great first lesson because it is both bad and good.

The Expert’s Conundrum

The other issue at hand is that experts tend to feel like complicated subjects they understand are easy to explain. If that’s the case, then the Explain Like I’m Five Reddit shouldn’t exist. It turns out that trying to explain a complex topic to people without expert knowledge is super hard because they lack the frame or reference you need to help them understand. If you are a network engineer that doesn’t believe me then go ask a friend in the medical profession to explain how the endocrine system works. Don’t make they do a simple explanation. Make them tell you like they’d teach a medical student.

We lose sight of the fact that complex topics can’t be mastered quickly and certain amounts of introductory knowledge need to be introduced to bring people along on the journey. We teach about older models and retired protocols because the knowledge they contain can help us understand why we moved away from them. It also helps us to have a baseline level of knowledge about things that permeate the way we do our jobs.

If we removed things that we never use from the teaching curriculum we’d never have the OSI model since it is never implemented in the pure form in any protocol. We’d teach the TCP/IP model and just tell people that the other things don’t really matter. In fact, they do matter because the pure OSI model takes other things into account that aren’t just focused on the network protocol stack. It may seem silly to us as experts to say that we teach one thing but reality looks much different but that’s how learning works.

We still teach older methods of network topologies like bus and ring even though we don’t use them much any more. We do this because entry-level people need to know why we arrived at the method we use now. Even with the move toward new setups like 3-tier network design and leaf/spine architecture you need to know where Ethernet started to understand why we are where we are today.

Tom’s Take

It’s always important to get a reminder that people are just starting out in the learning process. While it’s easy to be an expert and sit back and claim that iBGP is the best way to approach a routing protocol problem you also have to remember that learners sometimes just need a quick-and-dirty lab setup to test another function. If they’re going to spend hours configuring BGP neighbor relationships instead of just enabling RIP on all router interfaces and moving on to the next part then they’re not learning the right things. Knowledge is important, even if it’s outdated. We still teach RIP and Frame Relay and Token Ring in networking because people need to understand how they operate and why we’ve moved on. They may not ever configure them in practice but they may also never configure BGP either. The value of information doesn’t decrease because it’s old.

Technical Debt or Underperforming Investment?

Posted on by
2

In this week’s issue of the Packet Pushers Human Infrastructure newsletter, there was an excellent blog post from Kam Lasater about how talking about technical debt makes us sound silly. I recommend you read the whole thing because he brings up some very valid points about how the way the other departments of the organization perceive our issues can vary. It also breaks down debt in a very simple format that takes it away from a negative connotation and shows how debt can be a leverage instrument.

To that end, I want to make a modest proposal to help the organization understand the challenges that IT faces with older systems and integration challenges. Except we need some new branding. So, I propose we start referring to technical debt as “underperforming technical investments”.

I’d Buy That For A Dollar

Technical debt is just a clever way to refer to the series of layered challenges we face from decisions that were made to accomplish tasks. It’s a burden we carry negatively throughout the execution of our job because it adds extra time to the process. We express it as debt because it’s a price that must be paid every time we need to log in to the old accounting system to make changes or we have to wake up on the third Sunday of the month to reboot the access points because of a software bug that we can’t fix.

As Lasater points out, the financial departments look at debt differently. It’s not always negative. Commercial paper allows for purchasing power beyond cash-on-hand. Mortgages and loans allow us to buy houses and cars that enable us to do much more with our lives. Reasonable debt can be good so long as what we get out of it is a net positive. Debt doesn’t become an issue until there’s too much of it for the return we get, which eventually leads to bankruptcy.

Let’s point out some of the challenges we face with technical things that we refer to as debt. We usually use it in reference to outdated systems or slower equipment that doesn’t allow us to do our jobs as effectively as possible. Unlike a house which still provides the value of a dwelling at any age given reasonable maintenance or a car that still operates with the same maintenance, technology changes more rapidly. The original iPhone is unable to operate in a modern environment because of outdated software or inability to connect to the mobile network.

In consumer tech, the idea of technical debt doesn’t exist. Because when things are “slow” or “broken” they get discarded and people spend money to get something new. Someone that is willing to live with a shattered phone screen because they don’t want to pay to fix it might be willing to wait another two months to get a new device because it supports the latest OS updates or has a better camera. The chase for new features is usually enough of a driver to get people to upgrade.

In the enterprise, the infrastructure equipment is expensive enough that we need to recognize the value of what we installed. We can’t just swap out all the switches in three years if we spent hundreds of thousands of dollars on them. Likewise, enterprise tech vendors are more likely to provide patches and updates for this gear knowing that customers are going to hold on to it for longer periods of time. And even then, just because something is EoL doesn’t mean it’s End of Use.

However, the disconnect between these two things comes when we talk about how equipment that is functional is also causing issues with our job. Maybe it’s an older, slower switch that drops more packets than the rest of the newer gear. It’s a problem but not big enough to justify buying a new one. It could be an older wireless controller that doesn’t have the capability to run the newest code because of an older CPU. We have gear that works but it doesn’t work as well as it could. But we also don’t have enough justification to get new stuff because the old things haven’t broken completely yet.

Putting Your Money Where Their Mouth Is

As Lasater says in the above article, talking about how it’s hard to do your job because everything is old doesn’t resonate with management. No one really wants to hear a department head whining about how their job is tough because they don’t have the latest toys. Admit it, you’d be upset if your CEO told you they couldn’t answer their email because they don’t have the newest MacBook Pro. Capable means functional in every enterprise.

However, I think the challenge we face and the solution to the problem is how we frame the discussion of the challenge itself. We talk about it as a debt that we feel saddled with. It’s something we did because we had to or something someone else chose to do that we have to live with. It feels like we’re paying a tax every day over something that we don’t like. But choices need to be made and we have to work with what we have. Unless we can provide for a way to get people to understand the tradeoffs.

Hence, my proposal above. First, we stop referring to IT infrastructure gear as “technical debt”. It’s not something we bought and now have to deal with. Our accounting departments are amortizing the acquisition costs of the equipment over a period of time and writing it off of taxes against recognized revenue. There is a real dollar amount attached to every month of the lifetime of a switch or a server. That’s how finance sees your equipment. It’s not debt.

It’s a technical investment.

That’s your key to helping them understand the challenges you face. You’re not complaining about doing your job more slowly. Instead, you’re pointing out that you are investing a resource (your time) into a process that could have reduced resource requirements (again, your time and job execution time) if they would just invest in a different solution.

You’re not telling them they bought a lemon. Instead, you’re just telling them that the thing they invested in years ago isn’t performing as well as it could based on new data. If you need to break it down even further for your executive team you can just equate it to a home loan refinancing. If interest rates have dropped since you bought your home, doesn’t it make a ton of sense to refinance and take advantage of the savings? Likewise, if the speed or power of a system has increased in the past four years doesn’t it make sense to invest in something new?

When you change the discussion to focus on investment of resources instead of dealing with debt you change the way that the people will look at the return on that investment. If you just talk about debt the executives are going to easily be able to dismiss you because there is no upside to the conversation. They have to pay money to replace things so why spend it? If you talk about how much they could save on labor or how much more efficiently things could run if they made some changes or bought something new then that is harder to ignore. If their existing investment is underperforming some baseline on the stock market they’re going to want to move it right away. If the company’s stock is underperforming you can better believe some of the shareholders are going to want to move their investment too.

Here Comes the Money

One word of warning here. You can’t go in to this conversation with vague assurances that spending more money will make money. You have to come with numbers that support your case. Some of them are easy. You know what the new system will cost. You know how much time it will take to implement it. What you need to bring is the cost of what you are paying now to do things inefficiently.

If your team of 5 has to spend two extra hours a week doing something because the system is old or doesn’t work well with others then you have ten hours of lost time. You need to figure out what the value of your time is. Big note here: Don’t just divide your yearly salary by 2000 and take the averages. That’s what you’re getting paid. That’s the debt that the company incurs to keep you on staff. Instead, find out what your rate is. If you are billed at a higher rate than your earnings, which is almost a guarantee, use that number instead. If you’re an engineer for a reseller your hourly billing rate is easy to find. If you work in corporate IT the rate is harder to figure out but if you need a rule of thumb just figure out what your hourly pay rate is (yearly salary divided by 2000 hours per year) and double it. That’s the kind of return that any company would love to have on an investment. If it’s costing the company that much to do something inefficiently then you can make your case to get it fixed or replaced.

Lastly, make sure to record the changes and the results if you manage to make it happen. It’s easy to chart when you take more time to do something to prove a point. But it’s easy to forget to do it when you have what you want. Eventually, someone that manages investments needs to know if it paid off. If there’s a simple number like a stock price it’s cut-and-dried. If there’s more to the calculation you need to do the work to prove why it’s better now. And you want to keep a running total so you can provide it on demand when asked. Just tell them you have more time to track those kinds of things now that they made a wiser investment.

Tom’s Take

No one likes debt. Even the best debt is still an obligation to pay back to someone. However, investments are something positive. Yes, they still require resources to be exchanged to receive a good or service. However, changing the terminology changes the perception of the result. Debts are incurred and paid back begrudgingly. Investments grow and provide additional resources throughout their lives. And, importantly, when an investment isn’t paying off the way to fix it is to reinvest. It’s a clever trick but one that should work much better than just whining about drowning in debt.