Quality To Be Named Later

programming

First off, go watch this excellent video from Ken Duda of Arista at Networking Field Day 28. It’s the second time he’s knocked it out of the park when it comes to talking about code quality:

One of the things that Ken brings up in this video that I thought would be good to cover in a bit more depth is the idea of what happens to the culture of your organization, specifically code quality, when you acquire a new company. Every team goes through stages of development from formation through disagreement and finally to success and performance. One of the factors that can cause a high-performing team to regress back to a state of challenges is adding new team members to the group.

Let’s apply this lesson to your existing code infrastructure. Let’s say you’ve spent a lot of time building the best organization that has figured out and your dev teams are running like a well-oiled machine. You’re pushing out updates left and right and your users are happy. Then, you buy a company to get a new feature or add some new blood to the team. What happens when that new team comes on-board? Are they going to integrate into what you’ve been doing all this time? Do they have their own method for doing development? Are they even using the same tools that you have used? How much upheaval are you in for?

Buying Your Way To Consistency

Over a decade ago, United Airlines bought Continental Airlines. Two years later, the companies finally merged their ticketing systems. Well, merged might be a bit of a stretch. United effectively moved all their reservations to the Continental system and named the whole thing United. There are always challenges with these kinds of integrations but one might think that part of the reason for the acquisition was to move to a more modern reservation system.

United’s system was called Apollo and built in the 1970s. How could they move to a more modern system? Was the reason for the huge purchase of another airline directly related to their desire to adopt a newer, more flexible reservation system? There have certainly been suggestions of that for a number of years. But, more importantly, they also saw the challenges faced by one of their Star Alliance partner US Airways in 2007 when they tried a different approach to merging booking systems. The two radically different code bases clashed and created issues. And that’s for something as simple as an airline reservation system!

In the modern day we have much more control over the way that code is developed. We know the process behind what we do and what we write. When we build something we control it the entire way. However, that is true of everyone that writes code. And even with a large number of “best practices” out there no two developers are going to approach the problem the same way unless they work for the same company. So when you bring someone on board to your team through acquisition you’re also bringing in their processes, procedures, and habits. You have to own what they do because now their development quirks are part of your culture.

Bracing For the Impact

There’s a lot of due diligence that happens when companies are purchased. There’s an army of accountants that pore over the books and find all the potential issues. I’d argue that any successful merger in today’s world also needs to include a complete and thorough code review as well. You need to know how their culture is going to integrate into what you’re doing. Never mind the intellectual property issues. How do they name variables? Do they call the same memory allocation routines? Do they use floats instead of integers because that’s how they were taught? What development tools do they use and can those tools adapt to your workflow?

It may sound like I’m being a bit pedantic when I talk about variable naming being a potential issue but when it comes to code that is not the case. You’re going to have to train someone in procedure and you need to know who that is before they start committing code to your codebase. Those little differences are going to create bugs. Those bugs are going to creep into what you’re working on and create even more problems. Pretty soon something as simple as trying to translate from one IDE to another is going to create a code quality problem. That means your team is going to spend hours solving an issue that could have been addressed up front by figuring out how things were done in two different places. If you think that’s crazy, remember NASA lost a satellite over a unit conversion problem.


Tom’s Take

You may never find yourself in the shoes of someone like Ken Duda. He’s committed to quality software and also in charge of trying to integrate acquisitions with his existing culture. However, you can contribute to a better software culture by paying attention to how things are done. If you do things a certain way you need to document everything. You need to ensure that if someone new comes into the team that they can understand your processes quickly. That way they don’t spend needless hours troubleshooting problems that were lost in translation at the start of the process. Do the hard work up front so you aren’t calling people names later.

Friday Thoughts on the Full Stack

It’s been a great week at Networking Field Day 28 this week with some great presentations and even better discussions outside of the room. We recorded a couple of great podcasts around some fun topics, including the Full Stack Engineer.

Some random thoughts about that here before we publish the episode of the On-Premise IT Roundtable in the coming weeks:

  • Why do you need a full stack person in IT? Isn’t the point to have people that are specialized?
  • Why does no one tell the developers they need to get IT skills? Why is it more important for the infrastructure team to learn how to code?
  • We see full stack doctors, which are general practitioners. Why are there no full stack lawyers or full stack accountants?
  • If the point of having a full stack understanding is about growing non-tech skills why not just say that instead?
  • There’s value in having someone that knows a little bit about everything but not too much. But that value is in having them in a supervisor role instead of an operations or engineering role. Do you want the full stack doctor doing brain surgery? or do you want him to refer you to a brain surgeon?

Tom’s Take

Note that I don’t have all the answers and there are people that are a lot smarter than me out there that have talked a lot about the full stack engineering issue. My biggest fear is that this is going to become another buzzword just like 10x engineer that carries a lot of stigma about being less than useful while being a know-it-all. When the episode of the podcast comes out perhaps it will generate some good discussion on how we should be handling it.

Ease of Use or Ease of Repair

HammerAndSaw

Have you tried to repair a mobile device recently? Like an iPad or an MacBook? The odds are good you’ve never even tried to take one apart, let alone put in replacement parts. Devices like these are notorious to try and repair because they aren’t designed to be fixed by a normal person.

I’ve recently wondered why it’s so hard to repair things like this. I can recall opening up my old Tandy Sensation computer to add a Sound Blaster card and some more RAM back in the 90s but there’s no way I could do that today, even if the devices were designed to allow that to happen. In my thinking, I realized why that might be.

Build to Rebuild

When you look at the way that car engine bays were designed in the 80s and 90s you might be surprised to see lots of wasted space. There’s room to practically crawl in beside the engine and take a nap. Why is that? Why waste all that space? Well, if you’re a mechanic that wants to get up close and personal with some part of the engine you want all the space you can find. You’d rather waste a little space and not have busted knuckles.

A modern engine isn’t designed to be repaired by amateurs. The engine may be sideways or have replaceable parts put in places that aren’t easily accessed without specialized equipment. The idea is that professionals would have everything they need to remove the engine if necessary. Moreover, by making it harder for amateurs to try and fix the issues you’re are preventing them from doing something potentially problematic that you’ll have to fix later.

The same kind of thing applies to computing equipment. Whereas before you could open the case of a computer and try to fix the issues inside with a reasonable level of IT or electrical engineering knowledge you now find yourself faced with a compact device that needs specialized tools to open and even more specialized knowledge to fix. The average electrical engineer is going to have a hard time understanding the modern system-on-a-chip (SoC) designs when you disassemble your phone.

Usability Over Uniqueness

The question of usability is always going to come into play when you talk about devices. The original IBM PC was a usable design for the time it was built compared to mainframes. The iPhone is a usable design for the time when it was built as well. But compared to each other, which is the more usable design today?

Increasing usability often comes at the cost of flexibility. I was talking to someone at a fabrication lab about printer drivers for a laser etching machine. He said that installing the drivers for the device on Windows is easy but almost impossible on a Mac. I chuckled because the difficulty of installing a non-standard printer on a Mac is part of the reason why it’s easy to use for other kinds of printers. The printing subsystem traded usability with things like Bonjour printer installation over flexibility, like installing strange printers on a laptop.

Those trade offs extend into the hardware. Anyone that worked on PCs prior to Windows 95 knows all about fun things like IRQ conflicts, DIP switch settings for COM port addresses, and even more esoteric problems. We managed to get rid of those in the modern computing era but along with that went the ability to customize the device. You no longer have to do the dance of finding an available IRQ but you also can’t install a device that would conflict with another through hardware trickery.

By creating devices for the average user focused on usability you have to sacrifice the flexibility that makes them easier to fix. If you knew you were constantly going to need to be tweaking the jets on a carburetor you’d make it easy to access, right? But if a modern fuel injection system never needs to be adjusted except by a specialized professional why would you make it accessible to anyone that doesn’t have tools? You can see this in systems that use proprietary screws to keep users out of the system or glue parts together to prevent easy access.


Tom’s Take

I miss the ability to open up my laptop and fix issues with the add-in cards. The tinkerer in me likes to learn about how a system works. But I don’t miss the necessity to go in and do it. The usability of a system that “just works” is way more useful to me. It reminds me of my dad in some ways. He may have loved the ability to open up the engine and play with the carburetor but he also had to do it frequently to make things “work right”. In some ways, removing our ability to repair things in the IT space has forced manufacturers to build devices that don’t need to be frequently repaired. It’s not the optimal solution for sure, but the trade off it more than worth it in my mind.

In Defense of Subscriptions

It’s not hard to see the world has moved away from discrete software releases to a model that favors recurring periodic revenue. Gone are the days of a bi-yearly office suite offering or a tentpole version of an operating system that might gain some features down the road. Instead we now pay a yearly fee to use the program or application and in return we get lots of new things on a somewhat stilted cadence.

There are a lot of things to decry about software subscription models. I’m not a huge fan of the way that popular features are put behind subscription tiers that practically force you to buy the highest, most expensive one because of one or two things you need that can only be found there. It’s a callback to the way that cable companies put their most popular channels together in separate packages to raise the amount you’re paying per month.

I’m also not a fan of the way that the subscription model is a huge driver for profits for investors. If your favorite software program doesn’t have a subscription model just yet you’d better hope they never take a big investment. Because those investors are hungry for profit. Profit that is easier to see on a monthly basis because people are paying a fee for the software monthly instead of having big numbers pop up every couple of years.

However, this post is about the positive aspects of software subscriptions. So let’s talk about a couple of them.

Feature Creatures

If you’ve ever complained that an application or a platform doesn’t have a feature that you really want to see you know that no software is perfect. Developers either put in the features they think you want to see or they wait for you to ask for them and then race to include them in the next release. This push-pull model dominates the way that traditional software has been written.

How do those features get added to software? They do have to be written and tested and validated before they’re published. That takes time and resources. You need a team to add the code and tools to test it. You need time to make it all work properly. In other words, a price must be paid to make it happen. In a large organization, adding those features can be done if there is sufficient proof that adding them will recoup the costs or increase sales, therefore justifying their inclusion.

How about if no one wants to pay? You don’t have to look far for this one. Most of the comments on a walled garden app store are constantly talking about how “this app would be great if it had this feature” or “needs to do “X” then it’s perfect!” but no one wants to pay to have those features developed. If the developer is already charging a minimum for the app or giving it away in hope of making up the difference through ad revenue or something, how can we hope to encourage them to write the feature we want if we’re not willing to pay for it?

This is also the reason why you’re seeing more features being released for software outside of what you might consider a regular schedule. Instead of waiting for a big version release to pack in lots of new features in the hopes of getting customers to pay for the upgrade developers are taking a different approach. They’re adding features as they are finished to show people the value of having the subscription. This is similar to the model of releasing the new episodes of a series on a streaming service weekly instead of all at once. You get people subscribed for longer periods instead of doing it only long enough to get the feature they’re looking for. Note that this doesn’t cover licensing that you must hold to use the feature.

Respect for the Classics

One other side effect of subscription licensing is support for older hardware. You may think this sounds counterintuitive. However, the likelihood that an older model of hardware is going to be supported later in life is due to the fact that there is no longer a need to force upgrades to justify hardware development.

Yes, there are always going to be releases that don’t support older hardware. If it’s something that needs to use a new chip or a new hardware feature it’s going to be very difficult to export that back to a platform that doesn’t have the hardware and never will. It’s also harder to code the feature in such as way as to run hardware that has varying degrees of support and implementation capabilities. The fact that an iPhone 7 and an iPhone 13 can run the same version of iOS is a huge example of this.

More to the point, the driver to force hardware upgrades is less impactful. If you’re writing a new app or new piece of software it’s tempting to make the new features run only on the latest version of the hardware. That practically forces the users to upgrade to take advantage of the new kit and new apps. It feels disingenuous because the old hardware may still have had a few more years of life in it.

Another factor here is that the added revenue from the subscription model allows developers to essentially “pay” to support those older platforms. If you’re getting money from customers to support the software you have the resources you need to port it to older version as well as continuing to support those older platforms with things like security updates. You may not get your favorite switch supported for a decade now but you also don’t have to feel like you’re being forced to upgrade every three years either.


Tom’s Take

I go back and forth with subscription models. Some tools and platforms need them. Those are the ones that I use every day and make lots of sense to me to keep up with. Others feel like they’re using the lure of more frequent feature releases to rope users into paying for things and making it difficult to leave. When you couple that with the drive of investors to have regular recurring revenue it puts companies into a tough spot. You need resources to build things and you need regular revenue to have resources. You can’t just hope that what you’re going to release is going to be popular and worry about the rest later. I don’t know that our current glut of subscription models is the answer to all our issues. However, it’s not the most evil thing out there.

Helpdesk Skills Fit the Bill

RedLEDKeyboard

I saw a great tweet yesterday from Swift on Security that talked about helpdesk work and how it’s nothing to be ashamed of:

I thought it was especially important to call this out for my readers. I’ve made no secret that my first “real” job in IT was on the national helpdesk for Gateway Computers through a contractor. I was there for about six months before I got a job doing more enterprise-type support work. And while my skills today are far above what I did when I started out having customers search for red floppy disks and removing helper apps in MSCONFIG, the basics that I learned there are things I still carry with me today.

Rocket Science

Most people have a negative outlook on helpdesk work. They see it as entry-level and not worth admitting to. They also don’t quite understand just how hard it is to do this kind of work. It may be different today compared to when I did it twenty years ago through the advances in technology but there is one group of people that know just how hard it is to do remote work on a system you can’t physically touch:

NASA Engineers

If you think it’s easy to just jump in front of a keyboard and fix an issue with Outlook or Chrome, try doing it without being able to see what you’re looking at. Now try and describe what you want to do without technical jargon and without using the term “thingie”. Now do it with the patience that it takes to tell someone more than once what they need to do and how to get them to read you the error message verbatim so you can figure out what’s actually going on.

Inbound phone support is a lot like fix a space probe. You have a lag time between commands being sent and confirmation. You can’t make it too complicated because the memory of the thing you’re working with isn’t unlimited. And you have to hope what you did actually fixes the issue because you may not get a second chance at it. It might sound a touch hyperbolic but I’d argue that phone support teaches the skills needed to communicate effectively and concisely to people less technical than you. You know, executives.

Joking aside, being able to distill the issue from incomplete information and formulate a response while simultaneously describing it in non-technical terms to ease implementation is the kind of skill you don’t read out of a book. Don’t believe me? Call your parents and help them update their DNS settings. Unless your parents are in IT I doubt it’s going to be a fun call. Tech is obtuse on purpose to keep people from playing with it. The people that can cut through the jargon and keep users going are magicians.

Visualize the Result

I have a gift for analogies. They aren’t always 100% accurate but I can usually relate some kind of a technology to some real-world non-technical thing to help users understand what’s going on. That gift was developed in the six months I spent trying to explain bad capacitors, startup TSRs, and exactly what an FDISK, Format, Reload process did to the computer. It’s a super valuable skill to have outside of tech too.

Go ask your doctor to explain how the endocrine system works in non-medical terms. I bet they can because they have a way to describe most of the body functions in terms that someone who has never taken human anatomy can understand. You can’t explain technical subjects with technical terms to just anyone. And even the technical people that know what you’re saying often have a hard time visualizing what you’re talking about. That’s how we ended up with the “Internet as a series of tubes” meme.

Where does this skill come in handy? How about teaching advanced tech to your junior staff? Or educating the executives in a board meeting? Or even just trying to tell the users in your organization why it’s not the network this time? If you can only talk in technical jargon without giving analogies or helping them visualize what you’re saying you will only end up with confused angry users that think you’re being patronizing.

The helpdesk forces you to get better at being descriptive with shorter words and more visual descriptions. I’d argue that my helpdesk time made my writing better because it reminds me not to be loquacious when I should be economical in my word choices and sentence lengths. If you can’t help your user figure out what you’re doing they’ll never learn how to be better at telling people what they need support for.

Documenting All the Things

The last skill that the helpdesk instilled in me is documentation. I’m a bad note taker. I get distracted (thanks ADHD) and often forget to write down important things. Helpdesk work is a place where you absolutely need to write everything down. Ask questions and record info. Dig into error messages and find out when they started happening. Listen to what people are telling you and don’t jump to conclusions until you’ve written it all down.

I once had a call where the user told me that they couldn’t get into their computer. I thought this was an easy fix. I spent ten minutes walking through the Windows XP login process. Nothing worked. I was getting frustrated and was about to reboot to safe mode and start erasing password hashes. By chance the user mentioned that they saw the pop up for the buzzer and it still wasn’t working. After I questioned them on this I realized that they could log in to WinXP just fine. The “buzzer popup” was AOL’s login screen. They had an issue with the modem, not the operating system. If I’d asked more questions about when the problem started and what they saw instead of just jumping to the wrong conclusion I could have saved a ton of pain and wasted effort.

Likewise, when you’re doing IT work you have to write it all down. Troubleshooting makes a lot of sense here but so does implementation or other kinds of design work. If you’re doing a wireless survey on site and forget to write down what the walls are made out of you may find yourself with an ineffective design or, worse having to spend extra time and money fixing it on the fly because those reinforced concrete walls are way better at signal attenuation than you recalled from a spotty memory.

Get in the habit of taking good notes from the start and summarizing them when needed to weed out the working things from the non-working things. Honestly, having a record of all the steps you took to arrive at your conclusion helps in the future if the issue happens again or you find yourself at a brick wall and need to retrace your steps to figure out where you took the wrong turn. If you record it all you won’t need to spend too much effort scratching your head to figure out what you were thinking.


Tom’s Take

Every job teaches you skills you can carry forward. Even the worst job in the world teaches you something you can take with you. Some jobs have a negative stigma and really shouldn’t. Like Swift said above you should accentuate the positive aspects of your journey. Don’t look at the helpdesk as a slog through the trenches. See yourself as a NASA rocket scientist that can talk to normal people and document like a champion. That’s a career anyone would be proud to have.

Fast Friday Thoughts on Leadership

I’m once more taking part in the BSA Wood Badge leadership course for my local council. I enjoy the opportunity to hone my skills when it comes to leading others and teaching them how to train their own leaders. A lot of my content around coaching, mentoring, and even imposter syndrome has come from the lessons I’ve learned during Wood Badge. It sounds crazy but I enjoy taking vacation time to staff something that looks like work because it feels amazing!

A few random thoughts from the week:

  • You need a sense of urgency in everything you do. You may not know exactly what’s coming or how to adjust for what needs to be done but you need to be moving with purpose to get it done. Not only does that help you with your vision to make things happen but it encourages others to do the same.
  • Team building happens when you’re not focused entirely on the goal. It doesn’t take much for your group to come together but it can only happen when they aren’t charging toward the finish line. Remember that taking a few moments here and there to reinforce the group dynamic can do a lot to build a cohesive unit. Stopping to smell the roses isn’t a bad thing when your team members chat about them too.
  • Make sure you appreciate your team when they succeed. The grindstone doesn’t feel quite so rough when you know that your effort is appreciated. It’s difficult some times to remember to recognize those that put in the work but if you do you’ll create happier people and more functional ones too.

Tom’s Take

I’m sure I’ll have more thoughts about leadership and team building as soon as I’m done with the practical application of it this weekend!

A Modest Proposal for Cisco Interface Naming

If you’re going to be configuring an interface in a switch, which one are you going to use? The interface has a name and a number based on where it is on the device. The numbering part is fairly easy to figure out. The module number comes first, followed by the slot, and finally the port. In the world of Cisco, which is the one I’m the most familiar with, that means a fixed configuration switch usually has interfaces labeled 0/24, with no module and the slot almost always being zero. With a modular switch the interface would be labeled 2/0/28 to indicate the 28th port on the second line card.

The issue arises when you factor in the first part of the interface naming convention. The nomenclature used in the Cisco world since the beginning of time has been the interface speed. If your interface is a 100Mbit Ethernet interface then the interface name is FastEthernet0/48. If you’re using a 1Gbit interface it’s GigabitEthernet0/48. If it’s a 10Gbit interface it becomes TenGigabitEthernet0/48. It’s a progression of interface speeds. Even if the port is capable of using 10/100/1000 the port is referred to at the highest speed. The 10Gbit ports don’t negotiate to a lower speed so they are always TenGigabit.

Now, with the new Catalyst 9000X Series switches you have the option of a port that can do 10/100/1000/2.5G/5G/10G. It’s a multi-gigabit capable port as outlined in this video from a recent Field Day event:

I don’t even want to think about the nightmare of trying to remember the name of the interface in a situation like that. It’s hard enough still to remember what the switches are capable of, whether it’s a gigabit port or a ten gigabit port. It’s hard enough for me but it’s impossible for anyone looking to make a script or automation that needs to be repeatable over a wide variety of switches.

Naming Convention Unity

The obvious solution here is actually present in another Cisco platform already. The Nexus series of switches running Cisco NX-OS already knows how to handle interface naming issues. It’s a problem that occurs when you could be running Gigabit Ethernet or Fibre Channel over Ethernet interfaces.

How does NX-OS fix this? The interface type is always Ethernet. It doesn’t matter what speed the interface is running at because it’s always Ethernet under the hood. So the every interface is labeled Ethernet(module/slot/port). It’s wonderful when you’re working on the device because you never have to make a typo based on the interface speed. It’s also consistent across every platform and module that could be installed.

The solution to the problem actually came from my friend Bruno Wollmann (@WollmanBruno) during the event. It’s well past time to adopt the NX-OS convention in every other version of Cisco IOS. You can alias the old interface names to the new nomenclature to ease the transition if you’d like. Otherwise it’s time to start making people use the basic convention of calling everything Ethernet.

This would ease new users into IOS by making all interfaces the same and focusing on the module/slot/port complexity as the primary place to learn about naming. It also helps with the development of automation and orchestration scripts. Rather than needing to detect device types or write different scripts for NX-OS or IOS-XE you can just use the same logic for both.


Tom’s Take

It’s time we move into the future of interface names. Bruno made an excellent point when he said that it’s time to adopt a standard across the whole line of switches. Cisco may not be ready to do a single OS but having unity between the features will go a long way to making it easier to work on the various platforms. You can do a lot with aliasing in the system to ease the transition but maybe by the time we get to IOS 20 we can use something that’s more manageable for the future.

No Is A Complete Sentence

Has someone asked you to do something recently that you know you don’t have time to do but felt like you needed to do anyway? Or has someone tried to get you to help with something and impressed upon you just how important it is? You probably told them “yes” out of guilt or obligation or some other kind of negative emotion. Sure, you could have declined but you thought about how bad you would feel if someone did the same to you.

Let me tell you clearly. “No” is a complete sentence. It requires no explanation or defense. It is the only thing you need to say when you know you won’t be able to do something no matter how much the other party tries to get you to agree.

Everything Sucking Equally

If you know anything about QoS, you know that once a given circuit reaches the limitation for bandwidth you can no longer send additional information. What’s counterintuitive about this is most people would assume that if you try to squeeze one more stream or packet into the mix that only that last packet would be affected and everything else would work perfectly fine, right? Only one of the many would suck.

Sadly, all of the packets or flows on the circuit are affected in this situation. Adding just one more thing to the mix means all things are affected because the system has to process the problems all at once. You suffer everywhere because the interface couldn’t say “no” to that last packet.

In fact, the whole mantra behind QoS is prioritizing information before it hits a given interface because the interface is just going to send whatever it is given. Those packets will queue up or eventually be dropped if they can’t be processed. Likewise, if you are trying to do too many things at once you will eventually miss something important and make a mistake because you couldn’t handle everything at once. You need to be able to prioritize all the things that are going on and know when you hit a limit you can’t get past.

Where the challenge comes into play is when others try to add more to the interface without knowing you’re at maximum capacity. It’s easy for someone to ask you to do just one more little thing. In their minds it’s not a huge ask, right? In your mind you’ve already begun trying to juggle all the things that need to be changed to make this one little thing work with all the other things you’ve spent so much time prioritizing. Unfortunately, it’s too easy to just say “yes” to avoid uncomfortable conversations and try to fix the problems as they come up rather than turning away something you know would put you over your limit.

Taking No For an Answer

It’s easy to tell people to say “no”, right? I mean it is possibly the shortest sentence in the English language. However, it’s not the person saying it that usually has an issue. The people doing the asking are the ones that almost invariably are the ones trying to convince, cajole, or outright impress upon the person how critical it is that whatever this is get done in addition to all the other things that are going on.

For those that are asking the question and expecting a positive answer, let me ask you a question of my own: Why does this person need to do the task? There usually are very good reasons for it. Perhaps it’s something this person specializes in. Maybe it’s something that only they have knowledge about. It could perhaps even be their job responsibility to get it done. But the next question you need to ask is even more important.

If they must do this task, what can I do to help them?

The calling card of bad management types is to show up, assign a bunch of tasks that only other people should do, and then fly away to measure their productivity and comment on how nothing seems to be getting done. Instead, management should be asking how they can help. Is this something that someone else could do? Do you have the bandwidth to do the task? Is there another task on your plate that I can do for you that would help you to take care of this?

People are often very fully tasked with all the things they have to take care of. Thinking that you can sneak one more thing onto their schedule without impacting all of it smacks of hubris. What if someone started adding tasks to your plate instead? I doubt you’d like that. Even if you said no, what if it was the CEO or the Chairman of the Board telling you that you had to do it? You’d probably be just as upset, especially if it was a task you couldn’t delegate.

For the people out there like me, try this next time someone asks you to do something and you’re fully tasked or unable to clear enough to accomplish it. Say “no”. Don’t explain. Don’t justify. Don’t defend. Just. Say. No. Watch the other person’s reaction. Are they getting frustrated because you’re not saying anything? Are they coming back in a helpful manner trying to understand why you declined? Or are they more upset that they now have to go find someone else to do it instead? If they come to you with respect and understanding you can tell them why you declined and are unable to help. They may offer suggestions or try to find a way to make it work. The ones that get upset are usually the ones that wouldn’t have taken “no” for an answer no matter what. They are the ones you don’t need to waste your breath on.


Tom’s Take

I will admit this post is as much for me as it is for others. I am the world’s worst for taking on additional things when I know I don’t have the bandwidth to do it all. I don’t like to disappoint. I really want to keep people happy. And instead I end up failing because I overestimate my ability to get work done and underestimate how much others could do if they knew I needed help. Next time you need to tell someone you can’t do something, don’t spend a ton of effort figuring out how to counter their eventual arguments. Just say “no” and move on. If it’s important or something that only you can do they will find a way to make it work. Otherwise, you will have saved so much more effort because you used a complete sentence instead of a needless paragraph.

Trust Will Do You In

If you’re a fan of the Gestalt IT Rundown that I do every week on the Gestalt IT YouTube channel, you have probably heard about the recent hacks of NVIDIA and Samsung. The original investigation into those hacks talked about using MDM platforms and other vectors to gain access to the information that was obtained by the hacking groups. An interesting tweet popped up on my feed yesterday that helped me reframe the attacks:

It would appear that the group behind these attacks are going after their targets the old fashioned way. With people. For illustration, see XKCD from 2009:

The Weakest Links

People are always the weakest link in any security situation. They choose to make something insecure through bad policy or by trying to evade the policy. Perhaps they are trying to do harm to the organization or even try to shine a light on corrupt practices. Whatever the reason, people are the weak link. Because you can change hardware or software to eliminate failures and bugs. You can’t reprogram people.

We’ve struggled for years to keep people out of our systems. Perimeter security and bastion hosts were designed to make sure that the bad actors stayed off our stage. Alas, as we’ve gotten more creative about stopping them we’ve started to realize that more and more of the attacks aren’t coming from outside but instead from inside.

There are whole categories of solutions dedicated to stopping internal attackers now. Data Loss Prevention (DLP) can catch data being exfiltrated by sophisticated attackers but it is more often used to prevent people from leaking sensitive data either accidentally or purposefully. There are solutions to monitor access to systems and replay logs to find out how internal systems folks were able to get privileges they should have.

To me, this is the crux of the issue. As much as we want to create policies that prevent people from exceeding their authority we seem to have a hard time putting them into practice. For every well-meaning solution or rule that is designed to prevent someone from gaining access to something or keep them secure you will have someone making a shortcut around it. I’ve done it myself so I know it’s pretty common. For every rule that’s supposed to keep me safe I have an example of a way I went around it because it got in my way.

Who Do YOU Trust?

This is one of the reasons why a Zero Trust Network Architecture (ZTNA) appeals to me. At its core it makes a very basic assumption that I learned from the X-Files: Trust No One. If you can’t verify who you are you can’t gain access. And you only gain access to the bare minimum you need and have no capability for moving laterally.

We have systems that operate like this today. Call your doctor and ask them a question. They will first need to verify that you are who you say you are with information like your birthdate or some other key piece of Personally Identifiable Information (PII). Why? Because if they make a recommendation for a medication and you’re not the person they think they’re talking to you can create a big problem.

Computer systems should work the same way, shouldn’t they? We should need to verify who we are before we can access important data or change a configuration setting. Yet we constantly see blank passwords or people logging in to a server as a super user to “just make it easier”. And when someone gains access to that system through clever use of a wrench, as above, you should be able to see the potential for disaster.

ZTNA just says you can’t do that. Period. If the policy says no super user logins from remote terminals they mean it. If the policy says no sensitive data access from public networks then that is the law. And no amount of work to short circuit the system is going to work.

This is where I think the value of ZTNA is really going to help modern enterprises. It’s not the nefarious actor that is looking to sell their customer lists that creates security issues. It does happen but not nearly as often as an executive that wants a special exception for the proxy server because one of their things doesn’t work properly. Or maybe it’s a developer that created a connection from a development server into production because it was easier to copy data back and forth that way. Whatever the reasons the inadvertent security issues cause chaos because they are configured and forgotten. At least until someone hacks you and you end up on the news.

ZTNA forces you to look at your organization and justify why things are the way they are. Think of it like a posture audit with immediate rule creation. If development servers should never talk to production units then that is the rule. If you want that to happen for some strange reason you have to explicitly configure it. And your name is attached to that configuration so you know who did it. Hopefully something like this either requires sign off from multiple teams or triggers a notification for the SOC that then comes in to figure out why policy was violated. At best it’s an accident or a training situation. At worst you may have just caught a bad actor before they step into the limelight.


Tom’s Take

Security isn’t perfect. We can always improve. Every time we build a good lock someone can build a better way to pick it. The real end goal is to make things sufficiently secure that we don’t have to worry about them being compromised with no effort while at the same time keeping it easy enough for people to get their jobs done. ZTNA is an important step because it creates rules and puts teeth behind them to prevent easy compromise of the rules by people that are trying to err on the side of easy. If you don’t already have plans to include ZTNA in your enterprise now you really should start looking at them. I’d tell you to trust me, but not trusting me is the point.

AI is a Promotion

When I worked at IBM as an intern, part of my job was writing a deployment script to help make our lives easier when installing new ThinkPads. In order to change an MTU setting on the token ring PCMCIA cards (long story), I had to write a script that iterated through all the possible combinations of adapters in the registry to find the one I was looking for and change the value.

Now, I was 22 at the time and green behind the ears, especially when it came to programming. I finally figured out that the most efficient way to do this in the language that I was using was a very deep nested if statement. It wasn’t my best work but it operated properly. I mentioned this to my mentors on my team with a remark of how hard it was to understand the logic at first. My comment was “You know, if it’s hard to read for anyone else then I never have to worry about gettin fired.”

To which the response was, “Yes, but you can never be promoted either.”

That sage wisdom brings me to the modern world and how AI can fix that issue.

Eternal Support

One of the first things you learn in IT is not to volunteer. If you know how to implement a technology or troubleshoot a specific issue then you become the go-to person for that problem. If you add a skill to the team like debugging Rust or configuring voice mail dial peers then you are the new owner for that particular skill.

This becomes annoying when you finally move past that skillset and you’re on to new and better things. If you are the person that knows how to extend database schemas or change auto attendant functions you are always going to be called upon to do those things. Rather than moving on to new challenges you will be called back to work on the things that no one else bothered to learn or couldn’t pawn off on someone else.

When you think about it, that’s part of the reason why so many entry-level positions start with some basics like VLAN programming or LUN configuration. Sure, you’re learning the basics and understanding how systems operate. But you’re also repeating the things that the junior person before you did. Why? Because they now have someone to do that work for them. And when the new person gets hired as a junior-level technician, you’re going to pass along those tasks to someone so they can learn and so you can move on. It’s the Circle of IT Life.

The reinforcement of knowledge is only so important to the organization. Yes, you need someone to keep the lights on and do the tasks regarded as menial by the senior teams. However, miring people in those tasks makes them feel undervalued after a while because no one wants to take them on. And, as above, if it’s something that no one else can be bothered to learn or take over you’re going to be stuck doing it until you leave or until the system in question is eventually replaced. And even then you’ll probably be asked to learn how to do the new thing because “this is your system”.

Learning, Not Thinking

The reason why I’m excited for applied AI in infrastructure and operations is because it will learn these menial tasks and do them as instructed every time without complaint or mistake. Yes, that means you need to train the algorithm correctly. Yes, that means you need to be on the lookout for changes and update accordingly when hardware changes or when procedure changes. Given those constraints AI will work for you, not against you.

When you move on to a new role you expect to face new challenges. If you move from a junior role to a senior role you want to work on harder things or solve new problems. You don’t want to get a new title with the same old work. If you change teams you want to experience new systems and implement new technology. You don’t want to be the cloud person that still deploys campus switches in wiring closets.

In a way, having AI take over the basic tasks of your job functions doesn’t remove your job. It promotes you to a new role. It could be that the first promotion is running the system that you implemented to do the work. A “train the trainer” kind of role, if you will. As soon as you can ensure that it is working correctly and that the rest of the team is aware of how to keep the engine running you can decide where you want to go next. You essentially have the freedom to learn and do and not have to be concerned about going back to doing the old things.

Is AI the Enemy?

AI doesn’t take away jobs. It takes away tasks. It streamlines processes and provides consistent, repeatable outcomes. If your job is a collection of tasks that need to be done then it is worth asking why it’s so easy for it to be replaced by an AI system. I’ve said on a number of occasions that there will always been room for jobs that AI can’t do, such as those that involve physical labor, as well as advanced thinking and creative problem solving.

Where AI does succeed is in the repeatable task department. If a role is comprised of repeatable tasks then the question becomes “why does this role exist?” If an algorithm can determine the best way to implement VLANs or offload clients to other devices then the job that was doing those roles needs to be examined. Can that person be encourage to do something else for the company? Can that role be reduced and given to someone with less experience? If those questions make you or someone on your team bristle, is that the fault of the AI? Or the fact of the job being less than ideal for the business?


Tom’s Take

AI can take away some of the unimportant and forgettable parts of your job and give you the freedom to do things that are more important to the business, possibly in a different role. Isn’t that the textbook definition of a promotion? Change makes people uncomfortable. It makes sense that no one wants to have their job taken away. But if you ask anyone in IT right now if they could get rid of 10-20% of the tasks in their job that they don’t like to do they’d agree in a heartbeat. AI isn’t going to remove your job. It’s going to give you the freedom to do something different. It’s also going to ensure you don’t have to keep getting called back to do that thing over and over again because no one else wants to learn it. I think it’s time we let new technologies like AI give everyone on the team the promotion they deserve.