About networkingnerd

Tom Hollingsworth, CCIE #29213, is a former network engineer and current organizer for Tech Field Day. Tom has been in the IT industry since 2002, and has been a nerd since he first drew breath.

Mind the Air Gap

I recently talked to some security friends on a CloudBytes podcast recording that will be coming out in a few weeks. One of the things that came up was the idea of an air gapped system or network that represents the ultimate in security. I had a couple of thoughts that felt like a great topic for a blog post.

The Gap is Wide

I can think of a ton of classical air gapped systems that we’ve seen in the media. Think about Mission: Impossible and the system that holds the NOC list:

Makes sense right? Totally secure unless you have Tom Cruise in your ductwork. It’s about as safe as you can make your data. It’s also about as inconvenient as you can make your data too. Want to protect a file so no one can ever steal it? Make it so no one can ever access it! Works great for data that doesn’t need to be updated regularly or even analyzed at any point. It’s offline for all intents and purposes.

Know what works great as an air gapped system? Root certificate authority servers. Even Microsoft agrees. So secure that you have to dig it out of storage to ever do anything that requires root. Which means you’re never going to have to worry about it getting corrupted or stolen. And you’re going to spend hours booting it up and making sure it’s functional if you ever need to do anything with it other than watch it collect dust.

Air gapping systems feels like the last line of defense to me. This data is so important that it can never be exposed in any way to anything that might ever read it. However, the concept of a gap between systems is more appealing from a security perspective. Because you can create gaps that aren’t physical in nature and accomplish much of the same idea as isolating a machine in a cool vault.

Gap Insurance

By now you probably know that concepts like zero-trust network architecture are the key to isolating systems on your network. You don’t remove them from being accessed. Instead, you restrict their access levels. You make users prove they need to see the system instead of just creating a generic list. If you consider the ZTNA system as a classification method you understand more how it works. It’s not just that you have clearance to see the system or the data. You also need to prove your need to access it.

Building on these ideas of ZTNA and host isolation are creating virtual air gaps that you can use effectively without the need to physically isolate devices. I’ve seen some crazy stories about IDS sensors that have the transmit pairs of their Ethernet cables cut so they don’t inadvertently trigger a response to an intruder. That’s the kind of crazy preparedness that creates more problems that it solves.

Instead, by creating robust policies that prevent unauthorized users from accessing systems you can ensure that data can be safely kept online as well as making it available to anyone that wants to analyze it or use it in some way. That does mean that you’re going to need to spend some time building a system that doesn’t have inherent flaws.

In order to ensure your new virtual air gap is secure, you need to use policy manual programming. Why? Because a well-defined policy is more restrictive than just randomly opening ports or creating access lists without context. It’s more painful at first because you have to build the right policy before you can implement it. However, a properly built policy can be automated and reused instead of a just pasting lines from a text document into a console.

If you want to be sure your system is totally isolated, start with the basics. No traffic to the system. Then you build out from there. Who needs to talk to it? What user or class of users? How should they access it? What levels of security do you require? Answering each of those questions also gives you the data you need to define and maintain policy. By relying on these concepts you don’t need to spend hours hunting down user logins or specific ACL entries to lock someone out after they leave or after they’ve breached a system. You just need to remove them from the policy group or change the policy to deny them. Much easier than sweeping the rafters for super spies.


Tom’s Take

The concept of an air gapped system works for spy movies. The execution of one needs a bit more thought. Thankfully we’ve graduated from workstations that need biometric access to a world where we can control the ways that users and groups can access things. By thinking about those systems in new ways and treating them like isolated islands that must be accessed on purpose instead of building an overly permissive infrastructure in the first place you’ll get ahead of a lot of old technical debt-laden thinking and build a more secure enterprise, whether on premises or in the cloud.

Broadening Your Horizons, or Why Broadcom Won’t Get VMware

You might have missed the news over the weekend that Broadcom is in talks to buy VMware. As of right now this news is still developing so there’s no way of knowing exactly what’s going to happen. But I’m going to throw my hat into the ring anyway. VMware is what Broadcom really wants and they’re not going to get it.

Let’s break some of this down.

Broad Street

Broadcom isn’t just one of the largest chip manufactures on the planet. Sure, they make networking hardware that goes into many of the products you buy. Yes, they do make components for mobile devices and access points and a whole host of other things, including the former Brocade fibre channel assets. So they make a lot of chips.

However, starting back in November 2018, Broadcom has been focused on software acquisitions. They purchased CA Technologies for $19 billion. They bought Symantec the next year for $10 billion. They’re trying to assemble a software arm to work along with their hardware aspirations. Seems kind of odd, doesn’t it?

Ask IBM how it feels to be the dominant player in mainframes. Or any other dominant player in a very empty market. It’s lonely and boring. And boring is the exact opposite of what investors want today. Mainframes and legacy computing may be the only thing keeping IBM running right now. And given that Broadcom’s proposed purchase of Qualcomm was blocked a few years ago you can see that Broadcom is likely at the limit of what they’re going to be able to do with chipsets.

Given the explosion of devices out there you’d think that a chip manufacturer would want to double and triple down on development, right? Especially given the ongoing chip shortage. However, you can only iterate on those chips so many times. There’s only so much you can squeeze before you run out of juice. Ask Intel and AMD. Or, better yet, see how they’ve acquired companies to diversify into things like FPGAs and ARM-based DPUs. They realize that CPUs alone aren’t going to drive growth. There have to be product lines that will keep the investor cash flowing in. And that doesn’t come from slow and steady business.

Exciting and New

VMware represents a huge potential business arena for Broadcom. They get to jump into the data center with both feet and drive hybrid cloud deployments. They can be a leader in the on-prem software market overnight with this purchase. Cloud migrations take time. Software needs to be refactored. And that means running it in your data center for a while until you can make it work in the cloud. You could even have software that is never migrated for technical or policy-based reasons.

However, that very issue is going to cause problems for VMware and Broadcom. Is there a growth market in the data center in an enterprise? Do you see companies adding new applications and resources to their existing enterprise data centers? Or do you see them migrating to the cloud first? Do you imagine given the choice between building more compute cluster in the existing hybrid data center or developing for the cloud the first time that companies are going to choose the former?

To me, the quandary of VMware isn’t that different from the one faced by IBM. Yes, you can develop new applications to run on mainframes or on-prem data centers. But you can also not do that too. Which means you have to persuade people to use your technology. Maybe it’s for ease-of-management for existing operations teams. Could be an existing relationship that allows you to execute faster. But no matter what choice you make you’re incurring some form of technical debt when you choose to use existing technology to do something that could also be accomplished with new ideas.

Know who else hates the idea of technical debt and slow steady growth? That’s right, investors. They want flashy year-over-year numbers every quarter to prove they’re right. They want to see that stock price climb so they can eventually sell it and invest in some other growth market. Gone are the days when people were proud to own a bit of company and see it prosper over time. Instead, investors have a vision that lasts about 90 days and is entirely focused on the bottom line. If you aren’t delivering growth you don’t have value. It’s not even about making money any more. You have to make more all the time.

The Broad Elephant

The two biggest shareholders of VMware would love to see it purchased for a ton of money. Given the current valuation north of $40 billion, Dell and Silver Lake would profit handsomely from a huge acquisition. That could be used to pay down even more debt and expand the market for Dell solutions. So they’re going to back it if the numbers look good.

The other side of this equation is the rest of the market that thinks Broadcom’s acquisition is a terrible idea. Twitter is filled with analysts and industry experts talking about how terrible this would be for VMware customers. Given that Symantec and CA haven’t really been making news recently would tend to lend credence to that assessment.

The elephant in the room is what happens when customers don’t want VMware to sell? Sure, if VMware is allowed to operate as an independent entity like it was during the EMC Federation days things are good. They’ll continue to offer support and make happy customers. However, there is always the chance something will change down the road and force the status quo to change. And that’s the thing no one wants to talk about. Does VMware reject a very good offer in favor of autonomy? They just got out from under the relationship with Dell Technologies that was odd to say the least. Do they really want to get snapped up right away?


Tom’s Take

My read on this is simple but likely too simple. Broadcom is going to make a big offer based on stock price so they can leverage equity. The market has already responded favorably to the rumors. Dell and Silver Lake will back the offer because they like the cash to change their leverage situation. Ultimately, regulators will step in and decide the deal. I’m betting the regulators will say “no” like they did with Qualcomm. When the numbers are this big there are lots of eyeballs on the deal.

Ultimately Broadcom may want VMware and the biggest partners may be on board with it. But I think we’re going to see it fall apart because the approvals will either take too long and the stock price will fall enough to make it no longer worth it or the regulators will just veto the deal outright. Either way we’re going to be analyzing this for months to come.

Quality To Be Named Later

programming

First off, go watch this excellent video from Ken Duda of Arista at Networking Field Day 28. It’s the second time he’s knocked it out of the park when it comes to talking about code quality:

One of the things that Ken brings up in this video that I thought would be good to cover in a bit more depth is the idea of what happens to the culture of your organization, specifically code quality, when you acquire a new company. Every team goes through stages of development from formation through disagreement and finally to success and performance. One of the factors that can cause a high-performing team to regress back to a state of challenges is adding new team members to the group.

Let’s apply this lesson to your existing code infrastructure. Let’s say you’ve spent a lot of time building the best organization that has figured out and your dev teams are running like a well-oiled machine. You’re pushing out updates left and right and your users are happy. Then, you buy a company to get a new feature or add some new blood to the team. What happens when that new team comes on-board? Are they going to integrate into what you’ve been doing all this time? Do they have their own method for doing development? Are they even using the same tools that you have used? How much upheaval are you in for?

Buying Your Way To Consistency

Over a decade ago, United Airlines bought Continental Airlines. Two years later, the companies finally merged their ticketing systems. Well, merged might be a bit of a stretch. United effectively moved all their reservations to the Continental system and named the whole thing United. There are always challenges with these kinds of integrations but one might think that part of the reason for the acquisition was to move to a more modern reservation system.

United’s system was called Apollo and built in the 1970s. How could they move to a more modern system? Was the reason for the huge purchase of another airline directly related to their desire to adopt a newer, more flexible reservation system? There have certainly been suggestions of that for a number of years. But, more importantly, they also saw the challenges faced by one of their Star Alliance partner US Airways in 2007 when they tried a different approach to merging booking systems. The two radically different code bases clashed and created issues. And that’s for something as simple as an airline reservation system!

In the modern day we have much more control over the way that code is developed. We know the process behind what we do and what we write. When we build something we control it the entire way. However, that is true of everyone that writes code. And even with a large number of “best practices” out there no two developers are going to approach the problem the same way unless they work for the same company. So when you bring someone on board to your team through acquisition you’re also bringing in their processes, procedures, and habits. You have to own what they do because now their development quirks are part of your culture.

Bracing For the Impact

There’s a lot of due diligence that happens when companies are purchased. There’s an army of accountants that pore over the books and find all the potential issues. I’d argue that any successful merger in today’s world also needs to include a complete and thorough code review as well. You need to know how their culture is going to integrate into what you’re doing. Never mind the intellectual property issues. How do they name variables? Do they call the same memory allocation routines? Do they use floats instead of integers because that’s how they were taught? What development tools do they use and can those tools adapt to your workflow?

It may sound like I’m being a bit pedantic when I talk about variable naming being a potential issue but when it comes to code that is not the case. You’re going to have to train someone in procedure and you need to know who that is before they start committing code to your codebase. Those little differences are going to create bugs. Those bugs are going to creep into what you’re working on and create even more problems. Pretty soon something as simple as trying to translate from one IDE to another is going to create a code quality problem. That means your team is going to spend hours solving an issue that could have been addressed up front by figuring out how things were done in two different places. If you think that’s crazy, remember NASA lost a satellite over a unit conversion problem.


Tom’s Take

You may never find yourself in the shoes of someone like Ken Duda. He’s committed to quality software and also in charge of trying to integrate acquisitions with his existing culture. However, you can contribute to a better software culture by paying attention to how things are done. If you do things a certain way you need to document everything. You need to ensure that if someone new comes into the team that they can understand your processes quickly. That way they don’t spend needless hours troubleshooting problems that were lost in translation at the start of the process. Do the hard work up front so you aren’t calling people names later.

Friday Thoughts on the Full Stack

It’s been a great week at Networking Field Day 28 this week with some great presentations and even better discussions outside of the room. We recorded a couple of great podcasts around some fun topics, including the Full Stack Engineer.

Some random thoughts about that here before we publish the episode of the On-Premise IT Roundtable in the coming weeks:

  • Why do you need a full stack person in IT? Isn’t the point to have people that are specialized?
  • Why does no one tell the developers they need to get IT skills? Why is it more important for the infrastructure team to learn how to code?
  • We see full stack doctors, which are general practitioners. Why are there no full stack lawyers or full stack accountants?
  • If the point of having a full stack understanding is about growing non-tech skills why not just say that instead?
  • There’s value in having someone that knows a little bit about everything but not too much. But that value is in having them in a supervisor role instead of an operations or engineering role. Do you want the full stack doctor doing brain surgery? or do you want him to refer you to a brain surgeon?

Tom’s Take

Note that I don’t have all the answers and there are people that are a lot smarter than me out there that have talked a lot about the full stack engineering issue. My biggest fear is that this is going to become another buzzword just like 10x engineer that carries a lot of stigma about being less than useful while being a know-it-all. When the episode of the podcast comes out perhaps it will generate some good discussion on how we should be handling it.

Ease of Use or Ease of Repair

HammerAndSaw

Have you tried to repair a mobile device recently? Like an iPad or an MacBook? The odds are good you’ve never even tried to take one apart, let alone put in replacement parts. Devices like these are notorious to try and repair because they aren’t designed to be fixed by a normal person.

I’ve recently wondered why it’s so hard to repair things like this. I can recall opening up my old Tandy Sensation computer to add a Sound Blaster card and some more RAM back in the 90s but there’s no way I could do that today, even if the devices were designed to allow that to happen. In my thinking, I realized why that might be.

Build to Rebuild

When you look at the way that car engine bays were designed in the 80s and 90s you might be surprised to see lots of wasted space. There’s room to practically crawl in beside the engine and take a nap. Why is that? Why waste all that space? Well, if you’re a mechanic that wants to get up close and personal with some part of the engine you want all the space you can find. You’d rather waste a little space and not have busted knuckles.

A modern engine isn’t designed to be repaired by amateurs. The engine may be sideways or have replaceable parts put in places that aren’t easily accessed without specialized equipment. The idea is that professionals would have everything they need to remove the engine if necessary. Moreover, by making it harder for amateurs to try and fix the issues you’re are preventing them from doing something potentially problematic that you’ll have to fix later.

The same kind of thing applies to computing equipment. Whereas before you could open the case of a computer and try to fix the issues inside with a reasonable level of IT or electrical engineering knowledge you now find yourself faced with a compact device that needs specialized tools to open and even more specialized knowledge to fix. The average electrical engineer is going to have a hard time understanding the modern system-on-a-chip (SoC) designs when you disassemble your phone.

Usability Over Uniqueness

The question of usability is always going to come into play when you talk about devices. The original IBM PC was a usable design for the time it was built compared to mainframes. The iPhone is a usable design for the time when it was built as well. But compared to each other, which is the more usable design today?

Increasing usability often comes at the cost of flexibility. I was talking to someone at a fabrication lab about printer drivers for a laser etching machine. He said that installing the drivers for the device on Windows is easy but almost impossible on a Mac. I chuckled because the difficulty of installing a non-standard printer on a Mac is part of the reason why it’s easy to use for other kinds of printers. The printing subsystem traded usability with things like Bonjour printer installation over flexibility, like installing strange printers on a laptop.

Those trade offs extend into the hardware. Anyone that worked on PCs prior to Windows 95 knows all about fun things like IRQ conflicts, DIP switch settings for COM port addresses, and even more esoteric problems. We managed to get rid of those in the modern computing era but along with that went the ability to customize the device. You no longer have to do the dance of finding an available IRQ but you also can’t install a device that would conflict with another through hardware trickery.

By creating devices for the average user focused on usability you have to sacrifice the flexibility that makes them easier to fix. If you knew you were constantly going to need to be tweaking the jets on a carburetor you’d make it easy to access, right? But if a modern fuel injection system never needs to be adjusted except by a specialized professional why would you make it accessible to anyone that doesn’t have tools? You can see this in systems that use proprietary screws to keep users out of the system or glue parts together to prevent easy access.


Tom’s Take

I miss the ability to open up my laptop and fix issues with the add-in cards. The tinkerer in me likes to learn about how a system works. But I don’t miss the necessity to go in and do it. The usability of a system that “just works” is way more useful to me. It reminds me of my dad in some ways. He may have loved the ability to open up the engine and play with the carburetor but he also had to do it frequently to make things “work right”. In some ways, removing our ability to repair things in the IT space has forced manufacturers to build devices that don’t need to be frequently repaired. It’s not the optimal solution for sure, but the trade off it more than worth it in my mind.

In Defense of Subscriptions

It’s not hard to see the world has moved away from discrete software releases to a model that favors recurring periodic revenue. Gone are the days of a bi-yearly office suite offering or a tentpole version of an operating system that might gain some features down the road. Instead we now pay a yearly fee to use the program or application and in return we get lots of new things on a somewhat stilted cadence.

There are a lot of things to decry about software subscription models. I’m not a huge fan of the way that popular features are put behind subscription tiers that practically force you to buy the highest, most expensive one because of one or two things you need that can only be found there. It’s a callback to the way that cable companies put their most popular channels together in separate packages to raise the amount you’re paying per month.

I’m also not a fan of the way that the subscription model is a huge driver for profits for investors. If your favorite software program doesn’t have a subscription model just yet you’d better hope they never take a big investment. Because those investors are hungry for profit. Profit that is easier to see on a monthly basis because people are paying a fee for the software monthly instead of having big numbers pop up every couple of years.

However, this post is about the positive aspects of software subscriptions. So let’s talk about a couple of them.

Feature Creatures

If you’ve ever complained that an application or a platform doesn’t have a feature that you really want to see you know that no software is perfect. Developers either put in the features they think you want to see or they wait for you to ask for them and then race to include them in the next release. This push-pull model dominates the way that traditional software has been written.

How do those features get added to software? They do have to be written and tested and validated before they’re published. That takes time and resources. You need a team to add the code and tools to test it. You need time to make it all work properly. In other words, a price must be paid to make it happen. In a large organization, adding those features can be done if there is sufficient proof that adding them will recoup the costs or increase sales, therefore justifying their inclusion.

How about if no one wants to pay? You don’t have to look far for this one. Most of the comments on a walled garden app store are constantly talking about how “this app would be great if it had this feature” or “needs to do “X” then it’s perfect!” but no one wants to pay to have those features developed. If the developer is already charging a minimum for the app or giving it away in hope of making up the difference through ad revenue or something, how can we hope to encourage them to write the feature we want if we’re not willing to pay for it?

This is also the reason why you’re seeing more features being released for software outside of what you might consider a regular schedule. Instead of waiting for a big version release to pack in lots of new features in the hopes of getting customers to pay for the upgrade developers are taking a different approach. They’re adding features as they are finished to show people the value of having the subscription. This is similar to the model of releasing the new episodes of a series on a streaming service weekly instead of all at once. You get people subscribed for longer periods instead of doing it only long enough to get the feature they’re looking for. Note that this doesn’t cover licensing that you must hold to use the feature.

Respect for the Classics

One other side effect of subscription licensing is support for older hardware. You may think this sounds counterintuitive. However, the likelihood that an older model of hardware is going to be supported later in life is due to the fact that there is no longer a need to force upgrades to justify hardware development.

Yes, there are always going to be releases that don’t support older hardware. If it’s something that needs to use a new chip or a new hardware feature it’s going to be very difficult to export that back to a platform that doesn’t have the hardware and never will. It’s also harder to code the feature in such as way as to run hardware that has varying degrees of support and implementation capabilities. The fact that an iPhone 7 and an iPhone 13 can run the same version of iOS is a huge example of this.

More to the point, the driver to force hardware upgrades is less impactful. If you’re writing a new app or new piece of software it’s tempting to make the new features run only on the latest version of the hardware. That practically forces the users to upgrade to take advantage of the new kit and new apps. It feels disingenuous because the old hardware may still have had a few more years of life in it.

Another factor here is that the added revenue from the subscription model allows developers to essentially “pay” to support those older platforms. If you’re getting money from customers to support the software you have the resources you need to port it to older version as well as continuing to support those older platforms with things like security updates. You may not get your favorite switch supported for a decade now but you also don’t have to feel like you’re being forced to upgrade every three years either.


Tom’s Take

I go back and forth with subscription models. Some tools and platforms need them. Those are the ones that I use every day and make lots of sense to me to keep up with. Others feel like they’re using the lure of more frequent feature releases to rope users into paying for things and making it difficult to leave. When you couple that with the drive of investors to have regular recurring revenue it puts companies into a tough spot. You need resources to build things and you need regular revenue to have resources. You can’t just hope that what you’re going to release is going to be popular and worry about the rest later. I don’t know that our current glut of subscription models is the answer to all our issues. However, it’s not the most evil thing out there.

Helpdesk Skills Fit the Bill

RedLEDKeyboard

I saw a great tweet yesterday from Swift on Security that talked about helpdesk work and how it’s nothing to be ashamed of:

I thought it was especially important to call this out for my readers. I’ve made no secret that my first “real” job in IT was on the national helpdesk for Gateway Computers through a contractor. I was there for about six months before I got a job doing more enterprise-type support work. And while my skills today are far above what I did when I started out having customers search for red floppy disks and removing helper apps in MSCONFIG, the basics that I learned there are things I still carry with me today.

Rocket Science

Most people have a negative outlook on helpdesk work. They see it as entry-level and not worth admitting to. They also don’t quite understand just how hard it is to do this kind of work. It may be different today compared to when I did it twenty years ago through the advances in technology but there is one group of people that know just how hard it is to do remote work on a system you can’t physically touch:

NASA Engineers

If you think it’s easy to just jump in front of a keyboard and fix an issue with Outlook or Chrome, try doing it without being able to see what you’re looking at. Now try and describe what you want to do without technical jargon and without using the term “thingie”. Now do it with the patience that it takes to tell someone more than once what they need to do and how to get them to read you the error message verbatim so you can figure out what’s actually going on.

Inbound phone support is a lot like fix a space probe. You have a lag time between commands being sent and confirmation. You can’t make it too complicated because the memory of the thing you’re working with isn’t unlimited. And you have to hope what you did actually fixes the issue because you may not get a second chance at it. It might sound a touch hyperbolic but I’d argue that phone support teaches the skills needed to communicate effectively and concisely to people less technical than you. You know, executives.

Joking aside, being able to distill the issue from incomplete information and formulate a response while simultaneously describing it in non-technical terms to ease implementation is the kind of skill you don’t read out of a book. Don’t believe me? Call your parents and help them update their DNS settings. Unless your parents are in IT I doubt it’s going to be a fun call. Tech is obtuse on purpose to keep people from playing with it. The people that can cut through the jargon and keep users going are magicians.

Visualize the Result

I have a gift for analogies. They aren’t always 100% accurate but I can usually relate some kind of a technology to some real-world non-technical thing to help users understand what’s going on. That gift was developed in the six months I spent trying to explain bad capacitors, startup TSRs, and exactly what an FDISK, Format, Reload process did to the computer. It’s a super valuable skill to have outside of tech too.

Go ask your doctor to explain how the endocrine system works in non-medical terms. I bet they can because they have a way to describe most of the body functions in terms that someone who has never taken human anatomy can understand. You can’t explain technical subjects with technical terms to just anyone. And even the technical people that know what you’re saying often have a hard time visualizing what you’re talking about. That’s how we ended up with the “Internet as a series of tubes” meme.

Where does this skill come in handy? How about teaching advanced tech to your junior staff? Or educating the executives in a board meeting? Or even just trying to tell the users in your organization why it’s not the network this time? If you can only talk in technical jargon without giving analogies or helping them visualize what you’re saying you will only end up with confused angry users that think you’re being patronizing.

The helpdesk forces you to get better at being descriptive with shorter words and more visual descriptions. I’d argue that my helpdesk time made my writing better because it reminds me not to be loquacious when I should be economical in my word choices and sentence lengths. If you can’t help your user figure out what you’re doing they’ll never learn how to be better at telling people what they need support for.

Documenting All the Things

The last skill that the helpdesk instilled in me is documentation. I’m a bad note taker. I get distracted (thanks ADHD) and often forget to write down important things. Helpdesk work is a place where you absolutely need to write everything down. Ask questions and record info. Dig into error messages and find out when they started happening. Listen to what people are telling you and don’t jump to conclusions until you’ve written it all down.

I once had a call where the user told me that they couldn’t get into their computer. I thought this was an easy fix. I spent ten minutes walking through the Windows XP login process. Nothing worked. I was getting frustrated and was about to reboot to safe mode and start erasing password hashes. By chance the user mentioned that they saw the pop up for the buzzer and it still wasn’t working. After I questioned them on this I realized that they could log in to WinXP just fine. The “buzzer popup” was AOL’s login screen. They had an issue with the modem, not the operating system. If I’d asked more questions about when the problem started and what they saw instead of just jumping to the wrong conclusion I could have saved a ton of pain and wasted effort.

Likewise, when you’re doing IT work you have to write it all down. Troubleshooting makes a lot of sense here but so does implementation or other kinds of design work. If you’re doing a wireless survey on site and forget to write down what the walls are made out of you may find yourself with an ineffective design or, worse having to spend extra time and money fixing it on the fly because those reinforced concrete walls are way better at signal attenuation than you recalled from a spotty memory.

Get in the habit of taking good notes from the start and summarizing them when needed to weed out the working things from the non-working things. Honestly, having a record of all the steps you took to arrive at your conclusion helps in the future if the issue happens again or you find yourself at a brick wall and need to retrace your steps to figure out where you took the wrong turn. If you record it all you won’t need to spend too much effort scratching your head to figure out what you were thinking.


Tom’s Take

Every job teaches you skills you can carry forward. Even the worst job in the world teaches you something you can take with you. Some jobs have a negative stigma and really shouldn’t. Like Swift said above you should accentuate the positive aspects of your journey. Don’t look at the helpdesk as a slog through the trenches. See yourself as a NASA rocket scientist that can talk to normal people and document like a champion. That’s a career anyone would be proud to have.

Fast Friday Thoughts on Leadership

I’m once more taking part in the BSA Wood Badge leadership course for my local council. I enjoy the opportunity to hone my skills when it comes to leading others and teaching them how to train their own leaders. A lot of my content around coaching, mentoring, and even imposter syndrome has come from the lessons I’ve learned during Wood Badge. It sounds crazy but I enjoy taking vacation time to staff something that looks like work because it feels amazing!

A few random thoughts from the week:

  • You need a sense of urgency in everything you do. You may not know exactly what’s coming or how to adjust for what needs to be done but you need to be moving with purpose to get it done. Not only does that help you with your vision to make things happen but it encourages others to do the same.
  • Team building happens when you’re not focused entirely on the goal. It doesn’t take much for your group to come together but it can only happen when they aren’t charging toward the finish line. Remember that taking a few moments here and there to reinforce the group dynamic can do a lot to build a cohesive unit. Stopping to smell the roses isn’t a bad thing when your team members chat about them too.
  • Make sure you appreciate your team when they succeed. The grindstone doesn’t feel quite so rough when you know that your effort is appreciated. It’s difficult some times to remember to recognize those that put in the work but if you do you’ll create happier people and more functional ones too.

Tom’s Take

I’m sure I’ll have more thoughts about leadership and team building as soon as I’m done with the practical application of it this weekend!

A Modest Proposal for Cisco Interface Naming

If you’re going to be configuring an interface in a switch, which one are you going to use? The interface has a name and a number based on where it is on the device. The numbering part is fairly easy to figure out. The module number comes first, followed by the slot, and finally the port. In the world of Cisco, which is the one I’m the most familiar with, that means a fixed configuration switch usually has interfaces labeled 0/24, with no module and the slot almost always being zero. With a modular switch the interface would be labeled 2/0/28 to indicate the 28th port on the second line card.

The issue arises when you factor in the first part of the interface naming convention. The nomenclature used in the Cisco world since the beginning of time has been the interface speed. If your interface is a 100Mbit Ethernet interface then the interface name is FastEthernet0/48. If you’re using a 1Gbit interface it’s GigabitEthernet0/48. If it’s a 10Gbit interface it becomes TenGigabitEthernet0/48. It’s a progression of interface speeds. Even if the port is capable of using 10/100/1000 the port is referred to at the highest speed. The 10Gbit ports don’t negotiate to a lower speed so they are always TenGigabit.

Now, with the new Catalyst 9000X Series switches you have the option of a port that can do 10/100/1000/2.5G/5G/10G. It’s a multi-gigabit capable port as outlined in this video from a recent Field Day event:

I don’t even want to think about the nightmare of trying to remember the name of the interface in a situation like that. It’s hard enough still to remember what the switches are capable of, whether it’s a gigabit port or a ten gigabit port. It’s hard enough for me but it’s impossible for anyone looking to make a script or automation that needs to be repeatable over a wide variety of switches.

Naming Convention Unity

The obvious solution here is actually present in another Cisco platform already. The Nexus series of switches running Cisco NX-OS already knows how to handle interface naming issues. It’s a problem that occurs when you could be running Gigabit Ethernet or Fibre Channel over Ethernet interfaces.

How does NX-OS fix this? The interface type is always Ethernet. It doesn’t matter what speed the interface is running at because it’s always Ethernet under the hood. So the every interface is labeled Ethernet(module/slot/port). It’s wonderful when you’re working on the device because you never have to make a typo based on the interface speed. It’s also consistent across every platform and module that could be installed.

The solution to the problem actually came from my friend Bruno Wollmann (@WollmanBruno) during the event. It’s well past time to adopt the NX-OS convention in every other version of Cisco IOS. You can alias the old interface names to the new nomenclature to ease the transition if you’d like. Otherwise it’s time to start making people use the basic convention of calling everything Ethernet.

This would ease new users into IOS by making all interfaces the same and focusing on the module/slot/port complexity as the primary place to learn about naming. It also helps with the development of automation and orchestration scripts. Rather than needing to detect device types or write different scripts for NX-OS or IOS-XE you can just use the same logic for both.


Tom’s Take

It’s time we move into the future of interface names. Bruno made an excellent point when he said that it’s time to adopt a standard across the whole line of switches. Cisco may not be ready to do a single OS but having unity between the features will go a long way to making it easier to work on the various platforms. You can do a lot with aliasing in the system to ease the transition but maybe by the time we get to IOS 20 we can use something that’s more manageable for the future.

No Is A Complete Sentence

Has someone asked you to do something recently that you know you don’t have time to do but felt like you needed to do anyway? Or has someone tried to get you to help with something and impressed upon you just how important it is? You probably told them “yes” out of guilt or obligation or some other kind of negative emotion. Sure, you could have declined but you thought about how bad you would feel if someone did the same to you.

Let me tell you clearly. “No” is a complete sentence. It requires no explanation or defense. It is the only thing you need to say when you know you won’t be able to do something no matter how much the other party tries to get you to agree.

Everything Sucking Equally

If you know anything about QoS, you know that once a given circuit reaches the limitation for bandwidth you can no longer send additional information. What’s counterintuitive about this is most people would assume that if you try to squeeze one more stream or packet into the mix that only that last packet would be affected and everything else would work perfectly fine, right? Only one of the many would suck.

Sadly, all of the packets or flows on the circuit are affected in this situation. Adding just one more thing to the mix means all things are affected because the system has to process the problems all at once. You suffer everywhere because the interface couldn’t say “no” to that last packet.

In fact, the whole mantra behind QoS is prioritizing information before it hits a given interface because the interface is just going to send whatever it is given. Those packets will queue up or eventually be dropped if they can’t be processed. Likewise, if you are trying to do too many things at once you will eventually miss something important and make a mistake because you couldn’t handle everything at once. You need to be able to prioritize all the things that are going on and know when you hit a limit you can’t get past.

Where the challenge comes into play is when others try to add more to the interface without knowing you’re at maximum capacity. It’s easy for someone to ask you to do just one more little thing. In their minds it’s not a huge ask, right? In your mind you’ve already begun trying to juggle all the things that need to be changed to make this one little thing work with all the other things you’ve spent so much time prioritizing. Unfortunately, it’s too easy to just say “yes” to avoid uncomfortable conversations and try to fix the problems as they come up rather than turning away something you know would put you over your limit.

Taking No For an Answer

It’s easy to tell people to say “no”, right? I mean it is possibly the shortest sentence in the English language. However, it’s not the person saying it that usually has an issue. The people doing the asking are the ones that almost invariably are the ones trying to convince, cajole, or outright impress upon the person how critical it is that whatever this is get done in addition to all the other things that are going on.

For those that are asking the question and expecting a positive answer, let me ask you a question of my own: Why does this person need to do the task? There usually are very good reasons for it. Perhaps it’s something this person specializes in. Maybe it’s something that only they have knowledge about. It could perhaps even be their job responsibility to get it done. But the next question you need to ask is even more important.

If they must do this task, what can I do to help them?

The calling card of bad management types is to show up, assign a bunch of tasks that only other people should do, and then fly away to measure their productivity and comment on how nothing seems to be getting done. Instead, management should be asking how they can help. Is this something that someone else could do? Do you have the bandwidth to do the task? Is there another task on your plate that I can do for you that would help you to take care of this?

People are often very fully tasked with all the things they have to take care of. Thinking that you can sneak one more thing onto their schedule without impacting all of it smacks of hubris. What if someone started adding tasks to your plate instead? I doubt you’d like that. Even if you said no, what if it was the CEO or the Chairman of the Board telling you that you had to do it? You’d probably be just as upset, especially if it was a task you couldn’t delegate.

For the people out there like me, try this next time someone asks you to do something and you’re fully tasked or unable to clear enough to accomplish it. Say “no”. Don’t explain. Don’t justify. Don’t defend. Just. Say. No. Watch the other person’s reaction. Are they getting frustrated because you’re not saying anything? Are they coming back in a helpful manner trying to understand why you declined? Or are they more upset that they now have to go find someone else to do it instead? If they come to you with respect and understanding you can tell them why you declined and are unable to help. They may offer suggestions or try to find a way to make it work. The ones that get upset are usually the ones that wouldn’t have taken “no” for an answer no matter what. They are the ones you don’t need to waste your breath on.


Tom’s Take

I will admit this post is as much for me as it is for others. I am the world’s worst for taking on additional things when I know I don’t have the bandwidth to do it all. I don’t like to disappoint. I really want to keep people happy. And instead I end up failing because I overestimate my ability to get work done and underestimate how much others could do if they knew I needed help. Next time you need to tell someone you can’t do something, don’t spend a ton of effort figuring out how to counter their eventual arguments. Just say “no” and move on. If it’s important or something that only you can do they will find a way to make it work. Otherwise, you will have saved so much more effort because you used a complete sentence instead of a needless paragraph.