Technical Debt or Underperforming Investment?

In this week’s issue of the Packet Pushers Human Infrastructure newsletter, there was an excellent blog post from Kam Lasater about how talking about technical debt makes us sound silly. I recommend you read the whole thing because he brings up some very valid points about how the way the other departments of the organization perceive our issues can vary. It also breaks down debt in a very simple format that takes it away from a negative connotation and shows how debt can be a leverage instrument.

To that end, I want to make a modest proposal to help the organization understand the challenges that IT faces with older systems and integration challenges. Except we need some new branding. So, I propose we start referring to technical debt as “underperforming technical investments”.

I’d Buy That For A Dollar

Technical debt is just a clever way to refer to the series of layered challenges we face from decisions that were made to accomplish tasks. It’s a burden we carry negatively throughout the execution of our job because it adds extra time to the process. We express it as debt because it’s a price that must be paid every time we need to log in to the old accounting system to make changes or we have to wake up on the third Sunday of the month to reboot the access points because of a software bug that we can’t fix.

As Lasater points out, the financial departments look at debt differently. It’s not always negative. Commercial paper allows for purchasing power beyond cash-on-hand. Mortgages and loans allow us to buy houses and cars that enable us to do much more with our lives. Reasonable debt can be good so long as what we get out of it is a net positive. Debt doesn’t become an issue until there’s too much of it for the return we get, which eventually leads to bankruptcy.

Let’s point out some of the challenges we face with technical things that we refer to as debt. We usually use it in reference to outdated systems or slower equipment that doesn’t allow us to do our jobs as effectively as possible. Unlike a house which still provides the value of a dwelling at any age given reasonable maintenance or a car that still operates with the same maintenance, technology changes more rapidly. The original iPhone is unable to operate in a modern environment because of outdated software or inability to connect to the mobile network.

In consumer tech, the idea of technical debt doesn’t exist. Because when things are “slow” or “broken” they get discarded and people spend money to get something new. Someone that is willing to live with a shattered phone screen because they don’t want to pay to fix it might be willing to wait another two months to get a new device because it supports the latest OS updates or has a better camera. The chase for new features is usually enough of a driver to get people to upgrade.

In the enterprise, the infrastructure equipment is expensive enough that we need to recognize the value of what we installed. We can’t just swap out all the switches in three years if we spent hundreds of thousands of dollars on them. Likewise, enterprise tech vendors are more likely to provide patches and updates for this gear knowing that customers are going to hold on to it for longer periods of time. And even then, just because something is EoL doesn’t mean it’s End of Use.

However, the disconnect between these two things comes when we talk about how equipment that is functional is also causing issues with our job. Maybe it’s an older, slower switch that drops more packets than the rest of the newer gear. It’s a problem but not big enough to justify buying a new one. It could be an older wireless controller that doesn’t have the capability to run the newest code because of an older CPU. We have gear that works but it doesn’t work as well as it could. But we also don’t have enough justification to get new stuff because the old things haven’t broken completely yet.

Putting Your Money Where Their Mouth Is

As Lasater says in the above article, talking about how it’s hard to do your job because everything is old doesn’t resonate with management. No one really wants to hear a department head whining about how their job is tough because they don’t have the latest toys. Admit it, you’d be upset if your CEO told you they couldn’t answer their email because they don’t have the newest MacBook Pro. Capable means functional in every enterprise.

However, I think the challenge we face and the solution to the problem is how we frame the discussion of the challenge itself. We talk about it as a debt that we feel saddled with. It’s something we did because we had to or something someone else chose to do that we have to live with. It feels like we’re paying a tax every day over something that we don’t like. But choices need to be made and we have to work with what we have. Unless we can provide for a way to get people to understand the tradeoffs.

Hence, my proposal above. First, we stop referring to IT infrastructure gear as “technical debt”. It’s not something we bought and now have to deal with. Our accounting departments are amortizing the acquisition costs of the equipment over a period of time and writing it off of taxes against recognized revenue. There is a real dollar amount attached to every month of the lifetime of a switch or a server. That’s how finance sees your equipment. It’s not debt.

It’s a technical investment.

That’s your key to helping them understand the challenges you face. You’re not complaining about doing your job more slowly. Instead, you’re pointing out that you are investing a resource (your time) into a process that could have reduced resource requirements (again, your time and job execution time) if they would just invest in a different solution.

You’re not telling them they bought a lemon. Instead, you’re just telling them that the thing they invested in years ago isn’t performing as well as it could based on new data. If you need to break it down even further for your executive team you can just equate it to a home loan refinancing. If interest rates have dropped since you bought your home, doesn’t it make a ton of sense to refinance and take advantage of the savings? Likewise, if the speed or power of a system has increased in the past four years doesn’t it make sense to invest in something new?

When you change the discussion to focus on investment of resources instead of dealing with debt you change the way that the people will look at the return on that investment. If you just talk about debt the executives are going to easily be able to dismiss you because there is no upside to the conversation. They have to pay money to replace things so why spend it? If you talk about how much they could save on labor or how much more efficiently things could run if they made some changes or bought something new then that is harder to ignore. If their existing investment is underperforming some baseline on the stock market they’re going to want to move it right away. If the company’s stock is underperforming you can better believe some of the shareholders are going to want to move their investment too.

Here Comes the Money

One word of warning here. You can’t go in to this conversation with vague assurances that spending more money will make money. You have to come with numbers that support your case. Some of them are easy. You know what the new system will cost. You know how much time it will take to implement it. What you need to bring is the cost of what you are paying now to do things inefficiently.

If your team of 5 has to spend two extra hours a week doing something because the system is old or doesn’t work well with others then you have ten hours of lost time. You need to figure out what the value of your time is. Big note here: Don’t just divide your yearly salary by 2000 and take the averages. That’s what you’re getting paid. That’s the debt that the company incurs to keep you on staff. Instead, find out what your rate is. If you are billed at a higher rate than your earnings, which is almost a guarantee, use that number instead. If you’re an engineer for a reseller your hourly billing rate is easy to find. If you work in corporate IT the rate is harder to figure out but if you need a rule of thumb just figure out what your hourly pay rate is (yearly salary divided by 2000 hours per year) and double it. That’s the kind of return that any company would love to have on an investment. If it’s costing the company that much to do something inefficiently then you can make your case to get it fixed or replaced.

Lastly, make sure to record the changes and the results if you manage to make it happen. It’s easy to chart when you take more time to do something to prove a point. But it’s easy to forget to do it when you have what you want. Eventually, someone that manages investments needs to know if it paid off. If there’s a simple number like a stock price it’s cut-and-dried. If there’s more to the calculation you need to do the work to prove why it’s better now. And you want to keep a running total so you can provide it on demand when asked. Just tell them you have more time to track those kinds of things now that they made a wiser investment.


Tom’s Take

No one likes debt. Even the best debt is still an obligation to pay back to someone. However, investments are something positive. Yes, they still require resources to be exchanged to receive a good or service. However, changing the terminology changes the perception of the result. Debts are incurred and paid back begrudgingly. Investments grow and provide additional resources throughout their lives. And, importantly, when an investment isn’t paying off the way to fix it is to reinvest. It’s a clever trick but one that should work much better than just whining about drowning in debt.

VARs See You As Technical Debt

I’ve worked for a Value Added Reseller (VAR) in the past and it was a good run of my career before I started working at Tech Field Day. The market was already changing eight years ago when I got out of the game. With the advent of the pandemic that’s especially true today. Quite a few of my friends say they’re feeling the pressure from their VAR employer to stretch beyond what they’re accustomed to doing or outright being treated in such a way as to be forced out or leaving on their own. They tell me they can’t quite understand why that’s happening. After some thought on the matter I think I know. Because you represent debt they need to retire.

Skill Up

We don’t start our careers knowing everything we need to know to make it. The industry spends a lot of time talking about careers and skill paths and getting your legs under you. Networking people need to learn Cisco or Juniper or whatever configuration language makes the most sense for them. Wireless people need to learn how to do site surveys and configure access points. Server people need to learn operating systems and hypervisors. We start accumulating skills to land jobs to earn money and hopefully learn more important skills to benefit our careers.

Who benefits from that learning though? You certainly do because you gain new ways to further your career. But your VAR gains value as well because they’re selling your skills. The “value added” part is you. When you configure a device or deploy a network or design a system you’re adding value through your skills. That’s what the VAR is charging for. Your skills are their business model. No VAR stays in business just reselling hardware.

Accumulating skills is the name of the game. Those skills lead to new roles and more responsibility. Those new roles lead to more money. Perhaps that means moving on to new companies looking to hire someone that has your particular expertise in an area. That’s a part of the game too, especially for VARs. And that’s where the whole debt mess starts.

Double Down on Debt

Your skills are valuable. They’re also debt. They represent a cost in time, money, and resources. The investment that your VAR makes in you is a calculated return on that debt. If your company primarily deploys Cisco networks then the training you get to install and configure Cisco switches is a return on your VAR being able to hire you out to do that skill. Being able to install and configure Juniper switches isn’t a valuable skill set for them unless they move into a new line of business.

People are no different. We acquire skills that suit us for a time that we may or may not use forever. It’s like riding a bike. We use it a lot when we’re young. We stop using it when we start to drive. We may start again when we need to use a bike for college or for living in a large city or if we pick up cycling or mountain biking as a sport. However, the bike riding skill is always there. It is a sunk cost for us because we acquired it and keep it with us.

For a VAR, your skill is not a sunk cost. It’s a graph of keeping the amount of billable hours you contribute above the line of debt that you create to the company. If you spend 85% of your time installing Cisco switches you are well above the debt line to the company. But if your company stops installing so many switches your value starts to fall as well. It could be that the technology is old and no one is buying it. It could be that companies have shifted the way they do business and need different resources and technology. It could be that a new partnership has created competition inside your organization.

No one wants to the be a last buggy whip manufacturer. VARs thrive on attacking markets that are hot with huge potential for profits. When a skill set becomes a commodity VARs are competing on pricing they can’t always win. That drives them to investigate new markets to offer to the customer base. In order to deliver those new technologies and solutions they need skilled people to install and configure them. The easiest solution is to acquire talent to make that happen. As above, VARs are always willing to pay top dollar to professionals with the specific skill sets they need. Bringing someone in to do that new line of business means they’re producing from Day One and keeping their value above the debt line of their salary.

The other way that VARs compete in these new markets is by training existing professionals on the new technology. Everyone that has ever worked in a VAR knows of the people that get tasked with learning how to deploy new storage systems, new network equipment, and even entirely new solutions that customers are asking for. I know I was that person at my old VAR. If it needed to be learned I was the one to do it first. I jumped in to deploying iSCSI storage, wireless access points, and even VoIP phone systems. Each time I had to spend time learning those new skills and adding them to my existing set. It was a cheaper method in the short term than bringing entirely new talent on board.

Get Out of Town

The friction in the training approach comes when it’s time to value your employees and their skill sets. If I’m getting paid to deploy Cisco switches and now my company wants me to learn how to install Palo Alto firewalls then I’m going to eventually get a raise or a new role to cover this expanded skill set. And rarely, if ever, do employee salaries get adjusted downward to compensate for old skills that are no longer relevant being supplanted by new marketable skills. Suddenly all those technologies I spent so much time learning are technical debt my VAR is paying for.

VARs need to be able to jump into new lines of business in order to survive. And that sometimes means shedding technical debt. If you’re a highly paid employee that earns twice as much as someone that has the specific skill set your VAR needs for a new project then your value to the at this current moment is likely much closer to the negative line of skills versus debt. You may have more experience or more familiarity with the process but that doesn’t translate as well into real value. If it did contractors wouldn’t be as well compensated as they are.

Now your VAR has a choice: keep paying you a lot and investing in their technical debt or bring on someone new that moves more closely with their new lines of business and start the escalator ride all over again. Unless you’re an exceptional employee or you are moved into a management role that usually means you’re let go or encourage to find another role somewhere. Maybe you get lucky and another VAR needs exactly what you offer and they’re willing to pay to get it. No matter what, the VAR is ridding themselves of technical debt. It should be no different than retiring an old laptop or installing new software to do help desk ticketing. But because it’s a person with a life and a family it feels wrong.

Rise Above

Is there an answer to this problem? If there is I don’t think we’ve found it yet. Obviously the solution would be to keep people on staff and pay them what their skill set is worth to the company. But that could entail retraining or readjustment in compensation that people aren’t always willing to do. VARs aren’t going to pay hefty salaries for skills that aren’t making them money. Other VARs may want to pay you for your skills but that’s not always a guarantee, especially if your skill set is extremely specific.

The other possibility is more akin to the contractor system, where you’re only hired for your skills for the period of time that they are needed. In theory that works very well. In practice the challenges of capital asset acquisition and personal benefits make contracting full-time almost as much of a hassle as changing jobs every few years chasing a bigger paycheck or a company that values your skills. There isn’t a clear-cut answer. Part of that reasoning is because the system works just fine the way it is. Why fix it if it’s not broken? It would take a massive shift in IT toward a new paradigm to force the kind of soul searching necessary to change the way VARs handle their staff. Cloud is close. So too is DevOps and programmatic IT. But for the kind of change we’re talking about it’s going to take something even bigger than those two things combined.


Tom’s Take

After reading this I’m sure half of you are scared to death and swear you will never work for a VAR. That’s a bit short-sighted. Remember that they’re a great source of training and experience. Customer networks stay fairly static and only require specific kinds of maintenance from time to time outside of deployments. If you want to hone your skills on a variety of technologies and get very good at troubleshooting then VAR life is absolutely where you need to be. Just remember that you are a resource with a value and a burden. Despite the mantra of being a “family” or other feel-good nonsense you will eventually reach the point of being the uncle that constantly incurs debt for very little return. Every family shuns those kinds of members. Make sure you know your value and how you can contribute. If that’s not possible where you are now then make sure it is wherever you land with whatever skills you need.

Fast Friday – Networking Field Day 22 Thoughts

Since I’m on the road again at Networking Field Day this week, I have had some great conversations with the delegates and presenters. A few stray thoughts that may develop into full blown blog posts at some point, but I figured I could get some of them out here for some quick entertainment.

  • The startup model means flexibility. That also means you can think about problems in a new light. So it would follow that you get to develop some new idea without a mountain of technical debt. Things like archaic platforms and crusty old user interfaces. You’d be surprised the amount of stuff that gets carried forward as technical debt.
  • Integrating products isn’t easy. Even if you think you’ve got the right slot for your newest acquisition you may find it isn’t the best fit overall. Or, even better, you may find a synergy you didn’t know existed because of a forgotten tool. Very rarely does anything just neatly fit into all your plans.
  • The more guest Wi-Fi I have to register for, the more I long for the days of Passport and OpenRoaming. If you already know who I am, why oh why must I continually register. Who wants to create Envoy, but for Wi-Fi?
  • There are days when I miss the CLI and doing stuff. Then I look at how complicated networks are now with the cloud and I realize I’d be in over my head. Also, no one wants to parse thousands of lines of log files. Even when I have insomnia.

Tom’s Take

I’ll have more good stuff soon. Don’t forget to check out the stuff I write for Gestalt IT, which includes posts from previous Field Day events and some briefings I’ve taken.

Chipping Away At Technical Debt

We’re surrounded by technical debt every day. We have a mountain of it sitting in distribution closets and a yard full of it out behind the data center. We make compromises for budget reasons, for technology reasons, and for political reasons. We tell ourselves every time that this is the last time we’re giving in and the next time it’s going to be different. Yet we find ourselves staring at the landscape of technical debt time and time again. But how can we start chipping away at it?

Time Is On Your Side

You may think you don’t have any time to work on the technical debt problem. This is especially true if you don’t have the time due to fixing problems caused by your technical debt. The hours get longer and the effort goes up exponentially to get simple things done. But it doesn’t have to be that way.

Every minute you spend trying to figure out where a link goes or a how a server is connected to the rest of the pod is a minute that should have been spent documenting it somewhere. In a text document, in a picture, or even on the back of a napkin stuck to the faceplate of said server!

I once watch an engineer get paid an obscene amount of money to diagram a network. Because it had never been done. He didn’t modify anything. He didn’t change a setting. All he did was figure out what went where and what the interface addresses were. He put it in Visio and showed it to the network administrators. They were so thrilled they had it printed in poster size and framed! He didn’t do anything above and beyond showing them what they already had.

It’s easy to say that you’re going to document something. It’s hard to remember to do it. It’s even harder when you forget to do it and have to spend an hour remembering why you did it in the first place. So it’s better to take a minute to write in an interface description. Or perhaps a comment about why you did a thing the way you did it. Every one of those minutes chips away a little more technical debt.

Ask yourself this question next time: If I spent less time solving documentation problems, what could I do to reduce debt overall?

Mad As Hell And Not Going To Design It Anymore

The other sources of technical debt are money and politics. And those usually come into play in the design phase. People jockeying for position or trying to save money for something else. I don’t mind honest budget saving measures. What I mind is trying to cut budgets for things like standing desks and quarterly bonuses. The perception of IT is that it is still a budget sink with no real purpose to support the business. And then the email server goes down. That’s why IT’s real value comes out.

When you design a network, you need to take into account many factors. What you shouldn’t take into account is someone dictating how things are going to be just because they like the way that something looks or sounds. When I worked at a VAR there were always discussions like this. Who should we use for the switching hardware? Which company has a great rebate this quarter? I have a buddy that works at Vendor X so we should throw him a bone this time.

As the networking professional, you need to stand up for your decisions. If you put the hard work and effort into making something happen you shouldn’t sit down and let it get destroyed by someone else’s bad idea. That’s how technical debt starts. And if you let it start this way you’ll never be able to get rid of it. Both because it will be outside of your control and because you’ll always identify someone else as the source of the problem.

So, how do you get mad and fix it before it starts? Know your project cold. Know every bolt and screw. Justify every decision. Point out the mistakes when they happen in front of people that don’t like to be embarrassed. In short, be a jerk. The kind of self-important, overly cerebral jerk that is smug because he knows he’s right. And when people know you’re right they’ll either take your word for things or only challenge you when they know they’re right.

That’s not to say that you should be smug and dismissive all the time. You’re going to be wrong. We all are. Well, everyone except my wife. But for the rest of us, you need to know that when you’re wrong you need to accept it and discuss the situation. No one is 100% infallible, but someone that can’t recognize when they are wrong can create a whole mountain range of technical debt before they’re stopped.


Tom’s Take

Technical Debt is just a convenient term for something we all face. We don’t like it, but it’s part of the job. Technical Debt compounds faster than any finance construct. And it’s stickier than student loans. But, we can chip away at it slowly with some care. Take a little time to stop the little problems before they happen. Take a minute to save an hour. And when you get into that next big meeting about a project and you see a huge tidal wave of technical debt headed your way, stand up and let it be known. Don’t be afraid to speak out. Just make sure you’re right and you won’t drown.