Positioning Policy Properly

Who owns the network policy for your organization? How about the security policy?Identity policy? Sound like easy questions, don’t they? The first two are pretty standard. The last generally comes down to one or two different teams depending upon how much Active Directory you have deployed. But have you ever really thought about why?

During Future:NET this week, those poll questions were asked to an audience of advanced networking community members. The answers pretty much fell in line with what I was expecting to see. But then I started to wonder about the reasons behind those decisions. And I realized that in a world full of cloud and DevOps/SecOps/OpsOps people, we need to get away from teams owning policy and have policy owned by a separate team.

Specters of the Past

Where does the networking policy live? Most people will jump right in with a list of networking gear. Port profiles live on switches. Routing tables live on routers. Networking policy is executed in hardware. Even if the policy is programmed somewhere else.

What about security policy? Firewalls are probably the first thing that come to mind. More advanced organizations have a ton of software that scans for security issues. Those policy decisions are dictated by teams that understand the way their tools work. You don’t want someone that doesn’t know how traffic flows through a firewall to be trying to manage that device, right?

Let’s consider the identity question. For a multitude of years the identity policy has been owned by the Active Directory (AD) admins. Because identity was always closely tied to the server directory system. Novell (now NetIQ) eDirectory and Microsoft AD were the kings of the hill when it came to identity. Today’s world has so much distributed identity that it’s been handed to the security teams to help manage. AD doesn’t control the VPN concentrator the cloud-enabled email services all the time. There are identity products specifically designed to aggregate all this information and manage it.

But let’s take a step back and ask that important question: why? Why is it that the ownership of a policy must be by a hardware team? Why must the implementors of policy be the owners? The answer is generally that they are the best arbiters of how to implement those policies. The network teams know how to translate applications in to ports. Security teams know how to create firewall rules to implement connection needs. But are they really the best people to do this?

Look at modern policy tools designed to “simplify” networking. I’ll use Cisco ACI as an example but VMware NSX certainly qualifies as well. At a very high level, these tools take into account the needs of applications to create connectivity between software and hardware. You create a policy that allows a database to talk to a front-end server, for example. That policy knows what connections need to happen to get through the firewall and also how to handle redundancy to other members of the cluster. The policy is then implemented automatically in the network by ACI or NSX and magically no one needs to touch anything. The hardware just works because policy automatically does the heavy lifting.

So let’s step back for moment and discuss this. Why does the networking team need to operate ACI or NSX? Sure, it’s because those devices still touch hardware at some point like switches or routers. But we’ve abstracted the need for anyone to actually connect to a single box or a series of boxes and type in lines of configuration that implement the policy. Why does it need to be owned by that team? You might say something about troubleshooting. That’s a common argument that whoever needs to fix it when it breaks is who needs to be the gatekeeper implementing it. But why? Is a network engineer really going to SSH into every switch and correct a bad application tag? Or is that same engineer just going to log into a web console and fix the tag once and propagate that info across the domain?

Ownership of policy isn’t about troubleshooting. It’s about territory. The tug-of-war to classify a device when it needs to be configured is all about collecting and consolidating power in an organization. If I’m the gatekeeper of implementing workloads then you need to pay tribute to me in order to make things happen.

If you don’t believe that, ask yourself this: If there was a Routing team and and Switching team in an organization, who would own the routed SVI interface on a layer 3 switch? The switching team has rights because it’s on their box. The routing team should own it because it’s a layer 3 construct. Both are going to claim it. And both are going to fight over it. And those are teams that do essentially the same job. When you start pulling in the security team or the storage team or the virtualization team you can see how this spirals out of control.

Vision of the Future

Let’s change the argument. Instead of assigning policy to the proper hardware team, let’s create a team of people focused on policy. Let’s make sure we have proper representation from every hardware stack: Networking, Security, Storage, and Virtualization. Everyone brings their expertise to the team for the purpose of making policy interactions better.

Now, when someone needs to roll out a new application, the policy team owns that decision tree. The Policy Team can have a meeting about which hardware is affected. Maybe we need to touch the firewall, two routers, a switch, and perhaps a SAN somewhere along the way. The team can coordinate the policy changes and propose an implementation plan. If there is a construct like ACI or NSX to automate that deployment then that’s the end of it. The policy is implemented and everything is good. Perhaps some older hardware exists that needs manual configuration of the policy. The Policy Team then contacts the hardware owner to implement the policy needs on those devices. But the Policy Team still owns that policy decision. The hardware team is just working to fulfill a request.

Extend the metaphor past hardware now. Who owns the AWS network when your workloads move to the cloud? Is it still the networking team? They’re the best team to own the network, right? Except there are no switches or routers. They’re all software as far as the instance is concerned. Does that mean your cloud team is now your networking team as well? Moving to the cloud muddies the waters immensely.

Let’s step back into the discussion about the Policy Team. Because they own the policy decisions, they also own that policy when it changes hardware or location. If those workloads for email or productivity suite move from on-prem to the cloud then the policy team moves right along with them. Maybe they add an public cloud person to the team to help them interface with AWS but they still own everything. That way, there is no argument about who owns what.

The other beautiful thing about this Policy Team concept is that it also allows the rest of the hardware to behave as a utility in your environment. Because the teams that operate networking or security or storage are just fulfilling requests from the policy team they don’t need to worry about anything other than making their hardware work. They don’t need to get bogged down in policy arguments and territorial disputes. They work on their stuff and everyone stays happy!


Tom’s Take

I know it’s a bit of stretch to think about pulling all of the policy decisions out of the hardware teams and into a separate team. But as we start automating and streamlining the processes we use to implement application policy the need for it to be owned by a particular hardware team is hard to justify. Cutting down on cross-faction warfare over who gets to be the one to manage the new application policy means enhanced productivity and reduced tension in the workplace. And that can only lead to happy users in the long run. And that’s a policy worth implementing.

The Sky is Not Falling For Ekahau

Ekahau Hat (photo courtesy of Sam Clements)

You may have noticed quite a few high profile departures from Ekahau recently. A lot of very visible community members, concluding Joel Crane (@PotatoFi), Jerry Olla (@JOlla), and Jussi Kiviniemi (@JussiKiviniemi) have all decided to move on. This has generated quite a bit of discussion among the members of the wireless community as to what this really means for the company and the product that is so beloved by so many wireless engineers and architects.

Putting the people aside for a moment, I want to talk about the Ekahau product line specifically. There was an undercurrent of worry in the community about what would happen to Ekahau Site Survey (ESS) and other tools in the absence of the people we’ve seen working on them for so long. I think this tweet from Drew Lentz (@WirelessNerd) best exemplifies that perspective:

So, naturally, I decided to poke back:

That last tweet is where I really want to focus this post.

The More Things Change

Let’s think about where Ekahau is with regards to the wireless site survey market right now. With no exaggeration, they are on top and clearly head and shoulders above the rest. What other product out there has the marketshare and mindshare they enjoy? AirMagnet is the former king of the hill but the future for that tool is still in flux with all of the recent movement of the tool between Netscout and now with NetAlly. IBWave is coming up fast but they’re still not quite ready to go head-to-head in the same large enterprise space. I rarely hear TamoGraph Site Survey brought up in conversation. And as for NetSpot, they don’t check enough boxes for real site survey to even really be a strong contender In the enterprise.

So, Ekahau really is the 800lb gorilla of the site survey market. This market is theirs to lose. They have a commanding lead. And speaking to the above tweets from Drew, are they really in danger of losing their customer base after just 12 months? Honestly? I don’t think so. Ekahau has a top-notch offering that works just great today. If there was zero development done on the platform for the next two years it would still be one of the best enterprise site survey tools on the market. How long did AirMagnet flounder under Fluke and still retain the title of “the best” back in the early 2010s?

Here Comes A Challenger

So, if the only really competitor that’s up-and-coming to Ekahau right now is IBWave, does that mean this is a market ripe for disruption? I don’t think that’s the case either. When you look at all the offerings out there, no one is really rushing to build a bigger, better survey tool. You tend to see this in markets where someone has a clear advantage. Without a gap to exploit there is no room for growth. NetSpot gives their tool away so you can’t really win on price. IBWave and AirMagnet are fighting near the top so you don’t have a way to break in beside them.

What features could you offer that aren’t present in ESS today? You’d have to spend 18-24 months to even build something comparable to what is present in the software today. So, you dedicate resources to build something that is going to be the tool that people wanted to use two years ago? Good luck selling that idea to a VC firm. Investors want return on their money today.

And if you’re a vendor that’s trying to break into the market, why even consider it? Companies focused on building APs and wireless control solutions don’t always play nice with each other. If you’re going to build a tool to help survey your own deployments you’re going to be unconsciously biased against others and give yourself some breaks. You might even bias your survey results in favor of your own products. I’m not saying it would be intentional. But it has been known to happen in the past.

Here’s the other thing to keep in mind: inertia. Remember how we posed this question with the idea that Ekahau wouldn’t improve the product at all? We all know that’s not the case. Sure, there are some pretty big names on that list that aren’t there any more. But those people aren’t all the people at Ekahau. Development teams continue to work on the product roadmap. There are still plans in place to look at new technologies. Nothing stopped because someone left. Even if the only thing the people on the development side of the house did was finish the plans in place there would still be another 12-18 months of new features on the horizon. That means trying to develop a competitor to ESS means developing technology to replace what is going to be outdated by the time you finish!

People Matter

That brings me back to the people. It’s a sad fact that everyone leaves a company sooner or later. Bill Gates left Microsoft. Steve Jobs left Apple. You can’t count on people being around forever. You have to plan for their departure and hope that, even if you did catch lightning in a bottle, you have to find a way to do it again.

I’m proud to see some of the people that Ekahau has picked up in the last few months. Folks like Shamree Howard (@Shamree_Howard) and Stew Goumans (@WirelessStew) are going to go a long way to keeping the community engagement alive that Ekahau is so known for. There will be others that are brought in to fill the shoes of those that have left. And don’t forget that for every face we see publicly in the community there is an army of development people behind the scenes working diligently on the tools. They may not be the people that we always associate with the brand but they will try hard to put their own stamp on things. Just remember that we have to be patient and let them grow into their role. They have a lot to live up to, so give them the chance. It may take more than 12 months for them to really understand what they got themselves into.


Tom’s Take

No company goes out of business overnight without massive problems under the hood. Even the biggest corporate failures of the last 40 years took a long time to unfold. I don’t see that happening to Ekahau. Their tools are the best. Their reputation is sterling. And they have a bit of a cushion of goodwill to get the next release right. And there will be a next release. And one after that. Because what Ekahau is doing isn’t so much scaling the mountain they climbed to unseat AirMagnet. It’s proving they can keep going no matter what.

Fast Friday – Mobility Field Day 4

This week’s post is running behind because I’m out in San Jose enjoying great discussions from Mobility Field Day 4. This event is bringing a lot of great discussion to the community to get everyone excited for current and future wireless technologies. Some quick thoughts here with more ideas to come soon.

  • Analytics is becoming a huge driver for deployments. The more data you can gather, the better everything can be. When you start to include IoT as a part of the field you can see why all those analytics matter. You need to invest in a lot of CPU horsepower to make it all work the way you want. Which is also driving lots of people to build in the cloud to have access to what they need on-demand from an infrastructure side of things.
  • Spectrum is a huge problem and source of potential for wireless. You have to have access to spectrum to make everything work. 2.4 GHz is pretty crowded and getting worse with IoT. 5 GHz is getting crowded as well, especially with LAA being used. And the opening of the 6 GHz spectrum could be held up in political concerns. Are there new investigations that need to happen to find bands that can be used without causing friction?
  • The driver for technology has to be something other than desire. We have to build solutions and put things out there to make them happen. Because if we don’t we’re going to stuck with what we have for a long time. No one wants to move and reinvest without clear value. But clear value often doesn’t develop until people have already moved. Something has to break the logjam of hesitance. That’s the reason why we still need bold startups with new technology jumping out to make things work.

Tom’s Take

I know I’ll have more thoughts when I get back from this event, but wireless has become the new edge and that’s a very interesting shift. The more innovation we can drive there means the more capable we can make our clients and empower users.

Fast Friday- Black Hat USA 2019

I just got back from my first Black Hat and it was an interesting experience. It was crazy to see three completely different security-focused events going on in town all at once. There was Black Hat, B-Sides Las Vegas, and DEFCON all within the space of a day or so of each other. People were flowing back and forth between them all and it was quite amazing.

A wanted to share a few quick thoughts about the event from my perspective being a first timer.

  • The show floor wasn’t as bit as VMworld or Cisco Live, but it was as big as it needed to be. Lots of companies that I’ve heard of, but several more that were new to me. That’s usually a good sign of lots of investment in the security space.
  • Speaking of which, I talked to quite a few companies about a variety of analytics, telemetry, and insider threat monitoring solutions. And almost all of them had a founder from Israel or someone that was involved in the cybersecurity areas of the IDF. That’s a pretty good track record for where the investment is going.
  • The Vegas booth gimmicks never change. I think I’ve spent too much time at Vegas conferences because I’m starting to recognize the magicians and other “performers” at the booths. I’m glad they can get some work but I don’t know if the companies realize that there needs to be some new blood out there.
  • I found it very different that you could print pretty much any name on your badge that you wanted. I saw a few El Chapos, Pablo Escobars, and even a generic “IT Buyer”. Consequently, people were a little curious about my Twitter badge flag. I guess the idea of announcing your identity to people is a bit strange at a security conference.
  • Being on the press list for the event meant that I got to see some cool briefings. But it also meant sorting through some things that didn’t make sense. And there there was the Quasi-Prime Number presentation spam that I got. I don’t go into much more detail other than to point you to this Twitter thread which is a comedy goldmine of the presentation referenced in said email. Thanks to @MalwareJake for pointing out the original thread and all the amazing comments about how the harmony of music can be an input into crypto randomization.
  • Lastly, I wish I would have had more time to go down and check out DEFCON. A lot of my friends that were in town were there and seemed to be having the time of their lives. DEFCON seems more in line with my Batman job instead of my Bruce Wayne job though. Guess I’ll have to take some vacation to check out DEFCON next year.

Ultimately, I had a great time checking out Black Hat. There were some parts that needed polish and some things about having 20,000+ in Vegas that I’m not keen on. But it’s a successful conference and likely will be one I attend in 2020. If for no other reason than to give my VPN a workout again!

 

Conference Packing – The Little Things

It seems like conference season never really ends. Between RSA, Cisco Live, Black Hat, and VMworld, I’m always running around to something. I enjoy being able to meet new people and talk to companies at these events but I also find that a little bit of planning ahead helps immensely.

There’s always a lot of discussion from people about what to pack for a conference. There have been some great posts written about it, like this one from Bob McCouch in 2014. He definitely covers all the important stuff that people would want to know, such as comfortable shoes and a bag big enough to carry extra things just in case you come back with enough fidget spinners to sink an aircraft carrier.

However, I’ve found in recent years that the difference between just surviving a conference and really being prepared involves a few extra items I never thought I’d need to bring back when I first started doing this in 2006. Maybe it’s the Scoutmaster in me, but being prepared has gone from being a suggestion to a necessity. And here are a few of those little necessities that I have found I can’t live without.

First? Aid.

I’ve found that traveling with a first aid kit is a huge upgrade in Quality of Conference Life. I’m not talking about one of the crazy backpack-style ones that first responders carry. Or even the small plastic ones that you can find in a local department store that have everything under the sun. No, the best first aid kit is the one you pack yourself. So you know you have what you need and you know what you have.

For my first aid kit, I pack small:

  • 3-4 bandages. Preferred to be breathable (not plastic or cute)
  • Antibiotic ointment
  • Moleskin for blisters
  • Cotton balls
  • Small alcohol swab (for cleaning and drying out blisters)
  • Q-Tips or other cotton swabs
  • Cuticle scissors

It’s a simple kit but it works wonders. You can take care of minor cuts and scrapes, blisters (which are the bane of every conference), and other things like wound treatment. You can even use cotton balls as earplugs in a pinch. The rest is designed to travel light.

Note that I didn’t list any pain relievers in there. That’s because I separately carry a lot of ibuprofen in my bag to help with tired muscles after standing all day and headaches after waking up. I carry enough that it won’t easily fit in the Ziploc bag that I use for my travel kit. It’s also easier to access in my bag without having to go into another bag. Make your kit easy to use and easy to access so you can get to it when you need it.

Portable Power

It’s funny how we’ve come to depend so much on our mobile devices now. I’ve gone from not even caring if I left my Nokia phone in my room to not being able to function without a smart device or two on me at all times. That also means that I’ve become hyper aware of how long I’m going to be able to use my device. And in places where there are a lot of phones competing for signal or a lot of interference, you’re going to drain your device battery a lot faster.

The other issue is that modern devices have much bigger batteries than in the past. My iPhone XS has a battery thats almost 2,700 mAh. My iPad Pro battery is 8,100 mAh. The battery in a MacBook is almost as much as well. Which means you’re going to either need to be tied to a power outlet often or you need to carry a battery pack.

Most conference guides I’ve seen will tell you to bring at least one battery pack. Since I’m crazy prepared, I always have two. One of them is bigger and designed to provide power on a regular basis away from a power outlet. It’s usually something above 10,000 mAh that takes a while to charge when it’s fully depleted. I’m about to upgrade to a newer unit that has USB-C PD charging and delivery and can recharge all my devices more quickly. The Wirecutter has some great reviews of bigger power banks to recharge all kinds of devices.

I also still need to carry a smaller battery pack for just my phone, especially when I want to travel light. And since I’m trying to travel light I don’t want to carry any extra things, like cables. Normally, I try to have a USB-C, micro USB, and Lightning cable at all times to handle any charging needs. But if it’s after hours and I’m just looking to have my phone charged so it doesn’t die, all I need it a Lightning cable. I’ve been using this Ventev PowerCell 6010+ for the last year thanks to an awesome friend and it does exactly what I need it to do. It recharges my phone more than once and fits in my pocket. The Lightning cable is also attached so I don’t need to worry about anything dangling out of my pocket. And in a pinch it can give a little extra juice to my iPad. You should check them out if you just need something small and simple.

Can You Hear Me Now?

The final thing I pack in my kit that seems out of the ordinary is earplugs. Why? Well, it turns out that conferences are loud. Like, really loud. And that means that you can’t even hear yourself think sometimes. This is especially true if you end up going to the big closing event. This usually involves a DJ or a band playing as loud as possible. And, depending on where you’re sitting or standing you may not be able to hear them clearly for the ringing in your ears.

Likewise, the conference floor is often a jumbled mess of booths, music, and even once a marching band! You need to have some kind of way to block out the noise without completely drowning out what is going on around you. Yes, I know it’s really easy to pop in a set of earbuds or put on a pair of over-the-ear headphones while you walk around. But in my line of work, I don’t want to be distracted by music either. I want the din of all the crowd to die down while I concentrate. It’s also a great way to make any workroom instantly quiet when I need to write up a report during an event.

If you happen to have a custom pair of earplugs already for some reason, such as swimming or shooting sports, you’re already ahead of the curve. Those things probably do an amazing job of blocking out everything. For those of us not lucky enough to have something custom, just hop down to a drugstore or department store and pick up on a set or three of the really cheap foam plugs. You can pass them out to your friends and even make a new one or two. Just don’t expect to converse a lot!


Tom’s Take

I find the little things are needed to make life more bearable. Because knowing that I have them makes me less likely to stress about all the crazy stuff that can happen during a conference. The unexpected happens all the time. Yet, by definition, we can’t expect it! But, if we know how to prepare for the majority of those things we can focus on having a good conference experience. We may not need a cell phone jammer or an oddly-specific size of metric wrench, but carrying the things above has really helped me when it comes to relaxing a bit at conferences.

IT Burnout – The Task List

Sadly, this picture above is me. I used to think I had one of the best memories in the world. It turns out my memory is well-suited for bar trivia and routing protocol esoterics. My memory doesn’t appear so adept at remembering other little things that are of more important, such as remembering to buy a gift for a birthday or following up on an email that I sent last week.

Human brains are great at processing information. But some of the ones that are best at processing it are horrible at recalling it. I think of it not unlike a three-tiered storage array. The fast access tasks are in the fastest storage tier where they are needed. The longer term but less important info goes into the near-line tier where it can be recalled when needed. And in my case, the bandwidth to that tier is slow and unreliable.

Exciting Things!

One of my solutions to this problem is getting better with task management. As bad as my memory is, it’s also not well suited to writing things down to remember them. The irony is almost too delicious to ignore. I need to write things down so I don’t forget them, but I forget to write them down. I’ve been studying more and more processes like Getting Things Done or Zen To Done to help me change the way I store and process information.

I’ve really been trying to use Things as my go-to task manager. Everyone has their favorites and the best task manager is the one you use on a regular basis. I’m trying to get better about using it to collect my thoughts so I can focus on things like writing posts and answering emails without dedicating time and mental bandwidth to actually remembering to do those things.

It’s handy to have a task manager on every device I use. I can drop tasks where I need them and when I need them. The ability to transfer notes and other writings to my devices and have it all sync automatically is wonderful. But it’s also a curse in disguise.

Too Many Things!

The curse of trying to capture all the information you want to remember is that you have to then process the things you remember for yourself. And trying to do that has made me realized I have a lot of things I want to try and keep going all at once. And even just seeing all the daily things I want to try and keep track of is reminding me that I have lots of juggling that I do frequently.

This idea of seeing all the things I have to do on a regular basis could easily lead to IT burnout in your job. I’ve felt it before in my engineering role when I looked at all the jobs on the board that needed to be done and realized I didn’t have enough hours in the day to finish them all on the schedule that they needed to be done.

It does feel good to see all the tasks that I’ve checked off the list as I get things accomplished, but I also know that I feel a sense of dread when I set a future date for an email or a contact to follow up on. The idea of putting it “out of sight, out of mind” lets me feel better about what I currently see. However, knowing there are things just waiting out there that need to be done another time or that things are getting carried over from day-to-day is enough to give pause.

More and more, I’m finding that the key that I need to adhere to is to work the tasks as they come up and find ways to get things accomplished instead of putting them off. Task management is great because it allows you to prioritize. But it also means it’s easy to delay and reschedule too. You have to find a way to make the things you do important enough so that getting them accomplished lets you clear your plate.

If you keep rescheduling and reassigning tasks, you’re going to get buried. If you can’t keep your plate clear it’s going to fill up before you know it. You have to find the big rocks and deal with them so you’re free to deal with the little ones.


Tom’s Take

No system is perfect. Everything can be refined given time. But you also have to figure out how best to work within your own limitations. For me, that’s realizing that I can’t start forgetting to write things down. It’s also realizing that I need to focus on the important stuff on my list and keep checking things off so I don’t get overwhelmed. Time is the ally of burnout. Given enough time anyone can end up burned out from overwork or from too much to accomplish. They key is to keep working through your list and don’t let things pile up on you.

I Was A 10x Engineer. And I’m Sorry.

You probably saw the big discussion this past weekend on Twitter about 10x Engineers. It all started with a tweet about how to recognize a 10x Engineer, followed by tons of responses about how useless they were and how people that had encountered them were happy to be rid of them. All that discussion made me think back to my old days as a Senior Network Rock Star. As I reminisced I realized that I was, in fact, a 10x Engineer. And I was miserable.

Pour Some Work On Me

I wasn’t always the epitome of engineering hatred. I used to be a wide-eyed technician with a hunger to learn things. I worked on a variety of systems all over the place. In fact, I was rising through the ranks of my company as a Novell Engineer in an environment with plenty of coverage. I was just learning the ropes and getting ready to take my place in a group of interchangeable people.

Then I started getting into networking. I spent more time learning about routers and switches and even firewalls. That meant that my skill set was changing from servers to appliances. It also meant that I was spending more and more time working on devices that no one else could work on. I had special knowledge that made me much more valuable to the organization. Soon, I found myself spending less and less time working on the Novell command line and more time working on the Cisco CLI.

That was the first extra “x” on my resume. Because I had special skills it meant that I was being relied upon more to do work that no one else could do. Suddenly I wasn’t just a replaceable cog in the machine. Instead, I was a critical part of the infrastructure that needed to be on-site for certain jobs and deployments. I knew that I needed to have someone else to help me out or I was going to quickly find myself overwhelmed with work. But networking wasn’t the thing that ended up pushing me all the way to 10x territory.

A Voice In The Wilderness

In order to truly become an insufferable 10x Engineering talent, I had to pivot into voice deployments. That’s because my skill set went from “important but we’re training others” to “so complicated no one else can understand this”. Thanks to my knowledge of networking, I was asked to pick up the voice banner and run with it. And I ran really, really far.

I was the only person in the office working on Cisco voice deployments. I had my own method of doing things. My own flow for deployments. My notes were contained in a OneNote file that only I had access to. You can probably see all the issues already. But I couldn’t. To me, this was my jam! I had all the tools and talent to make this happen. I could type MAC addresses faster than filling them into a BAT spreadsheet. I could configure crazy hacks to get around limitations thanks to all the extra research I was doing. I was invincible!

I was also gumming up the works. Voice deployments had to be constantly rescheduled if I was out of town. Vacations were a distant memory at best. I wasn’t just the most important cog in the machine – I was the machine. Nothing went forward unless I was doing it. And that’s not scalable at all. Even when my boss realized that I couldn’t scale any more he also knew that getting someone up to speed on my deployment methods and knowledge was a very daunting task at best.

This is the classic setup for a 10x Engineer. All the talent in the world and all of the hubris. Large portions of the line-of-business relying on their knowledge and process. No documentation. No way to get things done other than to go through me. If you’ve ever read The Phoenix Project by Gene Kim you realize that I was Brent through and through. In fact, when I finally did get around to reading it I self-identified with him before I’d even made it 100 pages in.

Pride Goeth Before

Ultimately, it wasn’t failure that caused my 10x Engineering career to come to an end. Instead, it was success. I left my old job six years ago to come to Tech Field Day. I knew I wouldn’t be fixing routers or voicemail systems any longer. I’d be in for an entirely new and different kind of work.

But I couldn’t see how my old job would be able to work without me. I secretly confided to some friends that I thought their business would fall apart without me. How could it survive? I was responsible for so much! I was the only one that knew all of these things! Even a lengthy two-week info dump wouldn’t be enough.

The pride and hubris I displayed is still shocking to me all these years later. To think that a company that had been in business for almost 20 years before I got there would go out of business months after I left because they couldn’t do what I was doing. They did stay in business. They changed, for sure. They moved away from my specialized knowledge and found new ways to utilize their talent. They were able to keep my systems up and running with my notes and when the time came to replace them they used new systems that didn’t need my help to manage and install. They survived because they realized that what I represented was special and couldn’t be replicated.

Me? I’m happy they did. I didn’t want my 10x Engineering efforts to be an anchor around either of our necks. I couldn’t go back to fixing my old systems with my new workload. And my old company couldn’t count on me being available to fix things when I was gone. They made the right choices to put themselves into a position to keep going. And just like most positive 10x Engineer stories they found a happy ending.


Tom’s Take

I’m not proud of my engineering roots when it comes to how bad I was at times. It wasn’t always intentional. It was a product of where I was and the work I was doing. But I totally couldn’t see the forest for the trees. I realize now that I should have taken more people under my wing and helped them understand what I was doing. I should have documented my work and used repeatable methods to build processes that could be done by anyone. My institutional knowledge should have been a resource, not a crutch. And I should have had the humility to understand that companies can live and grow past a single engineer. Knowing all that today makes me realize that I may have looked like a 10x Engineer but everyone is much better off now that I’m just a simple ex-engineer.