Ivan Pepelnjak has an interesting post up today about MacGyver-ing in the network. He and Simon Milhomme are right that most small-to-medium sized networks are pretty much non-reference architectures and really, really difficult to manage and maintain properly on the best of days. On the worst of days, they’re a nightmare that make you want to run screaming into the night. But why?

One Size Never Fits All

Part of the issue is that reference architectures and cookie-cutter designs aren’t made for SMEs. Sure, the large enterprise and cloud providers have their own special snowflakes. But so too do small IT shops that have been handed a pile of parts and told to make it work.

People like Greg Ferro and Peyton Maynard-Koran believe this is due to vendors and VARs pushing hardware and sales cycles like crazy. I have attributed it to the lack of real training and knowledge about networking. But, it also has a lot to do with the way that people see IT as a cost center. We don’t provide value like marketing. We don’t collect checks like accounting. At best, we’re no different than the utility companies. We’re here because we have to be.

Likewise, when IT is forced into making decisions based on some kind of rebate or sale we tend to get stuck with the blame when things don’t work correctly. People that wouldn’t bat an eye at buying a new Rolex or BMW get downright frugal when they’re deciding which switches to buy to run their operation for the next 10 years. So, they compromise. Surely you all can make this switch work? It only has half the throughput as the one you originally specced but it was on sale!

So, stuck with bad decisions and no real documentation or reference points, we IT wizards do what we can with what we have. Just like Angus MacGyver. Sure, those networks are a pain to manage. They’re a huge issue when it’s time to upgrade to the next discount switch. And given the number of fires that we have to fight on a daily basis we never get a chance to go back and fix when we nailed up in the first place. Nothing is more permanent than a temporary fix. And no install lasts longer than one that was prefaced with “let me just get this working for the next week”.

Bespoke Off The Rack

So, how do we fix the MacGyver-ing? It’s not going to be easy. It’s going to be painful and require us to step out of our comfort zones.

1. Shut Up And Do As I Say. This one is hard. Instead of having management breathing down your neck to get something installed or working on their schedule, you need to push back. You need to tell people that rushing is only going to complicate things. You need to fire back when someone hands you equipment that doesn’t meet your spec. You need to paraphrase the famous Shigeru Miyamoto – “A delayed network is eventually good, but a rushed installation is bad forever.”
2. Document Like It Will Be Read Aloud In Court. Documentation isn’t just a suggestion. It’s a necessity. We’ve heard about the “hit by a bus” test. It’s more than just that, though. You need to not only be able to replace yourself but also to be able to have references for why you made the decisions you did. MacGyver-ing happens when we can’t remember why we made the decisions we did. It happens when we need to come up with a last minute solution to an impossible problem created by other people (NSFW language). Better to have that solution documented in full so you know what’s going on three years from now.
3. Plan Like It’s Happening Tomorrow. Want to be ready for a refresh? Plan it today. Come back to your plan. Have a replacement strategy for every piece of equipment in your organization. Refresh the plan every few months. When someone comes to you with a new idea or a new thing, write up a plan. Yes, it’s going to be a lot of spinning your wheels. but it’s also going to be a better solution when someone comes to you and says that it’s time to make a project happen. Then, all you need to do is pull out your plan and make it happen. Imagine it like the opposite of a DR plan – Instead of the worst case scenario this is your chance to make it the best case.

Tom’s Take

There’s a reason the ringtone on my phone has been the theme from MacGyver for the last 15 years. I’m good at building something out of nothing. Making things work when they shouldn’t. And, as bad as it sounds, sometimes that’s what’s needed. Especially in the VAR world. But it shouldn’t be standard operating procedure. Instead, make your plan, execute it when the time comes, and make sure no one pushes you into a situation where MacGyver-ing is the last option you have.