During Networking Field Day 33 this week we had a great presentation from Graphiant around their solution. While the presentation was great you should definitely check out the videos linked above, Ali Shaikh said something in one of the sessions that resonated with me quite a bit:

Automation of an existing system doesn’t change the system.

Seems simple, right? It belies a major issue we’re seeing with automation. Making the existing stuff run faster doesn’t actually fix our issues. It just makes them less visible.

Rapid Rattletraps

Most systems don’t work according to plan. They’re an accumulation of years of work that doesn’t always fit well together. For instance, the classic XKCD comic:

When it comes to automation, the idea is that we want to make things run faster and reduce the likelihood of error. What we don’t talk about is how each individual system has its own quirks and may not even be a good candidate for automation at any point. Automation is all about making things work without intervention. It’s also dependent on making sure the process you’re trying to automate is well-documented and repeatable in the first place.

How many times have you seen or heard of someone spending hours trying to script a process that takes about five minutes to do once or even twice a year? The return on time investment in automating something like that doesn’t really make sense, does it? Sure, it’s cool to automate everything but it’s not really useful, especially if the task has changes every time it’s run that requires you to change in the inputs. It’s like building a default query for data that needs to be rewritten every time the query is run.

You’re probably laughing right now but you also have at least one or two things that would fit this bill. Rather than asking if you should be automating this task you should instead be asking why we’re doing it in the first place. Why are we looking to accomplish this goal if it only needs to be done on occasion? Is it something critical like a configuration backup? Or maybe just a sanity check to see that unused switch ports have been disabled or tagged with some kind of security configuration. Are you trying to do the task for safety or security? Or are you doing it for busy work purposes?

Streamlining the System

In all of those cases we have to ask why the existing system exists. That’s because investing time and resources into automating a system can result in a big overrun in budget when you run into unintended side effects or issues that weren’t documented in the first place. Nothing defeats an automation project faster than hitting roadblocks out of nowhere.

If you shouldn’t invest time in automating something that is already there, what should you do instead? How about reengineering the whole process instead? If you occasionally run configuration backups to make sure you have good copies of the devices why not institute change controls or rolling automatic backups? Instead of solving an existing problem with a script why shouldn’t you change the way you do things that might have other hidden benefits? If you’re scripting changes to ports to verify security status why not have a system in place that creates configuration on those ports when they’re configured and require change controls to enable them?

It feels like extra work. It always seems easier to jump in from the bottom up with both feet and work on a problem until you solve it. Top down means you’re changing the way the system does things instead so the problems either disappear or change to something more manageable. The important question to ask is “where are my resources best spent?” If you see your time as a resource to invest in projects are you better served making something existing work slightly faster? Or would it be better for you to take the time to do something in a different, potentially better way?

If you believe your process is optimized as much as possible and just needs to run on its own that makes for an easy conversation. But if you’re thinking you need to change the way you do things this is a great time to make those changes and use your time investment to do things properly this time around. You may have to knock down a few walls to get there but it’s way better than building a house of cards that is just going to collapse faster.

Tom’s Take

I’m a fan of automation. Batch files and scripting and orchestration systems have a big place in the network to reduce error and multiply the capabilities of teams. Automation isn’t a magic solution. It requires investment of time and effort and a return for the stakeholders to see value. That means you may need to approach the problem from a different perspective to understand what really should be done instead of just doing the same old things a little faster. Future you will thank you for reengineering today.