The Voice of SD-WAN

SD-WAN is about migrating your legacy hardware away from silos like MPLS and policy-based routing and instead integrating everything under one dashboard and one central location to make changes and see the impacts that those changes have. But there’s one thing that SD-WAN can’t really do yet. And that’s prepare us the for the end of TDM voice.

Can You Hear Me Now?

Voice is a way of life for some people. Cisco spent years upon years selling CallManager into every office they could. From small two-line shops to global organizations with multiple PRIs and TEHO configured everywhere. It was a Cisco staple for years. Which also had Avaya following along quickly to get into the act too.

Today’s voice world is a little less clear. Millenials hate talking on the phone. Video is an oddity when it comes to communications. Asynchronous chat programs like WhatsApp or Slack rule the day today. People would rather communicate via text than voice. We all have mobile devices and the phone may be one of the least used apps on it.

Where does that leave traditional voice services? Not in a good place for sure. We still need phone lines for service-focused businesses or when we need to call a hotline for support. But the office phone system isn’t getting any new features anytime soon. The phone system is like the fax machine in the corner. It’s a feature complete system that is used when it has to be used by people that are forced to use it unhappily.

Voice systems are going to stay where they are by virtue of their ubiquity. They exist because TDM technology hasn’t really advanced in the past 20 years. We still have twisted pair connections to deliver FXO lines. We still have the most basic system in place to offer services to our potential customers and users. I know this personally because when I finally traded out my home phone setup for a “VoIP” offering from my cable provider, it was really just an FXS port on the back of a residential cable modem. That’s as high-tech as it gets. TDM is a solved problem.

Call If You WANt To

So, how does SD-WAN play into this? Well, as it turns out, SD-WAN is replacing the edge router very quickly. Devices that used to be Cisco ISRs are now becoming SD-WAN edge devices. They aggregate WAN connections and balance between them. They take MPLS and broadband and LTE instead of serial and other long-forgotten connection methods.

But you know what SD-WAN appliances can’t aggregate? TDM lines. They don’t have cards that can accept FXO, FXS, or even PRI lines. They don’t have a way to provide for DSP add-in cards or even come with onboard transcoding resources. There is no way for an SD-WAN edge appliance to function as anything other than a very advanced packet router.

This is a good thing for SD-WAN companies. It means that they have a focused, purpose built device that has more software features than hardware muscle. SD-WAN should be all about data packets. It’s not a multitool box. Even the SD-WAN vendors that ship their appliances with LTE cards aren’t trying to turn them into voice routers. They’re just easing the transition for people that want LTE backup for data paths.

Voice devices were moved out of the TDM station and shelf and into data routers as Cisco and other companies tried to champion voice over IP. We’re seeing the fallout from those decisions today. As the data routing devices become more specialized and focused on the software aspects of the technology, the hardware pieces that the ISR platform specialized in are now becoming a yoke holding the platform back. Now, those devices are causing those platforms to fail to evolve.

I can remember when I was first thinking about studying for my CCIE Voice lab back in 2007-2008. At the time, the voice lab still have a Catalyst 6500 switch running in it that needed to be configured. It had a single T1 interface on a line card that you had to get up and running in CallManager. The catch? That line card would only work with a certain Supervisor engine that only ran CatOS. So, you have to be intimately familiar with CatOS in order to run that lab. I decided that it wasn’t for me right then and there.

Hardware can hold the software back. ISRs can’t operate voice interfaces in SD-WAN mode. You can’t get all the advanced features of the software until you pare the hardware down to the bare minimum needed to route data packets. If you need to have the router function as a TDM aggregator or an SBC/IPIPGW you realize that the router really should be dedicated to that purpose. Because it’s functioning more as a TDM platform than a packet router at that point.


Tom’s Take

The world of voice that I lived in five or six years ago is gone. It’s been replaced with texting and Slack/Spark/WebEx Teams. Voice is dying. Cell phones connect us more than we’ve ever been before but yet we don’t want to talk to each other. That means that the rows and rows of desk phones we used to use are falling by the wayside. And so too are the routers that used to power them. Now, we’re replacing those routers with SD-WAN devices. And when the time finally comes for use to replace those TDM devices, what will we use? That future is very murky indeed.

Advertisements

Is Training The Enemy of Progress?

Peyton Maynard-Koran was the keynote speaker at InteropITX this year. If you want to catch the video, check this out:

Readers of my blog my remember that Peyton and I don’t see eye-to-eye on a few things. Last year I even wrote up some thoughts about vendors and VARs that were a direct counterpoint to many of the things that have been said. It has even gone further with a post from Greg Ferro (@EtherealMind) about the intelligence level of the average enterprise IT customer. I want to take a few moments and explore one piece of this puzzle that keeps being brought up: You.

Protein Robots

You are a critical piece of the IT puzzle. Why? You’re a thinking person. You can intuit facts and extrapolate cause from nothing. You are NI – natural intelligence. There’s an entire industry of programmers chasing what you have. They are trying to build it into everything that blinks or runs code. The first time that any company has a real breakthrough in true artificial intelligence (AI) beyond complicated regression models will be a watershed day for us all.

However, you are also the problem. You have requirements. You need a salary. You need vacation time. You need benefits and work/life balance to keep your loved ones happy. You sometimes don’t pick up the phone at 3am when the data center blinks out of existence. It’s a challenge to get you what you need while still extracting the value that is needed from you.

Another awesome friend Andrew von Nagy (@RevolutionWiFi) coined the term “protein robots”. That’s basically what we are. Meatbags. Walking brains that solve problems that cause BGP to fall over when presented with the wrong routing info or cause wireless signals to dissipate into thing air. We’re a necessary part of the IT equation.

Sure, people are trying to replace us with automation and orchestration. It’s the most common complaint about SDN that I’ve heard to date: automation is going to steal my job. I’ve railed against that for years in my talks. Automation isn’t going to steal your job, but it will get you a better one. It will get you a place in the organization to direct and delegate and not debug and destroy. In the end, automation isn’t coming for your job as long as you’re trying to get a better one.

All Aboard The Train!

The unseen force that’s opposing upward mobility is training. In order to get a better job, you need to be properly trained to do it. Maybe it’s a lot of experience from running a network for years. Perhaps it’s a training class you get to go to or a presentation online you can watch. No matter what you need to have new skills to handle new job responsibilities. Even if you’re breaking new ground in something like AI development you’re still acquiring new skills along the way. Hopefully, if you’re in a new field, you’re writing them all down for people to follow in your footsteps one day.

However, training is in opposition to what your employer wants for you. It sounds silly, doesn’t it? Yet, we still hear the quote attributed to W. Edwards Deming – “What happens if we train our people and they leave? What happens if we don’t and they stay?” Remember, as we said above you are a protein robot that needs things like time off and benefits. All of those things are seen as an investment in you.

Training is another investment that companies like to tout. When I worked at a VAR, we considered ourselves some of the most highly trained people around. The owner told me when I started that he was “going to put half a million dollars into training me.” When I asked him about that number after five years he told me it felt like he put a kid through college. And that was before my CCIE. The more trained people you have, the easier your job becomes.

But an investment in training can also backfire. Professionals can take that training and put it to use elsewhere. They can go to a different job and take more money. They can refuse to do a job until they are properly trained. The investments that people make in training are often unrealized relative to the amount of money that it costs to make it happen.

It doesn’t help that training prices have skyrocketed. It used to be that I just needed to go down to the local bookstore and find a copy of a CCNA study guide to get certified. I knew I’d reached a new point in my career when I couldn’t buy my books at the bookstore. Instead, I had to seek out the knowledge that I needed elsewhere. And yes, sometimes that knowledge came in the form of bootcamps that cost thousands of dollars. Lucky for me that my employer at the time looked at that investment and said that it was something they would pick up. But I know plenty of folks that can’t get that kind of consideration. Especially when the training budget for the whole department won’t cover one VMware ICM class.

Employers don’t want employees to be too trained. Because you have legs. Because you can get fed up. Because you can leave. The nice thing about making investments in hardware and software is that it’s stuck at a location. A switch is an asset. A license for a feature can’t be transferred. Objects are tied to the company. And their investments can be realized and recovered. Through deprecation or listing as an asset with competitive advantage companies can recover the value they put into a physical thing.

Professionals, on the other hand, aren’t as easy to deal with. Sure, you can list a CCIE as an important part of your business. But what happens if they leave? What happens when they decide they need a raise? Or, worse yet, when they need to spend six months studying for a recertification? The time taken to learn things is often not factored into the equation when we discuss how much training costs. Some of my old coworkers outright refused to get certified if they had to study on their own time. They didn’t want their free non-work time taken up by reading over MCSE books or CCNA guides. Likewise, the company didn’t want to waste billable hours from someone not providing value. It was a huge catch-22.

Running In Place

Your value is in your skillset at the company you work for. They derive that value from you. So, they want you to stay where you are. They want you trained just enough to do your job. They don’t want you to have your own skillset that could be valuable before they get their value from you. And they definitely don’t want you to take your skillset to a competitor. How can you fix that?

  1. Don’t rely on your company to pay for your training. There are a lot of great resources out there that you can use to learn without needed to drop big bucks for a bootcamp. Use bootcamps to solidify your learning after the fact. Honestly, if you’re in a bootcamp to learn something you’re in the wrong class. Read blogs. Buy books from Amazon. Get your skills the old fashioned way.
  2. Be ready to invest time. Your company doesn’t want you using their billable time for learning. So that means you are going to make an investment instead. The best part is that even an hour of studying instead of binge watching another episode of House is time well-spent on getting another important skill. And if it happened on your own time, you’re not going to have to pay that back.
  3. Be ready to be uncomfortable. It’s going to happen. You’re going to feel lost when you learn something new. You’re going to make mistakes while you’re practicing it. And, honestly, you’re going to feel uncomfortable going to your boss to ask for more money once you’re really good at what you’re doing. If you’re totally comfortable when learning something new, you’re doing it wrong.

Tom’s Take

Companies want protein robots. They want workers that give 125% at all times and can offer a wide variety of skills. They want compliant workers that never want to go anywhere else. Don’t be that robot. Push back. Learn what you can on your time. Be an asset but not an immobile one. You need to know more to escape the SDN Langoliers that are trying to eat your job. That means you need to be on the move to get where you need to be. And if you sit still you risk becoming a permanent asset. Just like the hardware you work on.

Time To Get Back To Basics?

I’ve had some fascinating networking discussions over the past couple of weeks at Dell Technologies World, Interop, and the spring ONUG meeting. But two of them have hit on some things that I think need to be addressed in the industry. Both Russ White and Ignas Bagdonas of the IETF have come to me and talked about how they feel networking professionals have lost sight of the basics.

How Stuff Works

If you walk up to any network engineer and ask them to explain how TCP works, you will probably get a variety of answers. Some will try to explain it to you in basic terms to avoid getting too in depth. Others will swamp you with a technical discussion that would make the protocol inventors proud. But still others will just shrug their shoulders and admit they don’t really understand the protocol.

It’s a common problem when a technology gets to the point of being mature and ubiquitous. One of my favorite examples is the fuel system on an internal combustion engine. On older cars or small engines, the carburetor is responsible for creating the correct fuel and air mixture that is used to power the cylinders. Getting that mix right is half science and half black magic. For the people that know how to do it properly it’s an easy thing that they can do to drive the maxim performance from an engine. For those that have tried it and failed, it’s something best left alone to run at defaults.

The modern engine uses fuel injection. It’s a black box. It can be reprogrammed or tinkered with but it’s something that is tuned in different ways from the traditional carburetor. It’s not something that’s designed to be played around with unless you really know what you’re doing. The technology has reached the point where it’s ubiquitous and easy to use, but very difficult to repair without a specialized skill set.

Most regular car drivers will look under the hood and quickly realize they know nothing about what’s going on. Some technical folks will be able to figure out what certain parts do by observing their behavior. But if you ask those same people how a fuel injection system or carburetor works they’ll probably give you a blank stare.

That’s the issue we find in modern networking. We’ve been creating VLANs and BGP route maps for years. Some people have spent their entire careers tuning multicast or optimizing layer 2 interconnects. But if you corner them and ask them how the protocol works or how best to design an architecture that takes advantage of their life’s work they can’t tell you aside from referencing some old blog post or some vendor’s validated design on their hardware.

Russ and Ignas each touch on something important. In the good old days before there were a hundred certification guides and a thousand SRNDs people had to do real work to find the best solution for a problem. They had to put pencil to paper and sort out the mess before they could make something work. That’s where the engineering side of the network comes from.

Today, it’s more “plug and play”. You drop in pieces of a solution and they should work together. In practice, that usually means all the pieces have to be from the same vendor or from approved partner sources. And anything that goes awry will need a team of experts and many, many consulting hours to figure out.

Imagine if we could only install networks without understanding how they work. Could you see a world where everything we install from a networking perspective is a black box like a fuel injector? That’s the case to a certain degree with cloud networking today. We don’t see what’s going on under the surface. We can only see what the interface exposes to us. That’s fine as long as the applications we are using support the things we’re trying to do with them. But when it comes to being able to fix the network at the level we’re used to seeing it could be difficult if not downright impossible.

Learning The Ropes

But, moreover, are the networking professionals that are configuring these networks even capable of making those changes? Does anyone other than Narbik really understand how EIGRP works? Facebook seems to think that lightweight messaging packets for routing protocols are outdated. So they used ZeroMQ without understanding why that’s a bad idea for slow speed links. They may understand how a routing protocol works in theory, but they don’t completely understand how it’s supposed to work in extreme cases.

Can we teach people the basics and understanding of protocols and design that they need in order to make proper designs outside of vendor reference documents? It’s a tall order for sure. Most blog posts are designed to talk about features or solve problems. Most literature from creators is designed to make their new widget work correctly. Very little documentation exists about integration or design. And a good portion of what does exist is a bit outmoded and needs to be spruced up.

We, as the stewards of networking, need to help this process along. We need to spend more time talking about design and theory. We need to dissect protocols and help people understand how to use the tools they have rather than hoping someone will build the best mousetrap ever to solve each piece of a complicated puzzle. We need to teach people to be thinkers and problem solvers. And, yes, that does mean a bit less complaining about things like vendor code quality and VAR behavior.

Why? Because those people are empowered by a lack of knowledge. Customers aren’t idiots. They have business reasons for the things they do. Our technology needs to support their business needs. Yes, that means we need to think critically about what we’re doing. And yes, they may mean eating our words now and then to avoid a showdown about something that’s ultimately unimportant in the long run.

If we increase the amount of knowledge about the important topics like design and protocols it should make the overall level of understanding go up for everyone. That means better designs. More integrated technology. Less reliance on being force-fed the bare minimum information necessary to make things work. And that means things will run faster and much more smoothly. Which is a win for everyone.


Tom’s Take

I’ll be the first to admit that I don’t know the dirty mechanics of Frame Relay switching or how to tune OSPF Hello timers for non-standard link types. It’s a skill I don’t use every day. But I know where to find them if I need them. And I know that it can help in certain situations where you see odd protocol behavior. Likewise, I know that if I need to go design a network for someone I need to get a lot of information about business needs and build the best network I can with the constraints that I have. Things like budget and time are universal. But one of those constraints shouldn’t be lack of knowledge about protocols and good design. Those are two things that should be ingrained into anyone that wants to have a title of “senior” anything.

Transitioning Away From Legacy IT

One of the more exciting things I saw at Dell Technologies World this week was the announcement by VMware that they are supporting Microsoft Azure now in additional to AWS. It’s interesting because VMware is trying to provide a proven, stable migration path for companies that are wanting to move to the cloud but still retain their investments in VMware and legacy virtualization. But is offing legacy transition a good idea?

Hold On For One More Day

If I were to mention VLAN 1002-1005 to networking people, they would likely jump up and tell me that I was crazy. Because those VLANs are not valid on any Cisco switches save for the Nexus line. But why? What makes these forbidden? Unless you’re studying for your CCIE you probably just know these are bad and move on.

Turns out, they are a legacy transition mechanism from the IOS-SX days. 1002 and 1004 were designed to bridge FDDI-to-Ethernet, and 1003 and 1005 did the same for Token Ring. As Greg Ferro points out here, this code was tightly bound into IOS-SX and likely couldn’t be removed for fear of breaking the OS. The reservation continued forward in all IOS branches except NX-OS, which pulled them due to lack of support for those protocols.

So, we’ve got a legacy transition mechanism causing problems for users well past the “use by” date. Token Ring was on the way out at IBM in 2001. And yet, for some reason seventeen years later I still have to worry about bridging it? Or how about the rumors that Windows skipped from version 8 to version 10 because legacy code bases assumed Windows 9 meant Windows 95? Something 23 years old forced a major version change?

We keep putting legacy bridges in place all the time to help migrate things. Virtualization isn’t the only culprit here. We’ve found all manner of things that we can do to “trick” systems into working with modern hardware. We even made one idea into a button. But we never really solve the underlying issues. We just keep creating workarounds until we’re forced to move.

The Dream Is Still Alive

As it turns out, it’s expensive to refactor code bases and update legacy software to support new hardware. We’ve hit this problem time and time again with all manner of products. I can remember when Cisco CallManager wouldn’t install on a spare server I had with the same model number as a support machine just because the CPU was exactly 100MHz too fast. It’s frustrating to say the least.

But, we also have to realize that legacy transition mechanisms are not permanent fixes. It’s right there in the name. Transition. We put them in place because it’s cheaper in the short term while we investigate long term methods to make everything work correctly. But it’s still important to find those long term solutions. Maybe it’s a new application. Or a patch to make it work with new hardware. Sometimes, as Apple has done, it’s a warning that old software will stop working soon.

As developers, it’s important to realize that your app may last long past the date you want to stop supporting it. If you could still install Office 2000 on a desktop, I’m almost positive that someone would try it. We still have ways to install and use DOS software! If you want to ensure that your software is being used correctly and that you aren’t issuing patches for it after you’ve retired to a comfortable island with no Internet connection, make sure you find a way to ease transitions and offer new connection options to users.

For those of you that are still stuck in the morass of supporting legacy software or hardware, take a look at what you’re using it for and try to make hard choices where appropriate. If your organization is moving to the cloud, maybe now is the time to cut off your support for an application that’s too old to survive the migration. Or maybe it’s time to retire the Domain Controller That Time Forgot. But you have to do it before you’re forced to virtualize it and support it in perpetuity in AWS or Azure.


Tom’s Take

I’ll be the first to admit that legacy hardware and software are really popular. I worked with a company one time that still had an AS/400 admin on staff because of one application. It just happened to the one that paid people. At Interop ITX this year, the CIO for Detroit mentioned that they had to bring a developer out of retirement to make sure people kept getting paid. But legacy can’t be a part of the future. You either need to find a way to move what you have while you look for something better or you need to cut it off at the knees and find a way to make those functions work somewhere else. Because you don’t want to be the last company running AS/400 payroll over token ring bridged to a Cisco switch on VLAN 1003.

A Review of RSA Conference

So, I recently went to my first RSA Conference. It’s something I’ve had on my radar for a while but never had the opportunity to do. However, with Security Field Day coming up later this year I thought it was high time I went to see what everything was about. Here are some ideas that I came up with during my pilgrimage to the big security conference.

  • It’s Huge. Like, really big. I’ve never seen a bigger conference before. I haven’t gone to Oracle OpenWorld or Dreamforce, but the size of the RSA show floor alone dwarfs anything I’ve seen. Three whole areas, including one dedicated to emerging vendors. That’s big. Almost too big in fact.
  • I Still Hate Moscone. It’s official. No conference should ever use this place again. It’s been 4 years since I railed against it and every word still applies. Doubly so this year, as RSA was being held during construction! Seriously. At this point, Moscone must be paying people to hold a convention there. RSA is too big. I don’t care if it’s cheap to ferry people up from Silicon Valley. Stop doing this to yourself and tarnishing your brand. Just go to Vegas if you want to stay close.
  • Everyone Has A Specialization. Maybe you’re developing a cloud plugin for lateral evasion detection. Or an endpoint hardening solution for mobile. Maybe you’re into detection, intrusion, pen testing, or perhaps just training. Everyone had representation of some kind or another. Everyone solved a problem in a unique or different way. But everyone had their niche. More than any other place I’ve been I saw companies doing all manner of things that are so specialized that I worry they’re either going to go out of business before they can develop a customer base or they’re going to get bought to add a feature onto someone’s existing platform. May fortune favor you all.
  • You Can’t Be Secure Enough. Funny enough, when the RSA Mobile app gets hacked you know someone’s having a bad day. The truth is that so many people out there are going to be taking shots at you when you’re visibility is high that you need to be prepared to take them. You can’t just assume you’re obscure enough to avoid detection. If Shodan has taught us anything, it’s that obscurity isn’t good enough any longer. You need to be looking at your posture and your weak areas. You need to be prepared to pare everything down to the point where no one can get in easily. Defense in depth needs to be very deep today.

Tom’s Take

RSA is similar to Cisco Live and VMworld, but it’s also much different. It’s bigger. People are more reserved. There are a TON of expo passes everywhere. I think I counted maybe 50 full conference passes the whole week. It’s too big for Moscone for sure. When the press room is located a block and half away from the convention center proper, you might need to rethink your strategy for a conference. But, RSA is still full of exciting companies and new ideas. And that’s going to keep it rolling along for a while no matter how many times the mobile app gets “hacked”.

It’s Time For Security Apprenticeships

Breaking into an industry isn’t easy. When you look at the amount of material that is necessary to learn IT skills it can be daunting and overwhelming. Don’t let the for-profit trade school ads fool you. You can’t go from ditch digger to computer engineer in just a few months. It takes time and knowledge to get there.

However, there is one concept in non-technical job roles that feels very appropriate to how we do IT training, specifically for security. And that’s the apprenticeship.

Building For The Future

Apprenticeship is a standard for electricians and carpenters. It’s the way that we train new people to do the work of the existing workforce. It requires time and effort and a lot of training. But, it also fixes several problems with the current trend of IT certification:

  1. You Can’t Get a Job Without Experience – Far too often we see people getting rejected for jobs at the entry level because they have no experience. But how are they supposed to get the experience without doing the job? IT roles paradoxically require you to be cheap enough to hire for nothing but expect you to do the job on day one. Apprenticeships fix this by showing that you’re willing to do the job and get trained at the same time. That way you can be properly trained on the job.
  2. You Need To Be Certified – Coupled tightly with the first problem is the certification trend. Yes, electricians and plumbers are certified to do the work. After 5 years of training. You don’t get to take a test after 3 months and earn Journeyman status. You have to earn it through training and work. It also ensures that every journeyman out there has the same basic level of work experience and training. Compare that with people getting a CCNA right out of college with no experience on devices or a desktop administrator position having an MCSE-level requirement to discourage people from applying.
  3. You Can’t Find a Mentor – This is a huge issue for entry-level positions. The people you work with are often so overtasked with work that they don’t have time to speak to you, let alone show you the ropes. By defining the role of the apprentice and the journeyman trainer, you can ensure that the mentoring relationship exists and is reinforced constantly through the training period. You don’t get through an apprenticeship in a vacuum.
  4. You Can’t Get Paid To Learn – This is another huge issue with IT positions. Employers don’t want to pay to train you. They want you to use the vast powers of the Internet to look up hacks on Stack Overflow and paste them into production equipment rather than buying a book for you or letting you take a class. Some employers are better than others about highly skilled workers, but those employers are usually leveraging those skills to make more money, like in a VAR or MSP. Apprenticeships fix this problem by including a classroom component in the program. You spend time in a classroom every other week while you’re doing the work.

Securing the Future

The apprenticeship benefits above go double for security-related professions. Whereas there is a lot of material to train networking and storage admins, security is more of a game of cat-and-mouse. As new protections are developed, we must always figure out of the old protections are still useful. We need to train people how to learn. We need to bring them up to speed on what works and what doesn’t and why googling for an old article about packet filtering firewalls isn’t as relevant in 2018 as it was a decade ago.

More than that, we need to get the security practitioners to a place where they can teach effectively and mentor people. We need to get away from the idea of just rattling off a long list of recommended protections to people as the way to train them about security. We need to make sure we’re teaching fundamentals and proper procedures. And they only way that can happen is with someone that has the time and motivation to learn. Like an apprentice.

This would also have the added luxury of ensure the next generation of security practitioners learns the pitfalls before they make mistakes that compromise the integrity of the data they’re protecting. Making a mistake in security usually means someone finding out something they aren’t supposed to know. Or data getting stolen and used for illicit means. By using apprenticeships as a way for people to grow, you can quickly bring people up to speed in a safe environment without worrying that their decisions will lead to problems with no one to check their work.


Tom’s Take

I got lucky in my IT career. I worked with two mentors that spent the time to train me like an apprentice. They showed me the ropes and made sure I learned things the right way. And I became successful because of their leadership. Today, I don’t see that mentality. I see people covering their own jobs or worried that they’re training a replacement instead of a valued team member. I worry that we’re turning out a generation of security, networking, and storage professionals that won’t have the benefit of knowing what to do beyond Google searches and won’t know how to teach the coming generation how to deal with the problems they face. It’s not too late to start by taking an apprentice under your wing today.

On Old Configs and Automation

I used to work with a guy that would configure servers for us and always include an extra SCSI card in the order. When I asked him about it one day, he told me, “I left it out once and it delayed the project. So now I just put them on every order.” Even after I explained that we didn’t need it over and over again, he assured me one day we might.

Later, when I started configuring networking gear I would always set a telnet password for every VTY line going into the switch. One day, a junior network admin asked me why I configured all 15 instead of just the first 5 like they learn in the Cisco guides. I shrugged my shoulders and just said, “That’s how I’ve always done it.”

The Old Ways

There’s no more dangerous phrase than “That’s the way it’s always been.”

Time and time again we find ourselves falling back on the old rule of thumb or an old working configuration that we’ve made work for us. It’s comfortable for the human mind to work from a point of reference toward new things. We find ourselves doing it all the time. Whether it’s basing a new configuration on something we’ve used before or trying to understand a new technology by comparing it to something we’ve worked on in the past.

But how many times have those old configurations caused us grief? How many times have we been troubleshooting a problem only to find that we configured something that shouldn’t have been configured in the way that it was. Maybe it was an old method of doing hunt groups. Or perhaps it was a legacy QoS configuration that isn’t supported any more.

Part of our issue as networking professionals is that we want to concentrate on the important things. We want to implement solutions and ideas that work for our needs and not concentrate on the minutia of configuration. Sure, the idea of configuration a switch from bare metal to working config is awesome the first time you do it. But the fifteenth time you have to configure one in a row is less awesome. That’s why copy-and-paste configurations are so popular with people that just want to get the job done.

New Hotness

This idea of using old configurations for new things takes even more importance when you start replacing the boring old configuration methods with new automation and API-driven configuration models. Yes, APIs make it a lot easier to configure a switch programmatically. And automation tools like Puppet and Ansible make it much faster to bring a switch online from nothing to running in the minimum amount of time.

However, even with this faster configuration method, are we still using old, outdated configurations to solve problems? Sure, I don’t have to worry about configuring VLANs on the switch one at a time. But if my configuration is still referencing VLANs that are no longer in the system that makes it very difficult to keep the newer switches running optimally. And that’s just assuming the configuration is old and outdated. What if we’re still using deprecated commands?

APIs are great because they won’t support unsupported things. But if we don’t scrub the configuration now and then to remove these old lines and protocols we’ll quickly find ourselves in a world of trouble. Because those outdated and broken things will bring the API to a halt. Yes, the valid commands will still be entered correctly, but if those valid commands rely on something invalid to work properly you’re going find things broken very fast.

What makes this whole thing even more irritating is that most configurations need to be validated and approved before being implemented. Which makes the whole process even more difficult to manage. Part of the reason why old configs live for so long is that they need weeks or months of validation to be implemented effectively. When new platforms or configuration methods crop up it could delay new equipment installation. This sometimes leads to people installing new gear with “approved” configs that might not be the best fit in order to get that new equipment into service.


Tom’s Take

So, how do we fix all this? What’s the trick? Well, it’s really a combination of things. We need to make sure we audit configs regularly to keep the old stuff from continuing on past the expiration dates. We also need to continually resubmit new configurations to the approvals process. Just like disaster recovery documentation, configurations are living, breathing documents that should always be current and refreshed. The more current your configurations, the less likely you are to have old cruft keeping your systems running at subpar performance. And the less likely you are to have to worry about breaking new things like APIs and automation systems in the future.