Time To Get Back To Basics?

I’ve had some fascinating networking discussions over the past couple of weeks at Dell Technologies World, Interop, and the spring ONUG meeting. But two of them have hit on some things that I think need to be addressed in the industry. Both Russ White and Ignas Bagdonas of the IETF have come to me and talked about how they feel networking professionals have lost sight of the basics.

How Stuff Works

If you walk up to any network engineer and ask them to explain how TCP works, you will probably get a variety of answers. Some will try to explain it to you in basic terms to avoid getting too in depth. Others will swamp you with a technical discussion that would make the protocol inventors proud. But still others will just shrug their shoulders and admit they don’t really understand the protocol.

It’s a common problem when a technology gets to the point of being mature and ubiquitous. One of my favorite examples is the fuel system on an internal combustion engine. On older cars or small engines, the carburetor is responsible for creating the correct fuel and air mixture that is used to power the cylinders. Getting that mix right is half science and half black magic. For the people that know how to do it properly it’s an easy thing that they can do to drive the maxim performance from an engine. For those that have tried it and failed, it’s something best left alone to run at defaults.

The modern engine uses fuel injection. It’s a black box. It can be reprogrammed or tinkered with but it’s something that is tuned in different ways from the traditional carburetor. It’s not something that’s designed to be played around with unless you really know what you’re doing. The technology has reached the point where it’s ubiquitous and easy to use, but very difficult to repair without a specialized skill set.

Most regular car drivers will look under the hood and quickly realize they know nothing about what’s going on. Some technical folks will be able to figure out what certain parts do by observing their behavior. But if you ask those same people how a fuel injection system or carburetor works they’ll probably give you a blank stare.

That’s the issue we find in modern networking. We’ve been creating VLANs and BGP route maps for years. Some people have spent their entire careers tuning multicast or optimizing layer 2 interconnects. But if you corner them and ask them how the protocol works or how best to design an architecture that takes advantage of their life’s work they can’t tell you aside from referencing some old blog post or some vendor’s validated design on their hardware.

Russ and Ignas each touch on something important. In the good old days before there were a hundred certification guides and a thousand SRNDs people had to do real work to find the best solution for a problem. They had to put pencil to paper and sort out the mess before they could make something work. That’s where the engineering side of the network comes from.

Today, it’s more “plug and play”. You drop in pieces of a solution and they should work together. In practice, that usually means all the pieces have to be from the same vendor or from approved partner sources. And anything that goes awry will need a team of experts and many, many consulting hours to figure out.

Imagine if we could only install networks without understanding how they work. Could you see a world where everything we install from a networking perspective is a black box like a fuel injector? That’s the case to a certain degree with cloud networking today. We don’t see what’s going on under the surface. We can only see what the interface exposes to us. That’s fine as long as the applications we are using support the things we’re trying to do with them. But when it comes to being able to fix the network at the level we’re used to seeing it could be difficult if not downright impossible.

Learning The Ropes

But, moreover, are the networking professionals that are configuring these networks even capable of making those changes? Does anyone other than Narbik really understand how EIGRP works? Facebook seems to think that lightweight messaging packets for routing protocols are outdated. So they used ZeroMQ without understanding why that’s a bad idea for slow speed links. They may understand how a routing protocol works in theory, but they don’t completely understand how it’s supposed to work in extreme cases.

Can we teach people the basics and understanding of protocols and design that they need in order to make proper designs outside of vendor reference documents? It’s a tall order for sure. Most blog posts are designed to talk about features or solve problems. Most literature from creators is designed to make their new widget work correctly. Very little documentation exists about integration or design. And a good portion of what does exist is a bit outmoded and needs to be spruced up.

We, as the stewards of networking, need to help this process along. We need to spend more time talking about design and theory. We need to dissect protocols and help people understand how to use the tools they have rather than hoping someone will build the best mousetrap ever to solve each piece of a complicated puzzle. We need to teach people to be thinkers and problem solvers. And, yes, that does mean a bit less complaining about things like vendor code quality and VAR behavior.

Why? Because those people are empowered by a lack of knowledge. Customers aren’t idiots. They have business reasons for the things they do. Our technology needs to support their business needs. Yes, that means we need to think critically about what we’re doing. And yes, they may mean eating our words now and then to avoid a showdown about something that’s ultimately unimportant in the long run.

If we increase the amount of knowledge about the important topics like design and protocols it should make the overall level of understanding go up for everyone. That means better designs. More integrated technology. Less reliance on being force-fed the bare minimum information necessary to make things work. And that means things will run faster and much more smoothly. Which is a win for everyone.

Tom’s Take

I’ll be the first to admit that I don’t know the dirty mechanics of Frame Relay switching or how to tune OSPF Hello timers for non-standard link types. It’s a skill I don’t use every day. But I know where to find them if I need them. And I know that it can help in certain situations where you see odd protocol behavior. Likewise, I know that if I need to go design a network for someone I need to get a lot of information about business needs and build the best network I can with the constraints that I have. Things like budget and time are universal. But one of those constraints shouldn’t be lack of knowledge about protocols and good design. Those are two things that should be ingrained into anyone that wants to have a title of “senior” anything.

8 thoughts on “Time To Get Back To Basics?

  1. Tom, I completely agree. The last few months I have been having very similar thoughts. I’m concerned about what will happen to our profession if the majority of us simply become conduits for vendor-“validated” designs. This is part of the reason that I’ve started blogging again. I think we have a long way to go as a profession but fortunately our world is built on discrete protocols that can be understood one at a time. Hopefully that means we can start understanding the fundamentals and see an almost immediate benefit.

  2. This is a real risk. All the different “SDN” solutions with a single “Do it” buttons make life very simple, but the people that use those solutions early in their careers don’t have any incentive to learn what really happens under the hood.
    I see a real risk that as time goes by it would be harder and harder to find people that can actually design the “under the hood” part or be able to troubleshoot when it breaks.

  3. I definitely see more and more people in IT in general that have no clue how things actually work. If you don’t understand the why and how, it’s nearly impossible to troubleshoot when it doesn’t do what it is supposed to do. The adage of “fixing the network one misconfigured server at a time” is all too true. I spend more time digging into packet captures to troubleshoot application issues than I do fixing actual network issues. In fact, I’m usually surprised when it really is the network. Unfortunately without knowing how TCP and some common protocols like SIP, HTTP and DHCP work, I wouldn’t be able to do this troubleshooting. Being in healthcare I’ve even learned DICOM and HL7 because the application specialists are lost if it doesn’t work in the GUI.

    Like Arie, I worry that eventually we will have no one left to troubleshoot the underlays because everyone became engineers strictly in overlay GUIs. Someone has to know what’s behind the curtain.

  4. Some disparate thoughts:
    I’m like which type of networking is not software defined? We have always used software to move bits from one place to the other.
    Why is Network engineers learning how to use programming languages a thing? Aerospace engineers, mechanical engineers, civil engineers have been using those tools forever.
    The business case above all else-what problem are you trying to solve? What business capability are you trying to engineer? That should be what determines what tech/method employed. The business case should define every device, every configuration.
    I believe engineering before metaengineering-learn, relearn and relearn the basics constantly and no “new” trend/product will disturb your peace.
    It is my primal belief that network engineering needs to be classed and taught the way the other engineering disciplines are. Certifications should become like the ACCA is to an accountant or passing the boards is for a doctor.

  5. Pingback: Worth Reading: Back to Basics? – rule 11 reader

  6. Unfortunately I am in agreement that there are more and more clueless people in IT. But I am not sure what could be done to change this

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s