About networkingnerd

Tom Hollingsworth, CCIE #29213, is a former network engineer and current organizer for Tech Field Day. Tom has been in the IT industry since 2002, and has been a nerd since he first drew breath.

Two Takes On ASIC Design

Making ASICs is a tough task. We learned this last year at Cisco Live Berlin from this conversation with Dave Zacks:

Cisco spent 6 years building the UADP ASIC that powers their next generation switches. They solved a lot of the issues with ASIC design and re-spins by creating some programmability in the development process.

Now, watch this video from Nick McKeown at Barefoot Networks:

Nick says many of the same things that Dave said in his video. But Nick and Barefoot took a totally different approach from Cisco. Instead of creating programmable elements in the ASIC design, then abstracted the entire language of function definition from the ASIC. By using P4 as the high level language and making the system compile the instruction sets down to run in the ASIC, they reduced the complexity, increased the speed, and managed to make the system flexible and capable of implementing new technologies even after the ASIC design is set in stone.

Oh, and they managed to do it in 3 years.

Sometimes, you have to think outside the box in order to come up with some new ideas. Even if that means you have to pull everything out of the box. By abstracting the language from the ASIC, Barefoot not only managed to find a way to increase performance but also to add feature sets to the switch quickly without huge engineering costs.

Some food for thought.

Culling The Community

exclusion

By now, you may have seen some bit of drama in the VMUG community around the apparent policy change that disqualified some VMUG leaders based on their employer. Eric Shanks (@Eric_Shanks) did a great job of covering it on his blog as did Matt Crape (@MattThatITGuy)with his post. While the VMUG situation has its own unique aspects, the question for me boils down to something simple: How do you remove people from an external community?

Babies And Bathwater

Removing unauthorized people from a community is nothing new under the sun. I was a Cisco Champion once upon a time. During the program’s second year I participated in briefings and events with the rest of the group, including my good friend Amy Arnold (@AmyEngineer). When the time came to reapply to the program for Year 3, I declined to apply again for my own reasons. Amy, however, was told that she couldn’t reapply. She and several other folks in the program were being disqualified for “reasons”. It actually took us a while to figure out why, and the answer still wasn’t 100% clear. To this day the best we can figure out is that there is some kind of conflict between anyone working with the public sector or government and the terms and conditions of the Champions program.

The lack of communication about the rules was the biggest issue by far with the whole transition. People don’t like being excluded. They especially don’t like being excluded from a group they were previously a member of. It takes time and careful explanation to help them understand why they are no longer able to be a part of a community. Hiding behind vague statements and pointing to rule sections doesn’t really help.

In the case of the VMUG issue above, the answer as to why the dismissed leaders were disqualified still isn’t clear. At least, it isn’t clear according to the official rules. There is still some debate as to the real reasoning behind everything, as the comments on Matt’s blog indicate. However, the community has unofficially settled on the reasoning being that those leaders were employed by someone that VMware, who is more-than-loosely affiliated with VMUG, has deemed a direct competitor.

I’m no stranger to watching companies go from friends to frenemies to competitors in the blink of an eye. VMware and Cisco. VMware and Scale Computing. Cisco and HP. All of these transitions took two aligned companies and put them on opposite sides of the firing line. And in a lot of cases, the shift in messaging was swift. Last week they were both great partners. The next week shifted to “We have always been at war with Eurasia.” Which didn’t bode well for people that were caught in the middle.

Correcting The Position

How do you correctly go about affecting changes in membership? How can you realistically make things work when a rule change suddenly excludes people? It’s not an easy path, but here are some helpful hints:

  • COMMUNICATION! – Above all else, it is absolutely critical to communicate at every step of the process. Don’t leave people guessing as to your reasoning. If you are contemplating a rule change, let everyone know. If you are looking to enforce a rule that was previously not enforced, warn everyone well in advance. Don’t let people come up with their own theories. Don’t make people write blogs asking for clarification on a situation.
  • If a person is being excluded because of a rule change, give the a bit of grace period to exit on their own terms. If that person is a community leader, they will need time to transition a new person into their role. If that person is a well-liked member of the community, give them a chance to say goodbye instead of being forced out. That grace period doesn’t need to be months long. Usually by the next official meeting or briefing time is enough. Giving someone the chance to say goodbye is much better than telling everyone they left. It provides closure and gives everyone a chance to discuss what the next steps will be.
  • If a rule change is in order that excludes members of the community, weigh it carefully. Ask yourself what you are gaining from it. Is it a legal reason? Does it need to be made to comply with some kind of regulation? Those are valid reasons and should be communicated with enough warning. People will understand. But if the reasoning behind your rule change is spite or retaliation for something, carefully consider your next steps. Realize that every rank-and-file member of the community has their own opinions and vision. Just because Evil CEO made your CEO mad doesn’t mean that his Local SE has the same feelings. And it absolutely doesn’t mean that Local SE is going to subvert your community for their own ends. These are the kinds of decisions that divide people at the expense of keeping your community free of “influences”.

It can’t be said enough that you need to talk to the community before you even begin debating action. There are no community organizations that blindly follow orders from on high. These are places where thinking people interact and share. And if they are suddenly told how things are going to be without any discussion or debate, you can better believe they are going to try and get to the bottom of it. Whether you want them to or not.


Tom’s Take

Kicking people out of something is never easy. Tech Field Day has rules about delegates being employed by presenting vendors. More than once I’ve had conversations with people about being disqualified from being a delegate. Most of them understand why that’s the case beforehand because our policy is straightforward. But if it’s ever changed, you can better believe that we’re going to let everyone know well in advance.

Communities run on communication. Discussion, debate, and ultimately acceptance are all driven by knowing what’s happening at all times. If you make rules under the cloak of secrecy for reasons which aren’t readily apparent, you risk alienating more than just the people you’re looking to exclude.

Blogging By The Refrigerator’s Light

Blogging isn’t starting off to a good 2017 so far. Ev Williams announced that Medium is cutting back and trying to find new ways to engage readers. The platform of blogging is scaling back as clickbait headlines and other new forms of media capture the collective attention for the next six seconds. How does that all relate to the humble tech blogger?

Mindshare, Not Eyeshare

One of the reasons why things have gotten so crazy is the drive for page views. Clickbait headlines serve the singular purpose of getting someone to click on an article to register a page view. Ever clicked on some Top Ten article only to find that it’s actually a series of 10 pages in a slideshow format? Page views. I’ve even gone so far as to see an article of top 7 somethings broken down into 33(!) pages, each with 19 ads and about 14 words.

Writers competing for eyeballs are always going to lose in the end. Because the attention span of the average human doesn’t dally long enough to make a difference. Think of yourself in a crowded room. Your eyes dart back and forth and all around trying to find something in the crowd. You may not even know what you’re looking for. But you’ll know it when you see it. Your attention wanders as you scan through the crowd.

Blogging, on the other hand, is like finding a good conversation in the crowd. It engages the mind. It causes deeper thinking and engagement that leads to lasting results. The best blog posts don’t have thousands of views in the first week followed by little to nothing for the rest of eternity. They have active commenters. They have response pieces. They have page views and search results that get traffic years after publication.

The 3am Ah Ha Moments

Good blogs shouldn’t just be about “going viral”. Good blogs should have something called Fridge Brilliance. Simply put, the best blogs hit you out of the blue a day after you read it standing in front of your fridge door. BANG. Now you get it! You run off to see how it applies to what you’re doing or even to give your perspective on things.

The mark of a truly successful blog is creating something that lasts and is memorable in the minds of readers. Even if all you’re really known for is “that one post” or a series of great articles, you’ve made an impression. And, as I’ve said before, you can never tell which post is going to hit it big. So the key is to keep writing what you write and making sure you’re engaging your audience at a deeper level than their corneas.

That’s not to say that you can’t have fun with blog posts now and then or post silly things here and there. But if you really want to be known as an authoritative source of content, you have to stay consistent. One of the things that Dave Henry (@DaveMHenry) saw in his 2016 wrap-up was that his most viewed posts were all about product announcements. Those tend to get lots of headlines, but for an independent blog it’s just as much about the perspective the writer lends as it is for the news itself. That’s how you can continue to engage people beyond the eyeball and into the brain.


Tom’s Take

I’ve noticed that people still like to write. They want to share thoughts. But they pick the wrong platforms. They want eyeballs instead of minds. They don’t want deep thoughts. They just want an audience. That’s the wrong way to look at it. You want engagement. You want disagreement and argument and 4,000 word response posts about why you’re completely wrong. Because that’s how you know you’ve hooked the reader. You’re a splinter in their mind that won’t go away. That’s the real draw. Keep your page views. I’d rather have memories and fridge brilliance instead.

Bringing 2017 To Everyone

calendar

It’s time once again for my traditional New Year’s Day navel gazing. As per tradition with my blog, I’m not going to make prognostications about networking or IT in general. Either I’m going to wind up totally wrong or be totally right and no one will care. I rather enjoy the ride as we go along, so trying to guess what happens is kind of pointless.

Instead, I’m going to look at what I want to accomplish in the coming year. It gives me a chance to analyze what I’m doing and what I want to be working on. And it’s a whole lot easier than predicting that SDN is going to take everyone’s job or OpenFlow being dead again.

Write Like the Wind

My biggest goal for 2016 was to write more. And that I did. I worked in writing any time I could. I wrote about ONUG, SD-WAN, and other fun topics. I even wrote a small book! Finding time to work all the extra typing in to my Bruce Wayne job at Tech Field Day was a bit challenging here and there. And more than once I was publishing a blog post at the deadline. But all that writing did help me talk about new subjects in the industry and develop great ideas at the same time.

I also encouraged more people to write. I wanted to get people putting their thoughts down in a form that didn’t require listening or watching video. Writing is still very important and I think it’s a skill that more people should develop. My list of blogs to read every day grew in 2016 and I was very happy to see it. I hope that it continues well into 2017 as well.

King Of The Hill

2017 is going to be an exciting year for me and Tech Field Day. I ran Networking Field Day 12 as the host of the event for the first time. In the coming year, Stephen and I are going to focus on our topics areas even deeper. For me, that means immersing myself in networking and wireless technologies more than ever before. I’m going to be learning as much as I can about all the new things going on. It’s a part of the role of being the host and organizer for both Networking Field Day and Mobility Field Day coming up this year.

I’m also going to be visiting lots of other conferences. Cisco Live, Interop, and even Open Networking Summit are on my list this year. We’re going to be working closely with those shows to put on even more great Tech Field Day content. I love hearing the excitement from my friends in the industry when they learn that Tech Field Day is going to be present at a show like Cisco Live. It means that we’re reaching a great audience and giving them something that they are looking for.

We’re also going to be looking at new ideas and new things to do with our growing media presence with Gestalt IT. There should be some interesting things there on the horizon as we embrace the new way that media is used to communicate with readers and fans alike. Stay tuned there for all the excitement we’ll be bringing your way in 2017!


Tom’s Take

Analyzing a year’s worth of work helps one see progress and build toward even more goals in the coming year. I’m going to keep moving forward with the projects that excite me and challenge me to be a better representative for the networking community. Along the way I hope to learn more about what makes our technology exciting and useful. And share than knowledge with everyone I know in the best way I can. Thanks for being here with me. I hope 2017 is a great year for you as well!

Automating Your Job Away Isn’t Easy

programming

One of the most common complaints about SDN that comes from entry-level networking folks is that SDN is going to take their job away. People fear what SDN represents because it has the ability to replace their everyday tasks and put them out of a job. While this is nowhere close to reality, it’s a common enough argument that I hear it very often during Q&A sessions. How is it that SDN has the ability to ruin so many jobs? And how is it that we just now have found a way to do this?

Measure Twice

One of the biggest reasons that the automation portion of SDN has become so effective in today’s IT environment is that we can finally measure what it is that networks are supposed to be doing and how best to configure them. Think about the work that was done in the past to configure and troubleshoot networks. It’s often a very difficult task that involves a lot of intuition and guesswork. If you tried to explain to someone the best way to do things, you’d likely find yourself at a loss for words.

However, we’ve had boring, predictable standards for many years. Instead of cobbling together half-built networks and integrating them in the most obscene ways possible, we’ve instead worked toward planning and architecting things properly so they are built correctly from the ground up. No more guess work. No more last minute decisions that come back to haunt us years down the road. Those kinds of things are the basic building blocks for automation.

When something is built along the lines of predictable rules with proper adherence to standards, it’s something that can be understood by a non-human. Going all the way back to Basic Computing 101, the inputs of a system determine the outputs. More simply, Garbage In, Garbage Out. If your network configuration looks like a messy pile of barely operational commands it will only really work when a human can understand what’s going on. Machines don’t guess. They do exactly what they are told to do. Which means that they tend to break when the decisions aren’t clear.

Cut Once

When a system, script, or program can read inputs and make procedural decisions on those inputs, you can make some very powerful things happen. Provided, that is, that your chosen language is powerful enough to do those things. I’m reminded of a problem I worked on fifteen years ago during my internship at IBM. I needed to change the MTU size for a network adapter in the Windows 2000 registry. My programming language of choice wasn’t powerful enough for me to say something like, “Read these values into an array and change the last 2 or 3 to the following MTU”. So instead, I built a nested if statement that was about 15 levels deep to ensure I caught every possible permutation of the adapter binding order. It was messy. It was ugly. And it worked. But there was no way it would scale.

The most important thing to realize about SDN and automation is that we’ve moved past simply understanding basic values. We’ve finally graduated to a place where programs can make complex decisions based on a number of inputs. We’ve graduated from simple if-then-else constructs and up to a point where programs can take a number of inputs and make decisions based on them. Sure, in many cases the inputs are simple little things like tags or labels. But what we’re gaining is the ability to process more and more of those labels. We can create provisioning scripts that ensure that prod never talks to dev. We can automate turn-up of a new switch with multiple VLANs on different ports through the use of labels and object classes. We can even extrapolate this to a policy-based network language that we can use to build a task once and execute it over and over again on different hardware because we’re doing higher level processing instead of being hamstrung by specific device syntax.

Automation is going to cost some people their jobs. That’s a given. Just like every other manufacturing position, the menial tasks of assembling simple pieces or performing repetitive tasks can easily be accomplished by a machine or software construct. But writing those programs and working on those machines is a new kind of job in and of itself. A humorous anecdote from the auto industry says that the introduction of robots onto assembly lines caused many workers to complain and threaten to walk off the job. However, one worker picked up the manual for the robot and realized that he could easily start working on the it instead of the assembly line.


Tom’s Take

Automation isn’t a magic bullet to fix all your problems. It only works if things are ordered and structured in such a way that you can predictably repeat tasks over and over. And it’s not going to stop with one script or process. You need to continue to build, change, and extend your environment. Which means that your job of programming switches should now be looked at in light of building the programs that program switches. Does it mean that you need to forget the basics of networking? No, but it does mean that they way in which you think about them will change.

Is It Really Always The Network?

Keep Calm and Blame the Network

Image from Thomas LaRock

I had a great time over the last month writing a series of posts with my friend John Herbert (@MrTugs) over on the SolarWinds Geek Speak Blog. You can find the first post here. John and I explored the idea that people are always blaming the network for a variety of issues that are often completely unrelated to the actual operation of the network. It was fun writing some narrative prose for once, and the feedback we got was actually pretty awesome. But I wanted to take some time to explain the rationale behind my madness. Why is it that we are always blaming the network?!?

Visibility Is Vital

Think about all the times you’ve been working on an application and things start slowing down. What’s the first thing you think of? If it’s a standalone app, it’s probably some kind of processing lag or memory issues. But if that app connects to any other thing, whether it be a local network or a remote network via the Internet, the first culprit is the connection between systems.

It’s not a large logical leap to make. We have to start by assuming the the people that made the application knew what they were doing. If hundreds of other people aren’t having this problem, it must not be with the application, right? We’ve already started eliminating the application as the source of the issues even before we start figuring out what went wrong.

People will blame the most visible part of the system for issues. If that’s a standalone system sealed off from the rest of the world, it obviously must be the application. However, we don’t usually build these kinds of walled-off systems any longer. Almost every application in existence today requires a network connection of some kind. Whether it’s to get updates or to interact with other data or people, the application needs to talk to someone or maybe even everyone.

I’ve talked before about the need to make the network more of a “utility”. Part of the reason for this is that it lowers the visibility of the network to the rest of the IT organization. Lower visibility means fewer issues being incorrectly blamed on the network. It also means that the network is going to be doing more to move packets and less to fix broken application issues.

Blame What You Know

If your network isn’t stable to begin with, it will soon become the source of all issues in IT even if the network has nothing to do with the app. That’s because people tend to identify problem sources based on their own experience. If you are unsure of that, work on a consumer system helpdesk sometime and try and keep track of the number of calls that you get that were caused by “viruses” even if there’s no indication that this is a virus related issue. It’s staggering.

The same thing happens in networking and other enterprise IT. People only start troubleshooting problems from areas of known expertise. This usually breaks down by people shouting out solutions like, “I saw this once so it must be that! I mean, the symptoms aren’t similar and the output is totally different, but it must be that because I know how to fix it!”

People get uncomfortable when they are faced with troubleshooting something unknown to them. That’s why they fall back on familiar things. And if they constantly hear how the network is the source of all issues, guess what the first thing to get blamed is going to be?

Network admins and engineers have to fight a constant battle to disprove the network as the source of issues. And for every win they get it can come crashing down when the network is actually the problem. Validating the fears of the users is the fastest way to be seen as the issue every time.

Mean Time To Innocence

As John and I wrote the pieces for SolarWinds, what we wanted to show is that a variety of issues can look like network things up front but have vastly different causes behind the scenes. What I felt was very important for the piece was the distinguish the the main character, Amanda, go beyond the infamous Mean Time To Innocence (MTTI) metric. In networking, we all too often find ourselves going so far as to prove that it’s not the network and then leave it there. As soon as we’re innocent, it’s done.

Cross-function teams and other DevOps organizations don’t believe in that kind of boundary. Lines between silos blur or are totally absent. That means that instead of giving up once you prove it’s not your problem, you need to work toward fixing what’s at issue. Fix the problem, not the blame. If you concentrate on fixing the problems, it will soon become noticeable that networking team may not always be the problem. Even if the network is at fault, the network team will work to fix it and any other issues that you see.


Tom’s Take

I love the feedback that John and I have gotten so far on the series we wrote. Some said it feels like a situation they’ve been in before. Others have said that they applaud the way things were handled. I know that the narrative allows us to bypass some of the unsavory things that often happen, like argument and political posturing inside an organization to make other department heads look bad when a problem strikes. But what we really wanted to show is that the network is usually the first to get blamed and the last to keep its innocence in a situation like this.

We wanted to show that it’s not always the network. And the best way for you to prove that in your own organization is to make sure the network isn’t just innocent, but helpful in solving as many problems as possible.

Is The Rise Of SD-WAN Thanks To Ethernet?

Ethernet

SD-WAN has exploded in the market. Everywhere I turn, I see companies touting their new strategy for reducing WAN complexity, encrypting data in flight, and even doing analytics on traffic to help build QoS policies and traffic shaping for critical links. The first demo I ever watched for SDN was a WAN routing demo that chose best paths based on cost and time-of-day. It was simple then, but that kind of thinking has exploded in the last 5 years. And it’s all thanks to our lovable old friend, Ethernet.

Those Old Serials

When I started in networking, my knowledge was pretty limited to switches and other layer 2 devices. I plugged in the cables, and the things all worked. As I expanded up the OSI model, I started understanding how routers worked. I knew about moving packets between different layer 3 areas and how they controlled broadcast storms. This was also around the time when layer 3 switching was becoming a big thing in the campus. How was I supposed to figure out the difference between when I should be using a big router with 2-3 interfaces versus a switch that had lots of interfaces and could route just as well?

The key for me was media types. Layer 3 switching worked very well as long as you were only connecting Ethernet cables to the device. Switches were purpose built for UTP cable connectivity. That works really well for campus networks with Cat 5/5e/6 cabling. Switched Virtual Interfaces (SVIs) can handle a large amount of the routing traffic.

For WAN connectivity, routers were a must. Because only routers were modular in a way that accepted cards for different media types. When I started my journey on WAN connectivity, I was setting up T1 lines. Sometimes they had an old-fashioned serial connector like this:

s-l300

Those connected to external CSU/DSU modules. Those were a pain to configure and had multiple points of failure. Eventually, we moved up in the world to integrated CSU/DSU modules that looked like this:

ehwic-2-ports-t-1-e-1

Those are really awesome because all the configuration is done on the interface. They also take regular UTP cables instead of those crazy V.35 monsters.

cisco_v35_old_large

But those UTP cables weren’t Ethernet. Those were still designed to be used as serial connections.

It wasn’t until the rise of MPLS circuits and Transparent LAN services that Ethernet became the dominant force in WAN connectivity. I can still remember turning up my first managed circuit and thinking, “You mean I can use both FastEthernet interfaces? No cards? Wow!”.

Today, Ethernet dominates the landscape of connectivity. Serial WAN interfaces are relegated to backwater areas where you can’t get “real WAN connectivity”. And in most of those cases, the desire to use an old, slow serial circuit can be superseded by a 4G/LTE USB modem that can be purchased from almost any carrier. It would appear that serial has joined the same Heap of History as token ring, ARCnet, and other venerable connectivity options.

Rise, Ethernet

The ubiquity of Ethernet is a huge boon to SD-WAN vendors. They no longer have to create custom connectivity options for their appliances. They can provide 3-4 Ethernet interfaces and 2-3 USB slots and cover a wide range of options. This also allows them to simplify their board designs. No more modular chassis. No crazy requirements for WIC slots, NM slots, or any other crazy terminology that Cisco WAN engineers are all too familiar with.

Ethernet makes sense for SD-WAN vendors because they aren’t concerned with media types. All their intelligence resides in the software running on the box. They’d rather focus on creating automatic certificate-based IPsec VPNs than figuring out the clock rate on a T1 line. Hardware is not their end goal. It is much easier to order a reference board from Intel and plug it into a box than trying to configure a serial connector and make a custom integration.

Even SD-WAN vendors that are chasing after the service provider market are benefitting from Ethernet ubiquity. Service providers may still run serial connections in their networks, but management of those interfaces at the customer side is a huge pain. They require specialized technical abilities. It’s expensive to manage and difficult to troubleshoot remotely. Putting Ethernet handoffs at the CPE side makes life much easier. In addition, making those handoffs Ethernet makes it much easier to offer in-line service appliances, like those of SD-WAN vendors. It’s a good choice all around.

Serial connectivity isn’t going away any time soon. It fills an important purpose for high-speed connectivity where fiber isn’t an option. It’s also still a huge part of the install base for circuits, especially in rural areas or places where new WAN circuits aren’t easily run. Traditional routers with modular interfaces are still going to service a large number of customers. But Ethernet connectivity is quickly growing to levels where it will eclipse these legacy serial circuits soon. And the advantage for SD-WAN vendors can only grow with it.


Tom’s Take

Ethernet isn’t the only reason SD-WAN has succeeded. Ease of use, huge feature set, and flexibility are the real reasons when SD-WAN has moved past the concept stage and into deployment. WAN optimization now has SD-WAN components. Service providers are looking to offer it as a value added service. SD-WAN has won out on the merits of the technology. But the underlying hardware and connectivity was radically simplified in the last 5-7 years to allow SD-WAN architects and designers to focus on the software side of things instead of the difficulties of building complicated serial interfaces. SD-WAN may not owe it’s entire existence to Ethernet, but it got a huge push in the right direction for sure.