Q And A Should Include The E

The IT world is cyclical for sure. I’ve seen trends and topics repeating themselves over and over again in my relatively short time here. I find it interesting that we keep solving similar problems over and over again. I also find it fascinating that this particular issue leads to the reason why blogs are so important.

Any Questions?

Questions abound in IT. It’s the nature of the industry. However, it’s not just new questions that we create when technology leaps past us. We keep asking the same questions over and over again. This is the field of study that created the FAQ, remember?

In recent memory, I find the same questions being asked over and over again:

  • What is SDN?
  • How can SDN help me?
  • What makes this different from what we’ve done before?

You’ve probably asked those very same questions. Perhaps you found the answers you were looking for. Perhaps you’re still trying to figure it out. The problem is that those questions are still being asked. The industry should have evolved to the point where the simple questions have been answered with simple answers. Complex questions, or those questions that need more in-depth discussion, should be treated as such. Yes, the question of what SDN really is would take more than a cursory paragraph on a blog, but we should be able to at least answer it with enough specificity to make the user not feel like they been slighted.

Questions will never stop coming in IT. But how should we handle them?

Any Answers?

Questions may abound in IT, but the answers drive IT. People make a career out of being the person with the answers. It’s in all the marketing jargon. It’s why we create blogs. Even though most of my writing in the last year has been focused on industry trends or non-technical focused posts, the top three posts on my blog are still answers to simple questions:

  1. When Is A Trunk Not A Trunk?
  2. Switchport Voice VLAN – What Does It Do?
  3. Why Is My SFP Not Working?

These posts are far and away the most popular. I even saw this a few months ago and it made me smile:

This would make it seem like people are in need of answers. Any blogger can look at the incoming search terms for their blog and see all the things that brought readers to them. People want answers and they will keep looking until they find them. But why?

Explain It

I never understood why people kept searching for answers until I thought about satisfaction. I think Randall Munroe summed up the satisfaction (or lack thereof) angle here:

Who are you, DenverCoder9?!? (Thanks XKCD)

Who are you, DenverCoder9?!? (Thanks XKCD)

People can find answers easily. But they won’t stop looking until they are satisfied with the answer. It’s easy to find people saying things like “That’s not supported” or “RTFM” when you’re looking for an answer to a particularly difficult problem. And if you’ve ever called a tech support line, you know how unfulfilling the unsupported answer can feel.

That’s when explanation comes into play for me. First, an admission: I’m a chronic explainer. If you’ve ever met me and had a conversation with me for more than three minutes, you know I explain things. I talk about comic books and movies and technical topics in more depth than I should. That’s because I want things explained to me. Explaining how OSPF area calculations are done is as important as explaining how Captain America ended up wielding Mjolnir.

Think about the following answers:

This is unsupported.


This is unsupported on that platform because the CPU doesn’t have enough horsepower to process the packets in real time. We tried cutting down on the processing time but it just overwhelmed the unit no matter how much we tried. So rather than dealing with poor performance, we marked it as unsupported.

Both answers are technically correct. But the second is much more satisfying because the explanation is there instead of just the distilled answer.

The IT world needs more explanation. We need to know why things work the way they do instead of just getting a response of a few words. The explanation has the keys to understanding the answer to the question in its totality. It prevents us from asking the same questions over and over again. It leaves us fulfilled and ready to seek out the next question that needs to be asked.

How Do You Spell That?

I spent a bit of my career on the phone doing support for a national computer vendor. In addition to the difficulties of walking people through opening the case and diagnosing motherboard issues, I found myself needing to overcome language barriers. While I only have a hint of an accent (or so I’ve been told), spelling out acronyms was a challenge. That’s where the phonetic alphabet comes into play

By now, almost everyone uses the NATO phonetic alphabet. It’s the most recognized in the world. The US joint Army/Navy version varies a bit but does have a lot of similarities. However, when I first started out using the NATO version quite a few callers didn’t know what Lima was or giggled when I said Tango.

I decided that some people have much more familiarity with first names. This was borne out when I kept using Mary for “M” instead of Mike. People immediately knew it. Same for Victor, Peter, and so on. So I cobbled together my own Name Phonetic Alphabet.

A – Adam
B – Barbara
C – Charlie
D – David
E – Edward
F – Frank
G – George
H – Harold
I – Irwin
J – John
K – Kevin
L – Larry
M – Mary
N – Nancy
O – Oliver
P – Peter
Q – Quincy (or queen)
R – Roger
S – Sam
T – Tom (my favorite)
U – Umbrella
V – Victor
W – William
X – X-Ray
Y – Yellow
Z – Zebra

Finding a name for Y and Z was pretty difficult, but everyone knows Yellow and Zebra. I was tempted to use Zander, but the more popular version of that name from Buffy the Vampire Slayer was spelled Xander. No sense confusing folks. As for X, if you don’t know X from the sound we need to have a chat.

Was it a duplication of effort? Certainly. But it works universally with everyone I’ve ever talked to, including children. It makes “Roger Adam Irwin David” easy to get across to people without trying to remember Romeo and India.

The key to communication with others is to find something that works for you.  If you can easily convey your information to someone else, the shortcuts you take don’t matter.  If first names work best, use them.  If drawing pictures works better, use those.  In the end, getting the point across is the goal.

SDN Use Case: Content Filtering

K-12 schools face unique challenges with their IT infrastructure.  Their user base needs access to a large amount of information while at the same time facing restrictions.  While it does sound like some corporate network policies, the restrictions in the education environment are legal in nature.  Schools must find new ways to provide the assurance of restricting content without destroying their network in the process.  Which lead me to ask: Can SDN Help?

Online Protection

The government E-Rate program gives schools money each year under Priority 1 funding for Internet access.  Indeed, the whole point of the E-Rate program is to get schools connected to the Internet.  But we all know the Internet comes with a bevy of distractions. Many of those distractions are graphic in nature and must be eliminated in a school.  Because it’s the law.

The Children’s Internet Protection Act (CIPA) mandates that schools and libraries receiving E-Rate funding for high speed broadband Internet connections must filter those connections to remove questionable content.  Otherwise they risk losing funding for all E-Rate services.  That makes content filters very popular devices in schools, even if they aren’t funded by E-Rate (which they aren’t).

Content filters also cause network design issues.  In the old days, we had to put the content filter servers on a hub along with the outbound Internet router in order to insure they could see all the traffic and block the bad bits.  That became increasing difficult as network switch speeds increased.  Forcing hundreds of megabits through a 10Mbit hub was counterproductive.  Moving to switchport mirroring did alleviate the speed issues, but still caused network design problems.  Now, content filters can run on firewalls and bastion host devices or are enabled via proxy settings in the cloud.  But we all know that running too many services on a firewall causes performance issues.  Or leads to buying a larger firewall than needed.

Another issue that has crept up as of late is the use of Virtual Private Networking (VPN) as a way to defeat the content filter.  Setting up an SSL VPN to an outside, non-filtered device is pretty easy for a knowledgeable person.  And if that fails, there are plenty of services out there dedicated to defeating content filtering.  While the aim of these service is noble, such as bypassing the Great Firewall of China or the mandated Internet filtering in the UK, they can also be used to bypass the CIPA-mandated filtering in schools as well.  It’s a high-tech game of cat-and-mouse.  Blocking access to one VPN only for three more to pop up to replace it.

Software Defined Protection

So how can SDN help?  Service chaining allows traffic to be directed to a given device or virtual appliance before being passed on through the network.  This great presentation from Networking Field Day 7 presenter Tail-f Networks shows how service chaining can force traffic through security devices like IDS/IPS and through content filters as well.  There is no need to add hubs or mirrored switch ports in your network.  There is also no need to configure traffic to transit the same outbound router or firewall, thereby creating a single point of failure.  Thanks to the magic of SDN, the packets go to the filter automatically.  That’s because they don’t really have a choice.

It also works well for providers wanting to offer filtering as a service to schools.  This allows a provider to configure the edge network to force traffic to a large central content filter cluster and ensure delivery.  It also allows the service provider network to operate without impact to non-filtered customers.  That’s very useful even in ISPs dedicated to education institutions, as the filter provisions for K-12 schools don’t apply to higher education facilities, like colleges and universities.  Service chaining would allow the college to stay free and clear while the high schools are cleansed of inappropriate content.

The VPN issue is a thorny one for sure.  How do you classify traffic that is trying to hide from you?  Even services like Netflix are having trouble blocking VPN usage and they stand to lose millions if they can’t.  How can SDN help in this situation? We could build policies to drop traffic headed for known VPN endpoints.  That should take care of the services that make it easy to configure and serve as a proxy point.  But what about those tech-savvy kids that setup SSL VPNs back home?

Luckily, SDN can help there as well.  Many unified threat management appliances offer the ability to intercept SSL conversations.  This is an outgrowth of sites like Facebook defaulting to SSL to increase security.  SSL intercept essentially acts as a man-in-the-middle attack.  The firewall decrypts the SSL conversation, scans the packets, and re-encrypts it using a different certificate.  When the packets come back in, the process is reversed.  This SSL intercept capability would allow those SSL VPN packets to be dropped when detected.  The SDN component ensures that HTTPS traffic is always redirected to a device that and do SSL intercept, rather than taking a path through the network that might lead to a different exit point.

Tom’s Take

Content filtering isn’t fun.  I’ve always said that I don’t envy the jobs of people that have to wade through the unsavory parts of the Internet to categorize bits as appropriate or not.  It’s also a pain for network engineers that need to keep redesigning the networking and introducing points of failure to meet federal guidelines for decency.  SDN holds the promise of making that easier.  In the above Tail-f example, the slide deck shows a UI that allows simple blocking of common protocols like Skype.  This could be extended to schools where student computers and wireless networks are identified and bad programs are disallowed while web traffic is pushed to a filter and scrubbed before heading out to the Wild Wild Web.  SDN can’t solve every problem we might have, but if it can make the mundane and time consuming problems easier, it might just give people the breathing room they need to work on the bigger issues.

An Educational SDN Use Case

During the VMUnderground Networking Panel, we had a great discussion about software defined networking (SDN) among other topics. Seems that SDN is a big unknown for many out there. One of the reasons for this is the lack of specific applications of the technology. OSPF and SQL are things that solve problems. Can the same be said of SDN? One specific question regarded how to use SDN in small-to-medium enterprise shops. I fired off an answer from my own experience:

Since then, I’ve had a few people using my example with regards to a great use case for SDN. I decided that I needed to develop it a bit more now that I’ve had time to think about it.

Schools are a great example of the kinds of “do more with less” organizations that are becoming more common. They have enterprise-class networks and needs and live off budgets that wouldn’t buy janitorial supplies. In fact, if it weren’t for E-Rate, most schools would have technology from the Stone Age. But all this new tech doesn’t help if you can’t find a way for it to be used to the fullest for the purposes of educating students.

In my example, I talked about the shift from written standardized testing to online assessments. Oklahoma and Indiana are leading the way in getting rid of Scantrons and #2 pencils in favor of keyboards and monitors. The process works well for the most part with proper planning. My old job saw lots of scrambling to prep laptops, tablets, and lab machines for the rigors of running the test. But no amount of pre-config could prepare for the day when it was time to go live. On those days, the network was squarely in the sights of the administration.

I’ve seen emails go around banning non-testing students from the computers. I’ve seen hard-coded DNS entries on testing machines while the rest of the school had DNS taken offline to keep them from surfing the web. Redundant circuits. QoS policies that would make voice engineers cry. All in the hope of keeping the online test bandwidth free to get things completed in the testing window. All the while, I was thinking to myself, “There has got to be an easier way to do this…”

Redefining with Software

Enter SDN. The original use case for SDN at Stanford was network slicing. The Next-Gen Network Team wanted to use the spare capacity of the network for testing without crashing the whole system. Being able to reconfigure the network on the fly is a huge leap forward. Pushing policy into devices without CLI cuts down on the resume-generating events (RGE) in production equipment. So how can we apply these network slicing principles to my example?

On the day of the test, have the configuration system push down a new policy that gives the testing machines a guaranteed amount of bandwidth. This reservation will ensure each machine is able to get what it needs without being starved out. With SDN, we can set this policy on a per-IP basis to ensure it is enforced. This slice will exist separate from the production network to ensure that no one starting a huge FTP transfer or video upload will disrupt testing. By leaving the remaining bandwidth intact for the rest of the school’s production network administrators can ensure that the rest of the student body isn’t impacted during testing. With the move toward flipped classrooms and online curriculum augmentation, having available bandwidth is crucial.

Could this be done via non-SDN means? Sure. Granted, you’ll have to plan the QoS policy ahead of time and find a way to classify your end-user workstations properly. You’ll also have to tune things to make sure no one is dominating the test machine pool. And you have to get it right on every switch you program. And remove it when you’re done. Unless you missed a student or a window, in which case you’ll need to reprogram everything again. SDN certainly makes this process much easier and less disruptive.

Tom’s Take

SDN isn’t a product. You can’t order a part number for SDN-001 and get a box labeled SDN. Instead, it’s a process. You apply SDN to existing environment and extend the capabilities through new processes. Those processes need use cases. Use cases drive business cases. Business cases provide buy in from the stakeholders. That’s why discussing cases like the one above are so important. When you can find a use for SDN, you can get people to accept it. And that’s half the battle.

CCNA Data Center on vBrownBag

vbrownbagSometimes when I’m writing blog posts, I forget how important it is to start off on the right foot.  For a lot of networking people just starting out, discussions about advanced SDN topics and new theories can seem overwhelming when you’re trying to figure out things like subnetting or even what a switch really is.  While I don’t write about entry level topics often, I had the good fortune recently to talk about them on the vBrownBag podcast.

For those that may not be familiar, vBrownBag is a great series that goes into depth about a number of technology topics.  Historically, vBrownBag has been focused on virtualization topics.  Now, with the advent of virtual networking become more integrated into virtualization the vBrownBag organizers asked me if I’d be willing to jump on and talk about the CCNA Data Center.  Of course I took the opportunity to lend my voice to what will hopefully be the start of some promising data center networking careers.

These are the two videos I recorded.  The vBrownBag is usually a one-hour show.  I somehow managed to go an hour and half on both.  I realized there is just so much knowledge that goes into these certifications that I couldn’t do it all even if I had six hours.

Also, in the midst of my preparation, I found a few resources that I wanted to share with the community for them to get the most out of the experience.

Chris Wahl’s CCNA DC course from PluralSight – This is worth the time and investment for sure.  It covers DCICN in good depth, and his work with NX-OS is very handy if you’ve never seen it before.

Todd Lamle’s NX-OS Simulator – If you can’t get rack time on a real Nexus, this is pretty close to the real thing.  You should check it out even if only to get familiar with the NX-OS CLI.

NX-OS and Nexus Switching, 2nd Edition – This is more for post-grad work.  Ron Fuller (@CCIE5851) helped write the definitive guide to NX-OS.  If you are going to work on Nexus gear, you need a copy of this handy. Be sure to use the code “NETNERD” to get it for 30% off!


Tom’s Take

Never forget where you started.  The advanced topics we discuss take a lot for granted in the basic knowledge department.  Always be sure to give a little back to the community in that regard.  The network engineer you help shepherd today may end up being the one that saves your job in the future.  Take the time to show people the ropes.  Otherwise you’ll end up hanging yourself.

SDN 101 at ONUG Academy


Software defined networking is king of the hill these days in the greater networking world.  Vendors are contemplating strategies.  Users are demanding functionality.  And engineers are trying to figure out what it all means.  What’s needed is a way for vendor-neutral parties to get together and talk about what SDN represents and how best to implement it.  Most of the talk so far has been at vendor-specific conferences like Cisco Live or at other conferences like Interop.  I think a third option has just presented itself.

Nick Lippis (@NickLippis) has put together a group of SDN-focused people to address concerns about implementation and usage.  The Open Networking User Group (ONUG) was assembled to allow large companies using SDN to have a semi-annual meeting to discuss strategy and results.  It allows Facebook to talk to JP Morgan about what they are doing to simplify networking through use of things like OpenFlow.

This year, ONUG is taking it a step further by putting on the ONUG Academy, a day-long look at SDN through the eyes of those that implement it.  They have assembled a group of amazing people, including the founder of Cumulus Networks and Tech Field Day’s own Brent Salisbury (@NetworkStatic).  There will be classes about optimizing networks for SDN as well as writing SDN applications for the most popular controllers on the market.  Nick shares more details about the ONUG academy here:

If you’re interested in attending ONUG either for the academy or for the customer-focused meetings, you need to register today.  As a special bonus, if you use the code TFD10 when you sign up, you can take 10% of the cost of registration.  Use that extra cash to go out and buy a cannoli or two.

I’ll be at ONUG with Tech Field Day interviewing customers and attendees about their SDN strategies as well as where they think the state of the industry is headed.  If you’re there, stop by and say hello.  And be sure to bring me one of those cannolis.

Know the Process, Not the Tool


If there is one thing that amuses me as of late, it’s the “death of CLI” talk that I’m starting to see coming from many proponents of software defined networking. They like to talk about programmatic APIs and GUI-based provisioning and how everything that network engineers have learned is going to fall by the wayside.  Like this Network World article. I think reports of the death of CLI are a bit exaggerated.

Firstly, the CLI will never go away. I learned this when I stared working with an Aerohive access point I got at Wireless Field Day 2. I already had a HiveManager account provisioned thanks to Devin Akin (@DevinAkin), so all I needed to do was add the device to my account and I would be good to go. Except it never showed up. I could see it on my local network, but it never showed up in the online database. I rebooted and reset several times before flipping the device over and finding a curious port labeled “CONSOLE”. Why would a cloud-based device need a console port. In the next hour, I learned a lot about the way Aerohive APs are provisioned and how there were just some commands that I couldn’t enter in the GUI that helped me narrow down the problem. After fixing a provisioning glitch in HiveManager the next day, I was ready to go. The CLI didn’t fix my problem, but I did learn quite a bit from it.

Basic interfaces give people a great way to see what’s going on under the hood. Given that most folks in networking are from the mold of “take it apart to see why it works” the CLI is great for them. I agree that memorizing a 10-argument command to configure something like route redistribution is a pain in the neck, but that doesn’t come from the difficulty of networking. Instead, the difficulty lies in speaking the language.

I’ve traveled to a foreign country once or twice in my life. I barely have a grasp of the English language at times. I can usually figure out some Spanish. My foreign language skills have pretty much left me at this point. However, when I want to make myself understood to people that speak another language, I don’t focus on syntax. Instead, I focus on ideas. Pointing at an object and making gestures for money usually gets the point across that I want to buy something. Pantomiming a drinking gesture will get me to a restaurant.

Networking is no different. When I started trying to learn CLI terminology for Brocade, Arista, and HP I found they were similar in some respects but very different in others. When you try to take your Cisco CLI skills to a Juniper router, you’ll find that you aren’t even in the neighborhood when it comes to syntax. What becomes important is *what* you’re trying to do. If you can think through what you’re trying to accomplish, there’s usually a help file or a Google search that can pull up the right way to do things.

This extends its way into a GUI/API-driven programming interface as well. Rather than trying to intuit the interface just think about what you want to do instead. If you want two hosts to talk to each other through a low-cost link with basic security you just have to figure out what the drag-and-drop is for that. If you want to force application-specific traffic to transit a host running an intrusion prevention system you already know what you want to do. It’s just a matter of find the right combination of interface programming to accomplish it. If you’re working on an API call using Python or Java you probably have to define the constraints of the system anyway. The hard part is writing the code to interface to accomplish the task.

Tom’s Take

Learning the process is the key to making it in networking. So many entry level folks are worried about *how* to do something. Configuring a route or provisioning a VLAN are the end goal. It’s only when those folks take a step back and think about their task without the commands that they begin to become real engineers. When you can visualize what you want to do without thinking about the commands you need to enter to do it, you are taking the logical step beyond being tied to a platform. Some of the smartest people I know break a task down into component parts and steps. When you spend more time on *what* you are doing and less on *how* you are doing it, you don’t need to concern yourself with radical shifts in networking, whether they be SDN, NFV, or the next big thing. Because the process will never change even if the tools might.