CCIE at 50k: Software Defined? Or Hardware Driven?


Congratulations to Ryan Booth (@That1Guy_15) on becoming CCIE #50117. It’s a huge accomplishment for him and the networking community. Ryan has put in a lot of study time so this is just the payoff for hard work and a job well done. Ryan has done something many dream of and few can achieve. But where is the CCIE program today? And where will it be in the future?

Who Wants To Be A CCIE?

A lot of virtual ink has been committed to opinions in the past couple of years about how the CCIE is become increasingly irrelevant in a world of software defined DevOps focused non-traditional networking teams. It has been said that the CCIE doesn’t teach modern networking concepts like programming or building networks in a world with no CLI access. While this is all true, I don’t think it diminishes the value of getting a CCIE.

The CCIE has never been about building a modern network. It has never been focused on creating anything other than a medium-sized enterprise network in the case of the routing and switching exam. It is not a test of best practices or of greenfield deployment scenarios. Instead, it has been a test of interoperability with an exisiting architecture. It tests the ability of the candidate to add devices and protocols to a stable existing network.

Other flavors of the CCIE test over different protocols or technologies, but the idea is still the same. The only one that even comes close to requiring programming is the CCIE Collaboration, which tests over the ability to customize Cisco Contact Center scripts. Otherwise, each test focuses on technology implementation and not architecture or operation.

Current logic dictates that people don’t want to take the CCIE because it doesn’t teach programming or API interaction. Yet candidates are showing up in droves. It’s almost as if the networks we have today are going to need to be maintained and built out over the coming years. These are the kinds of tasks that are well suited to a support-focused certification like the CCIE. The ideal CCIE candidate isn’t using Vagrant and Chef in a lab somewhere. They’re muddling through OSPF to RIP distribution somewhere in the dark corners of a network that got welded on after an acquisition.

Is Everyone A CCIE?

One thing I have noticed about the CCIE is the fact the number climb seems to have leveled off. It’s not the rapid explosion of certifications that it has been in the past, nor is it the eventual cliff of increased difficulty. Things seem to be marching more toward steady growth. I don’t know how much of that can be attributed to factors like the Cisco official CCIE training program or the upgrade to version 5 almost two years ago.

Lots of CCIEs doesn’t necessarily mean that the test has lost meaning. Microsoft had several thousand MCSEs by the time the certification became a punchline to countless call center jokes. Novell had a virtual army of Certified NetWare Engineers (CNEs) before software changes locked many of them into CNE 5 or CNE 6. Having a lot of certified individuals doesn’t devalue the certificaiton. It’s what people do with it that creates the reputation. Ask and Novell Certififed Directory Engineer (CDE) about the reputation garnered by a test and they can give you a lesson in hard exams that breed bright engineers.

Does that mean that we should brace ourselves for even more CCIEs in the future? It likely won’t be as bad as has been imagined. The written exam for version 5 has pointed out to me that Cisco is going to start closing ranks around technologies in the near future. The written exam serves as a testing ground for potential new topics on the exam. MPLS was a written topic long before it became a potential lab exam topic. The current written exam is full of technologies that make me think Cisco is starting to put more emphasis on the Cisco and less on the Internetworking in CCIE.

Cisco wants to have a legion of certified individuals that think about Cisco technology benefits. That’s why we’re starting to see a shift toward things like DMVPN and GETVPN in testing. In place of industry standard protocols, we get the Cisco improved versions. This locks candidates into the Cisco method of thinking and ensures that their go-to solutions will include some form of proprietary technology.

If this shift in thinking is really the start of the new way of certification testing, I worry for the future of the CCIE. Not because there are 50,000 CCIEs, but because the new inductees into the CCIE group will be focused on creating islands of Cisco in the sea of interoperable data center networks. That’s good for Cisco’s bottom line, but bad for the reputation of the CCIE. Could you imagine what would happen if a CCIE walked in and told you they couldn’t fix your MPLS VPN configuration issues because “I only know how to work on DMVPN”?

Tom’s Take

Every time someone I know passes the CCIE it makes me happy that they’ve completed a rigorous exam testing process. It tells me this person knows how to follow the lab instructions to create an interoperable enterprise network based on constraints. It also tells me that this person knows how to study material and doesn’t give up. Those are the kinds of people I would want in my networking group.

CCIEs are the perfect people to learn more modern network techniques like programmability and SDN. Not because they learned how to do it on their test. But because they are the kinds of people that learn well and will apply everything they have to picking up a new concept. But it needs to be pointed out here that Cisco must foster that kind of interoperable learning experience with CCIEs. Focusing too heavily on proprietary solutions to help create an army of unknowing Cisco SEs in the field will only serve to hurt Cisco in the future when that group of certified individuals must learn to work in the world of networking post-SD.


Q And A Should Include The E

The IT world is cyclical for sure. I’ve seen trends and topics repeating themselves over and over again in my relatively short time here. I find it interesting that we keep solving similar problems over and over again. I also find it fascinating that this particular issue leads to the reason why blogs are so important.

Any Questions?

Questions abound in IT. It’s the nature of the industry. However, it’s not just new questions that we create when technology leaps past us. We keep asking the same questions over and over again. This is the field of study that created the FAQ, remember?

In recent memory, I find the same questions being asked over and over again:

  • What is SDN?
  • How can SDN help me?
  • What makes this different from what we’ve done before?

You’ve probably asked those very same questions. Perhaps you found the answers you were looking for. Perhaps you’re still trying to figure it out. The problem is that those questions are still being asked. The industry should have evolved to the point where the simple questions have been answered with simple answers. Complex questions, or those questions that need more in-depth discussion, should be treated as such. Yes, the question of what SDN really is would take more than a cursory paragraph on a blog, but we should be able to at least answer it with enough specificity to make the user not feel like they been slighted.

Questions will never stop coming in IT. But how should we handle them?

Any Answers?

Questions may abound in IT, but the answers drive IT. People make a career out of being the person with the answers. It’s in all the marketing jargon. It’s why we create blogs. Even though most of my writing in the last year has been focused on industry trends or non-technical focused posts, the top three posts on my blog are still answers to simple questions:

  1. When Is A Trunk Not A Trunk?
  2. Switchport Voice VLAN – What Does It Do?
  3. Why Is My SFP Not Working?

These posts are far and away the most popular. I even saw this a few months ago and it made me smile:

This would make it seem like people are in need of answers. Any blogger can look at the incoming search terms for their blog and see all the things that brought readers to them. People want answers and they will keep looking until they find them. But why?

Explain It

I never understood why people kept searching for answers until I thought about satisfaction. I think Randall Munroe summed up the satisfaction (or lack thereof) angle here:

Who are you, DenverCoder9?!? (Thanks XKCD)

Who are you, DenverCoder9?!? (Thanks XKCD)

People can find answers easily. But they won’t stop looking until they are satisfied with the answer. It’s easy to find people saying things like “That’s not supported” or “RTFM” when you’re looking for an answer to a particularly difficult problem. And if you’ve ever called a tech support line, you know how unfulfilling the unsupported answer can feel.

That’s when explanation comes into play for me. First, an admission: I’m a chronic explainer. If you’ve ever met me and had a conversation with me for more than three minutes, you know I explain things. I talk about comic books and movies and technical topics in more depth than I should. That’s because I want things explained to me. Explaining how OSPF area calculations are done is as important as explaining how Captain America ended up wielding Mjolnir.

Think about the following answers:

This is unsupported.


This is unsupported on that platform because the CPU doesn’t have enough horsepower to process the packets in real time. We tried cutting down on the processing time but it just overwhelmed the unit no matter how much we tried. So rather than dealing with poor performance, we marked it as unsupported.

Both answers are technically correct. But the second is much more satisfying because the explanation is there instead of just the distilled answer.

The IT world needs more explanation. We need to know why things work the way they do instead of just getting a response of a few words. The explanation has the keys to understanding the answer to the question in its totality. It prevents us from asking the same questions over and over again. It leaves us fulfilled and ready to seek out the next question that needs to be asked.

How Do You Spell That?

I spent a bit of my career on the phone doing support for a national computer vendor. In addition to the difficulties of walking people through opening the case and diagnosing motherboard issues, I found myself needing to overcome language barriers. While I only have a hint of an accent (or so I’ve been told), spelling out acronyms was a challenge. That’s where the phonetic alphabet comes into play

By now, almost everyone uses the NATO phonetic alphabet. It’s the most recognized in the world. The US joint Army/Navy version varies a bit but does have a lot of similarities. However, when I first started out using the NATO version quite a few callers didn’t know what Lima was or giggled when I said Tango.

I decided that some people have much more familiarity with first names. This was borne out when I kept using Mary for “M” instead of Mike. People immediately knew it. Same for Victor, Peter, and so on. So I cobbled together my own Name Phonetic Alphabet.

A – Adam
B – Barbara
C – Charlie
D – David
E – Edward
F – Frank
G – George
H – Harold
I – Irwin
J – John
K – Kevin
L – Larry
M – Mary
N – Nancy
O – Oliver
P – Peter
Q – Quincy (or queen)
R – Roger
S – Sam
T – Tom (my favorite)
U – Umbrella
V – Victor
W – William
X – X-Ray
Y – Yellow
Z – Zebra

Finding a name for Y and Z was pretty difficult, but everyone knows Yellow and Zebra. I was tempted to use Zander, but the more popular version of that name from Buffy the Vampire Slayer was spelled Xander. No sense confusing folks. As for X, if you don’t know X from the sound we need to have a chat.

Was it a duplication of effort? Certainly. But it works universally with everyone I’ve ever talked to, including children. It makes “Roger Adam Irwin David” easy to get across to people without trying to remember Romeo and India.

The key to communication with others is to find something that works for you.  If you can easily convey your information to someone else, the shortcuts you take don’t matter.  If first names work best, use them.  If drawing pictures works better, use those.  In the end, getting the point across is the goal.

SDN Use Case: Content Filtering

K-12 schools face unique challenges with their IT infrastructure.  Their user base needs access to a large amount of information while at the same time facing restrictions.  While it does sound like some corporate network policies, the restrictions in the education environment are legal in nature.  Schools must find new ways to provide the assurance of restricting content without destroying their network in the process.  Which lead me to ask: Can SDN Help?

Online Protection

The government E-Rate program gives schools money each year under Priority 1 funding for Internet access.  Indeed, the whole point of the E-Rate program is to get schools connected to the Internet.  But we all know the Internet comes with a bevy of distractions. Many of those distractions are graphic in nature and must be eliminated in a school.  Because it’s the law.

The Children’s Internet Protection Act (CIPA) mandates that schools and libraries receiving E-Rate funding for high speed broadband Internet connections must filter those connections to remove questionable content.  Otherwise they risk losing funding for all E-Rate services.  That makes content filters very popular devices in schools, even if they aren’t funded by E-Rate (which they aren’t).

Content filters also cause network design issues.  In the old days, we had to put the content filter servers on a hub along with the outbound Internet router in order to insure they could see all the traffic and block the bad bits.  That became increasing difficult as network switch speeds increased.  Forcing hundreds of megabits through a 10Mbit hub was counterproductive.  Moving to switchport mirroring did alleviate the speed issues, but still caused network design problems.  Now, content filters can run on firewalls and bastion host devices or are enabled via proxy settings in the cloud.  But we all know that running too many services on a firewall causes performance issues.  Or leads to buying a larger firewall than needed.

Another issue that has crept up as of late is the use of Virtual Private Networking (VPN) as a way to defeat the content filter.  Setting up an SSL VPN to an outside, non-filtered device is pretty easy for a knowledgeable person.  And if that fails, there are plenty of services out there dedicated to defeating content filtering.  While the aim of these service is noble, such as bypassing the Great Firewall of China or the mandated Internet filtering in the UK, they can also be used to bypass the CIPA-mandated filtering in schools as well.  It’s a high-tech game of cat-and-mouse.  Blocking access to one VPN only for three more to pop up to replace it.

Software Defined Protection

So how can SDN help?  Service chaining allows traffic to be directed to a given device or virtual appliance before being passed on through the network.  This great presentation from Networking Field Day 7 presenter Tail-f Networks shows how service chaining can force traffic through security devices like IDS/IPS and through content filters as well.  There is no need to add hubs or mirrored switch ports in your network.  There is also no need to configure traffic to transit the same outbound router or firewall, thereby creating a single point of failure.  Thanks to the magic of SDN, the packets go to the filter automatically.  That’s because they don’t really have a choice.

It also works well for providers wanting to offer filtering as a service to schools.  This allows a provider to configure the edge network to force traffic to a large central content filter cluster and ensure delivery.  It also allows the service provider network to operate without impact to non-filtered customers.  That’s very useful even in ISPs dedicated to education institutions, as the filter provisions for K-12 schools don’t apply to higher education facilities, like colleges and universities.  Service chaining would allow the college to stay free and clear while the high schools are cleansed of inappropriate content.

The VPN issue is a thorny one for sure.  How do you classify traffic that is trying to hide from you?  Even services like Netflix are having trouble blocking VPN usage and they stand to lose millions if they can’t.  How can SDN help in this situation? We could build policies to drop traffic headed for known VPN endpoints.  That should take care of the services that make it easy to configure and serve as a proxy point.  But what about those tech-savvy kids that setup SSL VPNs back home?

Luckily, SDN can help there as well.  Many unified threat management appliances offer the ability to intercept SSL conversations.  This is an outgrowth of sites like Facebook defaulting to SSL to increase security.  SSL intercept essentially acts as a man-in-the-middle attack.  The firewall decrypts the SSL conversation, scans the packets, and re-encrypts it using a different certificate.  When the packets come back in, the process is reversed.  This SSL intercept capability would allow those SSL VPN packets to be dropped when detected.  The SDN component ensures that HTTPS traffic is always redirected to a device that and do SSL intercept, rather than taking a path through the network that might lead to a different exit point.

Tom’s Take

Content filtering isn’t fun.  I’ve always said that I don’t envy the jobs of people that have to wade through the unsavory parts of the Internet to categorize bits as appropriate or not.  It’s also a pain for network engineers that need to keep redesigning the networking and introducing points of failure to meet federal guidelines for decency.  SDN holds the promise of making that easier.  In the above Tail-f example, the slide deck shows a UI that allows simple blocking of common protocols like Skype.  This could be extended to schools where student computers and wireless networks are identified and bad programs are disallowed while web traffic is pushed to a filter and scrubbed before heading out to the Wild Wild Web.  SDN can’t solve every problem we might have, but if it can make the mundane and time consuming problems easier, it might just give people the breathing room they need to work on the bigger issues.

An Educational SDN Use Case

During the VMUnderground Networking Panel, we had a great discussion about software defined networking (SDN) among other topics. Seems that SDN is a big unknown for many out there. One of the reasons for this is the lack of specific applications of the technology. OSPF and SQL are things that solve problems. Can the same be said of SDN? One specific question regarded how to use SDN in small-to-medium enterprise shops. I fired off an answer from my own experience:

Since then, I’ve had a few people using my example with regards to a great use case for SDN. I decided that I needed to develop it a bit more now that I’ve had time to think about it.

Schools are a great example of the kinds of “do more with less” organizations that are becoming more common. They have enterprise-class networks and needs and live off budgets that wouldn’t buy janitorial supplies. In fact, if it weren’t for E-Rate, most schools would have technology from the Stone Age. But all this new tech doesn’t help if you can’t find a way for it to be used to the fullest for the purposes of educating students.

In my example, I talked about the shift from written standardized testing to online assessments. Oklahoma and Indiana are leading the way in getting rid of Scantrons and #2 pencils in favor of keyboards and monitors. The process works well for the most part with proper planning. My old job saw lots of scrambling to prep laptops, tablets, and lab machines for the rigors of running the test. But no amount of pre-config could prepare for the day when it was time to go live. On those days, the network was squarely in the sights of the administration.

I’ve seen emails go around banning non-testing students from the computers. I’ve seen hard-coded DNS entries on testing machines while the rest of the school had DNS taken offline to keep them from surfing the web. Redundant circuits. QoS policies that would make voice engineers cry. All in the hope of keeping the online test bandwidth free to get things completed in the testing window. All the while, I was thinking to myself, “There has got to be an easier way to do this…”

Redefining with Software

Enter SDN. The original use case for SDN at Stanford was network slicing. The Next-Gen Network Team wanted to use the spare capacity of the network for testing without crashing the whole system. Being able to reconfigure the network on the fly is a huge leap forward. Pushing policy into devices without CLI cuts down on the resume-generating events (RGE) in production equipment. So how can we apply these network slicing principles to my example?

On the day of the test, have the configuration system push down a new policy that gives the testing machines a guaranteed amount of bandwidth. This reservation will ensure each machine is able to get what it needs without being starved out. With SDN, we can set this policy on a per-IP basis to ensure it is enforced. This slice will exist separate from the production network to ensure that no one starting a huge FTP transfer or video upload will disrupt testing. By leaving the remaining bandwidth intact for the rest of the school’s production network administrators can ensure that the rest of the student body isn’t impacted during testing. With the move toward flipped classrooms and online curriculum augmentation, having available bandwidth is crucial.

Could this be done via non-SDN means? Sure. Granted, you’ll have to plan the QoS policy ahead of time and find a way to classify your end-user workstations properly. You’ll also have to tune things to make sure no one is dominating the test machine pool. And you have to get it right on every switch you program. And remove it when you’re done. Unless you missed a student or a window, in which case you’ll need to reprogram everything again. SDN certainly makes this process much easier and less disruptive.

Tom’s Take

SDN isn’t a product. You can’t order a part number for SDN-001 and get a box labeled SDN. Instead, it’s a process. You apply SDN to existing environment and extend the capabilities through new processes. Those processes need use cases. Use cases drive business cases. Business cases provide buy in from the stakeholders. That’s why discussing cases like the one above are so important. When you can find a use for SDN, you can get people to accept it. And that’s half the battle.

CCNA Data Center on vBrownBag

vbrownbagSometimes when I’m writing blog posts, I forget how important it is to start off on the right foot.  For a lot of networking people just starting out, discussions about advanced SDN topics and new theories can seem overwhelming when you’re trying to figure out things like subnetting or even what a switch really is.  While I don’t write about entry level topics often, I had the good fortune recently to talk about them on the vBrownBag podcast.

For those that may not be familiar, vBrownBag is a great series that goes into depth about a number of technology topics.  Historically, vBrownBag has been focused on virtualization topics.  Now, with the advent of virtual networking become more integrated into virtualization the vBrownBag organizers asked me if I’d be willing to jump on and talk about the CCNA Data Center.  Of course I took the opportunity to lend my voice to what will hopefully be the start of some promising data center networking careers.

These are the two videos I recorded.  The vBrownBag is usually a one-hour show.  I somehow managed to go an hour and half on both.  I realized there is just so much knowledge that goes into these certifications that I couldn’t do it all even if I had six hours.

Also, in the midst of my preparation, I found a few resources that I wanted to share with the community for them to get the most out of the experience.

Chris Wahl’s CCNA DC course from PluralSight – This is worth the time and investment for sure.  It covers DCICN in good depth, and his work with NX-OS is very handy if you’ve never seen it before.

Todd Lamle’s NX-OS Simulator – If you can’t get rack time on a real Nexus, this is pretty close to the real thing.  You should check it out even if only to get familiar with the NX-OS CLI.

NX-OS and Nexus Switching, 2nd Edition – This is more for post-grad work.  Ron Fuller (@CCIE5851) helped write the definitive guide to NX-OS.  If you are going to work on Nexus gear, you need a copy of this handy. Be sure to use the code “NETNERD” to get it for 30% off!


Tom’s Take

Never forget where you started.  The advanced topics we discuss take a lot for granted in the basic knowledge department.  Always be sure to give a little back to the community in that regard.  The network engineer you help shepherd today may end up being the one that saves your job in the future.  Take the time to show people the ropes.  Otherwise you’ll end up hanging yourself.

SDN 101 at ONUG Academy


Software defined networking is king of the hill these days in the greater networking world.  Vendors are contemplating strategies.  Users are demanding functionality.  And engineers are trying to figure out what it all means.  What’s needed is a way for vendor-neutral parties to get together and talk about what SDN represents and how best to implement it.  Most of the talk so far has been at vendor-specific conferences like Cisco Live or at other conferences like Interop.  I think a third option has just presented itself.

Nick Lippis (@NickLippis) has put together a group of SDN-focused people to address concerns about implementation and usage.  The Open Networking User Group (ONUG) was assembled to allow large companies using SDN to have a semi-annual meeting to discuss strategy and results.  It allows Facebook to talk to JP Morgan about what they are doing to simplify networking through use of things like OpenFlow.

This year, ONUG is taking it a step further by putting on the ONUG Academy, a day-long look at SDN through the eyes of those that implement it.  They have assembled a group of amazing people, including the founder of Cumulus Networks and Tech Field Day’s own Brent Salisbury (@NetworkStatic).  There will be classes about optimizing networks for SDN as well as writing SDN applications for the most popular controllers on the market.  Nick shares more details about the ONUG academy here:

If you’re interested in attending ONUG either for the academy or for the customer-focused meetings, you need to register today.  As a special bonus, if you use the code TFD10 when you sign up, you can take 10% of the cost of registration.  Use that extra cash to go out and buy a cannoli or two.

I’ll be at ONUG with Tech Field Day interviewing customers and attendees about their SDN strategies as well as where they think the state of the industry is headed.  If you’re there, stop by and say hello.  And be sure to bring me one of those cannolis.