Iron Chef: Certification Edition

My friend Joshua Williams (@802DotMe) texted me today with a great quote that I wanted to share with you that made me think about certifications:

You’ve probably already thought through this extensively, and maybe even written about it, but after sitting through another 8 hour practical exam yesterday I’m more convinced than ever that expert level exams from technical companies are more analogous to a gimmicky Food Network TV show than real world application of technical acumen. They don’t care so much about my skill level as they do about what kind of meal I can prepare in 30 minutes using Tialapia, grapes, and Dr. Pepper syrup with my salt shaker taken away halfway through.

I laughed because it’s true. And then I thought about it more and realized he’s way more than right. We know for a fact that companies love to increase the level of challenge in their exams from novice to expert. It’s a way to weed out the people that aren’t committed to learning about something. However, as the questions and tasks get harder it becomes much more difficult to get a good sense of how candidates are going to perform.

Boiling Water Isn’t Hard?

When you look at something like the CCNA, they’re trying to make sure you know how networks actually work. The simulations and lab exercises are pretty basic. Can you configure RIP correctly? Do you know the command to enable a switch port? There isn’t a need to get crazy with it. Using Joshua’s analogy from above, it’s not unlike a show like Worst Cooks in America, where the basics are the challenge that needs to be overcome. Not everyone is a superstar chef. Sometimes getting the building blocks right is more than half the battle.

As you move up the ladder, the learning gets harder. You dive deep into protocols and see how technologies build on each other. You need to configure BGP, but you also need to have some kind of other IGP running to distribute the routes. You need to remember that this spice goes in while the dish is cooking and this other goes on at the end so the flavor isn’t destroyed. I would liken this to a “fun” challenge cooking show, where the expert Food Network Chef faces off against someone that isn’t in the food business at a high professional level. Maybe they run a diner or are a short-order cook in a hotel restaurant. They aren’t looking to create their own signature dish. They know enough to cook what tastes good. But ask them to make hollandaise sauce or make pufferfish sashimi and they’re out.

Which brings us to the highest level of learning. The expert certification tracks. These are the crowing achievements of a career. They are the level that you have to be at to prove you know the technology inside and out. How do you test that, exactly? Microsoft had a great way of doing it back in the day with some of the mastery programs. You went to Redmond and you spent a couple of months learning the technology with the people that wrote it. It was very similar to a doctor’s internship in a hospital. You did the work with people that knew what you needed to know. They corrected you and helped you grown your knowledge. Even though you were an expert you understood what needed to be done and how to get there. At the end you took an exam to cover what you had learned and you earned your mastery.

Other certification programs don’t do that. Instead, they try to trip you up with tricky scenarios and make you make mistakes if you’re not paying attention. This is the Iron Chef round. You know your stuff, eh? Face off against this hard challenge. And by the way, here’s your curveball: You have to use this crazy extra ingredient. A show like Chopped does this a lot too. You need to make a meal using chicken, soy sauce, and candy corn. Are they testing your ability to prepare food? Or trying to figure out how creative you can be with a set of constraints that don’t make sense?

Ala Config!

The theory behind this kind of challenge is sound on paper. You never know what you’re going to walk into and what you’ll be forced to fix. I’ve had some real interesting problems that I’ve needed to solve over my career. But in every crazy case I never had to deal with the kinds of constrained setups that you get in lab-based exams. Configure this protocol, but don’t use these options. Make this connection work this way using one of these options but know that picking the wrong one will wreck your configuration in about two hours. Make trout-flavored ice cream. You name it and it’s a huge challenge for no good reason.

In theory, this is a great way to challenge your experts. In practice, it’s silly because you’re putting up barriers they will never see. Worse yet, you force them to start looking for the crazy constraints that don’t exist. One of my favorites is the overarching constraint in the CCIE lab that you are not allowed to use a static route to anything unless explicitly allowed in the question. Why? Because static routes don’t scale? Because they create administrative overhead? Or is it because a single static route fixes the problem and doesn’t require you to spend an hour tagging routes when redistribution happens? Static routes cut the Gordian Knot in the lab. So they can’t be allowed. Because that would make things too easy.


Tom’s Take

We need to move away from trivia and Iron Chef-style certifications. Instead of making our people dependent on silly tricks or restricting them from specific tools in their kit, we need to ensure their knowledge is at the right level. You would never ask a chef to cook an entire meal and not be able to use a saucepan. Why would you take away things like static routes or access lists from a network engineer’s arsenal? Instead of crafting the perfect tricky scenario to trap your candidates, spend the time instead teaching them what they need to know. Because once someone learns that trout is a horrible ice cream flavor we all win.

Thanks to Josh Williams for this great post idea!

The Certification Ladder

Are you climbing the certification ladder? If you’re in IT the odds are good that you are. Some people are just starting out and see certifications as a way to get the knowledge they need to do their job. Others see certs as a way to get out of a job they don’t like. Still others have plenty of certifications but want to get the ones at the top of their field. This last group are the ones that I want to spend some time talking about.

Pushing The Limit

Expert-level certifications aren’t easy on purpose. They’re supposed to represent the gap between being good at something and going above and beyond. For some that involves some kind of practical test of skills like the CCIE. For others it involves a board interview process like the VCDX. Or it could even involve a combination of things like the CWNE does with board review and documentation submissions.

Expert certifications aren’t designed to be powered through in a short amount of time. That’s because it’s difficult to become an expert at something without putting in the practice time. For some tests, that means meeting some minimum requirements. You can only attempt your VCDX when you have already passed the VCAP-DCA and VCAP-DCD test, for example. Or you may have a minimum requirement of time in the industry, such as the CISSP requirement of four years in the security industry.

But, more importantly, the requirement is that you truly are an expert. How many times have you bumped into someone that has a certification that you think to yourself, “How on earth did they pass that?” It should be fairly uncommon to run into a CCIE that you feel that way about. The test is rigorous and requires everyone to pass a very similar version of the practical exam. Sure, you still run into people that say the old 2-day exam was harder. But by and large, most CCIEs have had to endure the same kind of certification requirements.

Now, what people do after they get there is an entirely different matter altogether. There are a lot of people that get to the pinnacle of their certification journey and sit there on top of their mountain. They take time to survey the lands that they now watch over and they relax. They don’t see any need in going any further. They’ve done what they came to do. And for many that’s the way to go. Congratulations on your ride.

Still others use this opportunity negatively. They expect people to kiss the brass certificate and pay deference to them because of it. This can affect almost anyone. I remember years ago back to a time when I had just gotten my CCIE lab out of the way. I was working on a proposal for a customer. We had just gotten an email from the customer asking why we didn’t include a particular switch in the design. I told our team that we didn’t need it because the requirements of the design didn’t need something that cost three times over what we recommended. The customer’s response was, “Well, this other partner guy is a CCIE and he says we need that switch.” I replied back with, “Well, I’m a CCIE too, so let’s cut that crap and talk about the hardware.”

I’m not sure how many times that person had used the “I’m a CCIE” justification for their recommendations, but it shows me that some people believe a piece of paper speaks louder than their track record. Those people are usually the ones that fall back into the pattern of “listen to me because I passed tests” not “listen to me because I did the studying”. It’s important to ascribe value to passing a test, but remember that the test is a way to prove you have knowledge. It reminds me of this scene from Tommy Boy:

Throwing up a certification as justification for a recommendation is no different that just tossing a worthless guarantee on a box. Prove you know what you’re talking about instead of just saying you do.

Exceeding Your Reach

The last type of person that climbs the certification ladder is like the one in this tweet from my friend Hank Yeomans:

He looks at the ascent to the top of his certification ladder as a chance to do more. To build more. It’s not the end of the journey. It’s not bad to stop and look around at the new view from the top of your ladder when you’ve climbed it. But if you look at the journey as the start of something that you need to finish, you’re going to start immediately looking around to find the next thing that you need to do. Perhaps it’s learning a new technology related to the one that you just finished. Or maybe it’s that you want to figure out how to get even better at what you do.

People that never rest in their attempts to be better at the ones that ultimately change the way things are done. They don’t just accept that this is the way that things need to be. Instead, they use the top of their ladder to stretch out and see what they can reach. They realize that everything we do in life it just building on something else we’ve already done. We use Crawl, Walk, Run as a metaphor for building through a project or a process all the time. That’s because we know that you have to make steps all the time to progress. But what if someone just said, “You know what, I’ve mastered walking. I don’t need to run. All you people who only crawl listen to me because I’m better than you!” It would show how short-sighted they are when it comes to continuing the journey.


Tom’s Take

Many times, I’ve talked about the fact that I relaxed after I passed my CCIE and enjoyed not studying into the wee hours of the night. But after a while I started getting uncomfortable around 8-9pm. Because there was a little voice in the back of my head that kept telling me “You should be studying for something.” Instinctively, that voice knew that I needed to continue my journey. I would never be content resting on my laurels and I could never bring myself to use my certification as a crutch to make myself look important to others. Instead, I needed to push myself to build on what I’ve already done and make myself better. As Hank said, it’s just a foothill on a greater journey. Once you’ve learned how to use your ladder to increase your reach, even the sky isn’t the limit any longer.

The CCIE Times Are A Changing

Today is the day that the CCIE changes. A little, at least. The news hit just a little while ago that there are some changes to the way the CCIE certification and recertification process happens. Some of these are positive. Some of these are going to cause some insightful discussion. Let’s take a quick look at what’s changing and how it affects you. Note that these changes are not taking effect until February 24, 2020, which is in about 8 months.

Starting Your Engines

The first big change comes from the test that you take to get yourself ready for the lab. Historically, this has been a CCIE written exam. It’s a test of knowledge designed to make sure you’re ready to take the big lab. It’s also the test that has been used to recertify your CCIE status.

With the new change on Feb. 24th, the old CCIE written will go away. The test that is going to be used to qualify candidates to take the CCIE lab exam is the Core Technology exam from the CCNP track. The Core Technology exam in each CCNP track serves a dual purpose in the new Cisco certification program. If you’re going for your CCNP you need the Core Technology exam and one other exam from a specific list. That Core Technology exam also qualifies you to schedule a CCIE lab attempt within 18 months.

This means that the CCNP is going to get just a little harder now. Instead of taking multiple tests over routing, switching, or voice you’re going to have all those technologies lumped together into one long exam. There’s also going to be more practical questions on the Core Technologies exam. That’s great if you’re good at configuring devices. But the amount of content on the individual exam is going to increase.

Keeping The Home Fires Burning

Now that we’ve talked about qualification to take the lab exam, let’s discuss the changes to recertification. The really good news is that the Continuing Education program is expanding and giving more options for recertification.

The CCIE has always required you to recertify every two years. But if you miss your recertification date you have a one year “grace period”. Your CCIE status is suspended but you don’t lose your number until the end of the one-year period. This grace period has informally been called the “penalty box” by several people in the industry. Think of it like a time out to focus on getting your certification current.

Starting February 24, 2020, this grace period is now formalized as an extra year of certification. The CCIE will now be valid for 3 years instead of just 2. However, if you do not recertified by the end of the 3rd year, you lose your number. There is no grace period any longer. This means you need to recertify within the 3-year period.

As far as how to recertify, you now have some additional options. You can still recertify using CE credits. The amount has gone up from 100 to 120 credits to reflect the additional year that CCIEs get to recertify now. There is also a new way to recertify using a combination of CE credits and tests. You can take the Core Technologies exam and use 40 CE credits to recertify. You can also pass two Specialist exams and use 40 CE credits to recertify. This is a great way to pick up skills in a new discipline and learn new technologies. You can choose to pass a single Specialist exam and use 80 CE credits to recertify within the three-year period. This change is huge for those of us that need to recertify. It’s a great option that we don’t have today. They hybrid model offers great flexibility for those that are taking tests but also taking e-learning or classroom training.

The biggest change, however, is in the test-only option. Historically, all you needed to do is pass the CCIE written every two years to recertify. With the changes to the written exam used to qualify you to take the lab, that is no longer an option. As listed above, simply taking the Core Technologies exam is not enough. You must also take 40 CE credits.

So, what tests will recertify you? The first is the CCIE lab. If you take and pass a lab exam within the recertification period you’ll be recertified. You can also take three Specialist exams. The combination of three will qualify you for recertification. You can also take the Core Technologies exam and another professional exam to recertify. This means that passing the test required for the CCNP will recertify your CCIE. There is still one Expert-level exam that will work to recertify your CCIE – the CCDE written. Because no changes were made to the CCDE program in this project, the CCDE written exam will still recertify your CCIE.

Also, your recertification date is no longer dependent on your lab date. Historically your recert date was based on the date you took your lab. Now, it’s going to be whatever date you pass your exam or submit your CEs. The good news is this means that all your certifications are going to line up. Because your CCNA and CCNP dates have always been 3 years as well, recertifying your CCIE will sync up all your certifications to the date you recertify your CCIE. It’s a very welcome quality of life change.

Another welcome change is that there will no longer be a program fee when submitting your CE credits. As soon as you have amassed the right combination you just submit them and you’re good to go. No $300 fee. There’s also a great change for anyone that has been a CCIE for 20 years or more. If you choose to “retire” to Emeritus status you no longer have to pay the program fee. You will be a CCIE forever. Even if you are an active CCIE and you choose not to recertify after 20 years you will be automatically enrolled in the Emeritus program.

Managing Change

So, this is a big change. A single test will no longer recertify your number. You’re going to have to expand your horizons by investing in continuing education. You’re going to have to take a class or do some outside study on a new topic like wireless or security. That’s the encouragement from Cisco going forward. You’re not going to be able to just keep learning the same BGP and OSPF-related topics over and over again and hope to keep your certification relevant.

This is going to work out in favor of the people that complain the CCIE isn’t relevant to the IT world of today. Because you can learn about things like network automation and programmability and such from Cisco DevNet and have it count for CCIE recertification, you have no excuse not to bring yourself current to modern network architecture. You also have every opportunity to learn about new technologies like SD-WAN, ACI, and many other things. Increasing your knowledge takes care of keeping your CCIE status current.

Yes, you’re going to lose the ability to panic after two and a half years and cram to take a single test one or two times to reset for the next three years. You also need to be on top of your CCIE CE credits and your recert date. This means you can’t be lazy any longer and just assume you need to recertify every odd or even year. It means that your life will be easier without tons of cramming. But it means that the way things used to be aren’t going to be like that any longer.


Tom’s Take

Change is hard. But it’s inevitable. The CCIE is the most venerable certification in the networking world and one of the longest-lived certifications in the IT space. But that doesn’t mean it’s carved in stone as only being a certain way forever. The CCIE must change to stay relevant. And that means forcing CCIEs to stay relevant. The addition of the continuing education piece a couple of years ago is the biggest and best thing to happen in years. Expanding the ability for us to learn new technologies and making them eligible for us to recertify is a huge gift. What we need to do is embrace it and keep the CCIE relevant. We need to keep the people who hold those certifications relevant. Because the fastest way to fade into obscurity is to keep things the way they’ve always been.

You can find more information about all the changes in the Cisco Certification Program at http://Cisco.com/nextlevel

Home on the Palo Alto Networks Cyber Range

You’ve probably heard many horror stories by now about the crazy interviews that companies in Silicon Valley put you though. Sure, some of the questions are downright silly. How would I know how to weigh the moon? But the most insidious are the ones designed to look like skills tests. You may have to spend an hour optimizing a bubble sort or writing some crazy code that honestly won’t have much impact on the outcome of what you’ll be doing for the company.

Practical skills tests have always been the joy and the bane of people the world over. Many disciplines require you to have a practical examination before you can be certified. Doctors are one. The Cisco CCIE is probably the most well-known in IT. But what is the test really quizzing you on? Most people will admit that the CCIE is an imperfect representation of a network at best. It’s a test designed to get people to think about networks in different ways. But what about other disciplines? What about the ones where time is even more of the essence than it was in CCIE lab?

Red Team Go!

I was at Palo Alto Networks Ignite19 this past week and I got a chance to sit down with Pamela Warren. She’s the Director of Government and Industry Initiatives at Palo Alto Networks. She and her team have built a very interesting concept that I loved to see in action. They call it the Cyber Range.

The idea is simple enough on the surface. You take a classroom setting with some workstations and some security devices racked up in the back. You have your students log into a dashboard to a sandbox environment. Then you have your instructors at the front start throwing everything they can at the students. And you see how they respond.

The idea for the Cyber Range came out of military exercises that NATO used to run for their members. They wanted to teach their cyberwarfare people how to stop sophisticated attacks and see what their skill levels were with regards to stopping the people that could do potential harm to nation state infrastructure or worse to critical military assets during a war. Palo Alto Networks get involved in helping years ago and Pamela grew the idea into something that could be offered as a class.

Cyber Range has a couple of different levels of interaction. Level 1 is basic stuff. It’s designed to teach people how to respond to incidents and stop common exploits from happening. The students play the role of a security operations team member from a fictitious company that’s having a very bad week. You learn how to see the log files, collect forensics data, and ultimately how to identify and stop attackers across a wide range of exploits.

If Level 1 is the undergrad work, Cyber Range Level 2 is postgrad in spades. You dig into some very specific and complicated exploits, some of which have only recently been discovered. During my visit the instructors were teaching everyone about the exploits used by OilRig, a persistent group of criminals that love to steal data through things like DNS exfiltration tunnels. Level 2 of the Cyber Range takes you deep down the rabbit hole to see inside specific attacks and learn how to combat them. It’s a great way to keep up with current trends in malware and exploitive behavior.

Putting Your Money Where Your Firewall Is

To me, the most impressive part of this whole endeavor is how Palo Alto Networks realizes that security isn’t just about sitting back and watching an alert screen. It’s about knowing how to recognize the signs that something isn’t right. And it’s about putting an action plan into place as soon as that happens.

We talk a lot about automation of alerts and automated incident response. But at the end of the day we still need a human being to take a look at the information and make a decision. We can winnow that decision down to a simple Yes or No with all the software in the world but we need a brain doing the hard work after the automation and data analytics pieces give you all the information they can find.

More importantly, this kind of pressure cooker testing is a great way to learn how to spot the important things without failing in reality. Sure, we’ve heard all the horror stories about CCIE candidates that typed in debug IP packet detail on core switch in production and watched it melt down. But what about watching an attacker recon your entire enterprise and start exfiltrating data. And you being unable to stop them because you either don’t recognize the attack vector or you don’t know where to find the right info to lock everything down? That’s the value of training like the Cyber Range.

The best part for me? Palo Alto Networks will bring a Cyber Range to your facility to do the experience for your group! There are details on the page above about how to set this up, but I got a great pic of everything that’s involved here (sans tables to sit at):

How can you turn down something like this? I would have loved to put something like this on for some of my education customers back in the day!


Tom’s Take

I really wish I would have had something like the Cyber Range for myself back when I was fighting virus outbreaks and trying to tame Conficker infections. Because having a sandbox to test myself against scripted scenarios with variations run by live people beats watching a video about how to “easily” fix a problem you may never see in that form. I applaud Palo Alto Networks for their approach to teaching security to folks and I can’t wait to see how Pamela grows the Cyber Range program!

For more information about Palo Alto Networks and Cyber Range, make sure to visit http://Paloaltonetworks.com/CyberRange/

Certifications Are About Support

You may have seen this week that VMware has announced they are removing the mandatory recertification requirement for their certification program. This is a huge step from VMware. The VCP, VCAP, and VCDX are huge certifications in the virtualization and server industry. VMware has always wanted their partners and support personnel to be up-to-date on the latest and greatest software. But, as I will explain, the move to remove the mandatory recertification requirement says more about the fact that certifications are less about selling and more about supporting.

The Paper Escalator

Recertification is a big money maker for companies. Sure, you’re spending a lot money on things like tests and books. But those aren’t usually tied to the company offering the certification. Instead, the testing fees are given to the testing center, like Pearson, and the book fees go to the publisher.

The real money maker for companies is the first-party training. If the company developing the certification is also offering the training courses you can bet they’re raking in the cash. VMware has done this for years with the classroom requirement for the VCP. Cisco has also started doing in with their first-party CCIE training. Cisco’s example also shows how quality first-party content can drive out the third parties in the industry by not even suggesting to prospective candidates that this is another option to get their classroom materials.

I’ve railed against the VCP classroom requirement before. I think forcing your candidates to take an in-person class as a requirement for certification is silly and feels like it’s designed to make money and not make good engineers. Thankfully, VMware seems to agree with me in the latest release of info. They’re allowing the upgrade path to be used for their recertification process, which doesn’t necessarily require attendance in a classroom offering. I’d argue that it’s important to do so, especially if you’re really out of date with the training. But not needing it for certification is a really nice touch.

Keeping the Lights On

The other big shift with this certification change from VMware is the tacit acknowledgement that people aren’t in any kind of rush to upgrade their software right after the newest version is released. Ask any system administrator out there and they’ll tell you to wait for a service pack before you upgrade anything. System admins for VMware are as cautious as anyone, if not moreso. Too often, new software updates break existing functionality or cause issues that can’t be fixed without a huge time investment.

How is this affected by certification? Well, if I spent all my time learning VMware 5.x and I got my VCP on it because my company was using it you can better believe that my skill set is based around VCP5. If my company doesn’t decide to upgrade to 6.x or even 7.x for several years, my VCP is still based on 5.x technology. It shouldn’t expire just because I never upgraded to 6.x. The skills that I have are focused on what I do, not what I’m studying. If my company finally does decide to move to 6.x, then I can study for and receive my VCP on that version. Not before.

Companies love to make sure their evangelists and resellers are all on the latest version of their certifications because they see certifications as a sales tool. People certified in a technology will pick that solution over any others because they are familiar with it. Likewise, the sales process benefits from knowledgable sales people that understand the details behind your solution. It’s a win-win for both sides.

What this picture really ignores is the fact that a larger number of non-reseller professionals are actually using the certification as a study guide to support their organization. Perhaps they get certified as a way to get better support terms or a quicker response to support calls. Maybe they just learned so much about the product along the way that they want to show off what they’ve been doing. No matter what the reason, it’s very true that these folks are not in a sales role. They’re the support team keeping the lights on.

Support doesn’t care about upgrading at the drop of a hat. Instead, they are focused on keeping the existing stuff running as long as possible. Keeping users happy. Keeping executives happy. Keeping people from asking questions about availability or services. That’s not something that looks good on a bill of materials. But it’s what we all expect. Likewise, support isn’t focused on new things if the old things keep running. Certification, for them, is more about proving you know something instead of proving you can sell something.


Tom’s Take

I’ve had so many certifications that I don’t even remember them all. I got some of them because we needed it to sell a solution to a customer. I got others to prove I knew some esoteric command in a forgotten platform. But, no matter what else came up, I was certified on that platform. Windows 2000, NetWare 6.x, you name it. I was certified on that collection of software. I never rushed to get my certification upgraded because I knew what the reality of things really was. I got certified to keep the lights on for my customers. I got certified to help the people that believed in my skills. That’s the real value of a certification to me. Not sales. Just keeping things running another month.

The Magic of the CCIE

I stumbled across a great Reddit thread this week: Is the CCIE as impossible as it seems? There are a lot of great replies on that thread about people passing and the “good old days” of Banyan Vines, Appletalk, and more. It’s also a fascinating look into how the rest of the networking industry sees exams like the CCIE and JNCIE. Because those of us that have the numbers seem to be magicians to some.

Sleight of CLI Hand

Have you ever seen the cups and balls magic trick? Here’s an excellent example of it from the recently departed Ricky Jay:

Impressive, right? It’s amazing to behold a master craftsman at work. Every time I watch that video I’m amazed. I know he’s doing sleight of hand. But I can’t catch it. Now, watch this same video but with annotations turned on. SPOILER ALERT – The annotations will tell you EXACTLY where the tricks are done:

Is it more impressive now that you know how the tricks are done? Check out this demonstration from Penn and Teller that shows you exactly how they do the tricks as well:

Okay, so it’s a little less mystifying now that you’ve seen how all the sleight of hand happens. But it’s still impressive because, as a professional, you can appreciate how the execute their tradecraft. Knowing that it’s not magic doesn’t mean it’s not an impressive feat. It must means you appreciate something different about the performance.

Let’s apply that to the CCIE. When you’re just starting out in networking, every piece of knowledge is new. Everything you learn is something you didn’t know before. Subnet masks, routing tables, and even just addressing an interface are new skills that you acquire and try to understand. It’s like learning how to take a coin from someone’s ear. It’s simple but it provides the building blocks for future tricks.

When you reach the level of studying for the CCIE lab, it does look like a daunting task. If you’ve followed Cisco’s guidelines you probably have your CCNP or equivalent knowledge. However, there is still a lot you don’t know. If you don’t believe that, go pick up Jeff Doyle’s Routing TCP/IP Volume 1 book. That book taught me I still had a lot to learn about networking.

But, as I slogged through the CCIE, I realized that I was acquiring skills. Just like the magicians that practice the cups and balls every day to get it right, I was picking up the ability to address interfaces quickly and see potential routing loops before I made them like I did in my first lab attempt. Each thing I learned and practiced not only made me a better engineer but also made the CCIE seem less like a mountain and more like a hill that could be climbed.

And I truly realized this when I was thumbing through a copy of the CCIE Official Exam guide. Someone had given me a copy to take a look at and I was happy with the depth of knowledge that I found. I wanted to pass it along to another junior engineer because, as I said to myself, “If only I had this book when I started! I could have skipped over all those other books!”

Practice, Practice, Practice!

That’s where I went wrong. Because I jumped right to the end goal instead of realizing the process. Magicians don’t start out making the Statue of Liberty disappear. They start out pulling coins from your ear and finding your card in a deck. They build their basic skills and then move on to harder things. But they most grand tricks in the magician’s top hat all still use the basic skills: sleight of hand, misdirection, and preparation. To neglect those is to court folly on stage.

CCIEs are no different. Every person that asks me about the test asks “How hard is it to pass?” I usually respond with something like “Not hard if you study.” Some of the people I talk to pick up on the “not hard” part and get crushed by the lab their first time out. They even end up with a $1,500 soda for their efforts. The other people, the ones that focus on “study” in my answer, they are the people who pass on the first attempt or the ones that get it right pretty quickly thereafter.

The CCIE isn’t a test. It’s a course in studying. It’s the culmination of teaching yourself the minutia of protocols and how they interact. The exam itself is almost perfunctory. It tests specific combinations of things you might see in the real world. And if you ask any CCIE, the real world is often ten time stranger than the lab. But the lab makes you think about the things you’ve already learned in new ways and apply that knowledge to find ways to solve problems. The lab isn’t hard because it’s easy. The lab becomes easier when you practice enough to not think the knowledge is hard any longer. I think Bruce Lee said it best:

I fear not the man who has practiced 10,000 kicks once, but I fear the man who has practiced one kick 10,000 times.

Most people would agree that Bruce Lee was one of the best martial artists of all times. And even he practiced until his fingers bled and he body was exhausted. Because he knew that being the best wasn’t about passing an exam for a belt or about showing off for people. It was about knowing what you needed to know and practicing it until it was second nature.


Tom’s Take

The CCIE has a certain magical aura for sure. But it’s not magical in and of itself. It’s a test designed to ensure that the people that pass know their skills at a deep level. It’s a test designed to make you look deeper at a problem and exhaust all your options before throwing in the towel. The CCIE isn’t impossible any more than sawing someone in half is impossible. It’s all about how your practice and prepare for the show that makes the trick seem impressive.

Why Is The CCIE Lab Moving?

Cisco confirmed big CCIE rumor this week that the RTP lab was going to be moved to Richardson, TX.

The language Cisco used is pretty neutral. San Jose and RTP are being shut down as full time lab locations and everyone is moving to Richardson. We knew about this thanks to the detective work of Jeff Fry, who managed to figure this out over a week ago. Now that we know what is happening, why is it coming to pass?

They Don’t Build Them Like They Used To

Real estate is expensive. Anyone that’s ever bought a house will tell you that. Now, imagine that on a commercial scale. Many companies will get the minimum amount of building that they need to get by. Sometimes they’re bursting at the seams before they upgrade to a new facility.

Other companies are big about having lots of area. These are the companies that have giant campuses. Companies like Cisco, Dell EMC, Intel, and NetApp have multiple buildings spread across a wide area. It makes sense to do this when you’re a large company that needs the room to spread out. In Cisco’s case, each business unit had their own real estate. Wireless was in one building. Firewalls in another. Each part of the company had their own area to play in.

Cisco was a real estate maven for a while. They built out in anticipation of business. There was a story years ago of a buried concrete slab foundation in Richardson that was just waiting for the next big Cisco product to be developed so they could clear away the dirt and start construction. But, why not just build the building and be done with it?

Remember how I said that real estate is expensive? That expense doesn’t come completely from purchases. It comes from operations. You need to have utilities for the building. You need to have services for the building. You need to pay taxes on the building. And those things happen all the time. Even if you never have anyone in the building the electricity is still running. That’s one of the reasons why Cisco shuts down their offices between Christmas and New Year’s every year. And the taxes are still due. Hence the reason why the foundation in Richardson was buried.

Real estate is also not an infinite resource. Anyone that’s been to Silicon Valley knows that. They’re running out of room in the South Bay. And building the new 49ers stadium on the corner of Tasman Drive and Great America Parkway didn’t help either. Sports teams are as hungry for real estate as tech companies. The support structures that cropped up for the stadium ended up buying the Letter Buildings from Cisco, which is why the lab was moved from Building C to Building L years ago.

Home Is Where The Work Is

The other shifting demographic is that more workers are remote in today’s environment. A combination of factors have led people to be just as productive from their home office as their open-plan cubicle. Increased collaboration software coupled with changing job requirements means that people don’t have to go to their desk every day to be productive.

This is especially true now that companies like Cisco are putting more of a focus on software instead of hardware. In the good old days of hardware dominance you needed to go into the office to work on your chipset diagrams. You needed your desktop CAD program to draw the silicon traces on a switch. And you needed to visit the assembly lines and warehouses to see that everything was in order.

Today? It’s all code. Everything is written in an IDE and stored on a powerful laptop. You can work from anywhere. A green space outside your office window. A coffee shop. Your living room. The possibilities are endless. But that also means that you don’t need a permanent office desk. And if you don’t need a desk that means your company doesn’t need to pay for you to have one.

Now, instead of bustling buildings full of people working in their shared offices there are acres of empty open-plan cubicle farms lying fallow. People would rather work from Starbucks than go to the office. People would rather work in their pajamas than toil away in a cube. And so companies like Cisco are paying taxes and utilities for open spaces that don’t have anyone while the offices around the perimeter are filled with managers that are leading people that they don’t see.

CCIE Real Estate

But what does this all mean for the lab? Well, Cisco needs to downsize their big buildings in high-value real estate markets. They’re selling off buildings in San Jose as fast as the NFL will buy them. They are downsizing the workforce in RTP as well. The first hint of the CCIE move was David Blair trying to find a new job. As real estate becomes more and more costly to obtain, Cisco is going to need to expand in less expensive markets. The Dallas/Fort Worth (DFW) area is still one of the cheapest in the country.

DFW is also right in the middle of the country. It’s pretty much the same distance from everything. So people that don’t want to schedule a mobile lab can fly to Richardson and take the test there. RTP and San Jose are being transitioned to mobile lab facilities, which means people that live close to those areas can still take the test, just not on the schedule they may like. This allows Cisco to free up the space in those buildings for other purposes and consolidate their workforce down to areas that require less maintenance. They can also sell off unneeded buildings to other companies and take the profits for reinvestment in other places. Cutting costs and making money is what real estate is all about, even if you aren’t a real estate developer.


Tom’s Take

I’m sad to see the labs moving out of RTP and San Jose. Cisco has said they are going to frame the famous Wall of Pain in RTP as a tribute to the lab takers there. I have some fond memories of San Jose as well, but even those memories are from a building that Cisco doesn’t own any longer. The new reality of a software defined Cisco is that there isn’t as much of a need for real estate any more. People want to work remotely and not live in a cube farm. And when people don’t want an office, you don’t need to keep paying for them to have one. Cisco won’t be shutting everything down any time soon, but the CCIE labs are just the first part of a bigger strategy.

Editor’s Note: An earlier version of this post accidentally referred to David Mallory instead of David Blair. This error has been corrected.

How High Can The CCIE Go?

Congratulations to Michael Wong, CCIE #60064! And yes, you’re reading that right. Cisco has certified 30,000 new CCIEs in the last nine years. The next big milestone for CCIE nerds will be 65,536, otherwise known as CCIE 0x10000. How did we get here? And what does this really mean for everyone in the networking industry?

A Short Disclaimer

Before we get started here, a short disclaimer. I am currently on the Cisco CCIE Advisory Board for 2018 and 2019. My opinions here do not reflect those of Cisco, only me. No insider information has been used in the crafting of this post. Any sources are freely available or represent my own opinions.

Ticket To Ride

Why the push for a certified workforce? It really does make sense when you look at it in perspective. More trained people means more people that know how to implement your system properly. More people implementing your systems means more people that will pick that solution over others when they’re offered. And that means more sales. And hopefully also less support time spent by your organization based on the trained people doing the job right in the first place.

You can’t fault people for wanting to show off their training programs. CWNP just announced at Wi-Fi Trek 2018 that they’ve certified CWNE #300, Robert Boardman (@Robb_404). Does that mean that any future CWNEs won’t know what they’re doing compared to the first one, Devin Akin? Or does it mean that CWNP has hit critical mass with their certification program and their 900-page tome of wireless knowledge? I’d like to believe it’s the latter.

You can’t fault Cisco for their successes in getting people certified. Just like Novell and Microsoft, Cisco wants everyone installing their products to be trained. Which would you rather deal with? A complete novice who has no idea how the command line works? Or someone competent that makes simple mistakes that cause issues down the road? I know I’d rather deal with a semi-professional instead of a complete amateur.

The only way that we can get to a workforce that has pervasive knowledge of a particular type of technology is if the certification program expands. For everyone that claims they want to keep their numbers small you should have a bit of reflective doubt. Either they don’t want to spend the money to expand their program or they don’t have the ability to expand it. Because a rising tide lifts all boats. When everyone knows more about your solutions the entire community and industry benefit from that knowledge.

Tradition Is An Old Word

Another criticism of the CCIE today is that it doesn’t address the changing way we’re doing our jobs. Every month I hear people asking for a CCIE Automation or CCIE SDN or some thing like that. I also remember years ago hearing people clamoring for CCIE OnePK, so just take that with a grain of salt.

Why is the CCIE so slow to change? Think about it from the perspective of the people writing the test. It takes months to get single changes made to questions. it takes many, many months to get new topics added to the test via blueprints. And it could take at least two years (or more) to expand the number of topics tested by introducing a new track. So, why then would Cisco or any other company spend time introducing new and potentially controversial topics into one of their most venerable and traditional tests without vetting things thoroughly before finalizing them.

Cisco took some flak for introducing the CCIE Data Center with the Application Control Engine (ACE) module in version 1. Many critics felt that the solution was outdated and no one used it in real life. Yet it took a revision or two before it was finally removed. Imagine what would happen if something like that were to occur as someone was developing a new test.

Could you imagine the furor if Cisco had decided to build a CCIE OpenFlow exam? What would be tested? Which version would have been used? How will you test integration on non-Cisco devices? Which controller would you use? Why aren’t you testing on this esoteric feature in 1.1 that hasn’t officially been deprecated yet. Why don’t you just forget it because OpenFlow is a failure? I purposely picked a controversial topic to highlight how silly it would have been to build an OpenFlow test but feel free to attach that to the technology de jour, like IoT.


Tom’s Take

The CCIE is a bellwether. It changes when it needs to change. When the CCIE Voice became the CCIE Collaboration, it was an endorsement of the fact that the nature of communications was changing away from a focus on phones and more toward presence and other methods. When the CCIE Data Center was announced, Cisco formalized their plans to stay in the data center instead of selling a few servers and then exiting the market. The CCIE doesn’t change to suit the whims of everyone in the community that wants to wear a badge that’s shiny or has a buzzword on it. Just like the retired CCIE tracks like ISP Dial or Design, you don’t want to wear that yoke around your neck going into the future of technology.

I’m happy that Cisco has a force of CCIEs. I’m deeply honored to know quite a few of them going all the way back to Terry Slattery. I can tell you that every person that has earned their number has done so with the kind of study and intense concentration that is necessary to achieve this feat. Whether they get it through self-study, bootcamp practice, or good old fashioned work experience you can believe that, no matter what their number might be, they’re there because they want to be there.

Should We Build A Better BGP?

One story that seems to have flown under the radar this week with the Net Neutrality discussion being so dominant was the little hiccup with BGP on Wednesday. According to sources, sources inside AS39523 were able to redirect traffic from some major sites like Facebook, Google, and Microsoft through their network. Since the ISP in question is located inside Russia, there’s been quite a lot of conversation about the purpose of this misconfiguration. Is it simply an accident? Or is it a nefarious plot? Regardless of the intent, the fact that we live in 2017 and can cause massive portions of Internet traffic to be rerouted has many people worried.

Routing by Suggestion

BGP is the foundation of the modern Internet. It’s how routes are exchanged between every autonomous system (AS) and how traffic destined for your favorite cloud service or cat picture hosting provider gets to where it’s supposed to be going. BGP is the glue that makes the Internet work.

But BGP, for all of the greatness that it provides, is still very fallible. It’s prone to misconfiguration. Look no further than the Level 3 outage last month. Or the outage that Google caused in Japan in August. And those are just the top searches from Google. There have been a myriad of problems over the course of the past couple of decades. Some are benign. Some are more malicious. And in almost every case they were preventable.

BGP runs on the idea that people configuring it know what they’re doing. Much like RIP, the suggestion of a better route is enough to make BGP change the way that traffic flows between systems. You don’t have to be a evil mad genius to see this in action. Anyone that’s ever made a typo in their BGP border router configuration will tell you that if you make your system look like an attractive candidate for being a transit network, BGP is more than happy to pump a tidal wave of traffic through your network without regard for the consequences.

But why does it do that? Why does BGP act so stupid sometimes in comparison to OSPF and EIGRP? Well, take a look at the BGP path selection mechanism. CCIEs can probably recite this by heart. Things like Local Preference, Weight, and AS_PATH govern how BGP will install routes and change transit paths. Notice that these are all set by the user. There are not automatic conditions outside of the route’s origin. Unlike OSPF and EIGRP, there is no consideration for bandwidth or link delay. Why?

Well, the old Internet wasn’t incredibly reliable from the WAN side. You couldn’t guarantee that the path to the next AS was the “best” path. It may be an old serial link. It could have a lot of delay in the transit path. It could also be the only method of getting your traffic to the Internet. Rather than letting the routing protocol make arbitrary decisions about link quality the designers of BGP left it up to the person making the configuration. You can configure BGP to do whatever you want. And it will do what you tell it to do. And if you’ve ever taken the CCIE lab you know that you can make BGP do some very interesting things when you’re faced with a challenge.

BGP assumes a minimum level of competency to use correctly. The protocol doesn’t have any built in checks to avoid doing stupid things outside of the basics of not installing incorrect routes in the routing table. If you suddenly start announcing someone else’s AS with better metrics then the global BGP network is going to think you’re the better version of that AS and swing traffic your way. That may not be what you want. Given that most BGP outages or configurations of this type only last a couple of hours until the mistake is discovered, it’s safe to say that fat fingers cause big BGP problems.

Buttoning Down BGP

How do we fix this? Well, aside from making sure that anyone touching BGP knows exactly what they’re doing? Not much. Some Regional Internet Registrars (RIRs) require you to preconfigure new prefixes with them before they can be brought online. As mentioned in this Reddit thread, RIPE is pretty good about that. But some ISPs, especially ones in the US that work with ARIN, are less strict about that. And in some cases, they don’t even bring the pre-loaded prefixes online at the correct time. That can cause headaches when trying to figure out why your networks aren’t being announced even though your config is right.

Another person pointed out the Mutually Agreed Norms for Routing Security (MANRS). These look like some very good common sense things that we need to be doing to ensure that routing protocols are secure from hijacks and other issues. But, MANRS is still a manual setup that relies on the people implementing it to know what they’re doing.

Lastly, another option would be the Resource Public Key Infrastructure (RPKI) service that’s offered by ARIN. This services allows people that own IP Address space to specify which autonomous systems can originate their prefixes. In theory, this is an awesome idea that gives a lot of weight to trusting that only specific ASes are allowed to announce prefixes. In practice, it requires the use of PKI cryptographic infrastructure on your edge routers. And anyone that’s ever configured PKI on even simple devices knows how big of a pain that can be. Mixing PKI and BGP may be enough to drive people back to sniffing glue.


Tom’s Take

BGP works. It’s relatively simple and gets the job done. But it is far too trusting. It assumes that the people running the Internet are nerdy pioneers embarking on a journey of discovery and knowledge sharing. It doesn’t believe for one minute that bad people could be trying to do things to hijack traffic. Or, better still, that some operator fresh from getting his CCNP isn’t going to reroute Facebook traffic through a Cisco 2524 router in Iowa. BGP needs to get better. Or we need to make some changes to ensure that even if BGP still believes that the Internet is a utopia someone is behind it to ensure those rose colored glasses don’t cause it to walk into a bus.

Mythbusting the CCIE Continuing Education Program

It’s been about a month since the CCIE Continuing Education program was announced ahead of Cisco Live. There was a fair amount of discussion about it both on this blog as well as other places, like Jeff Fry’s post. Overall, the response has been positive. However, there are a few questions and ideas about the program that are simply not true. And no, this is not The Death Of The CCIE Program (just Google it). So, let’s take a look at this edition of Mythbusters for the CCIE CE program.

Myth #1: The CE Program Is Just A Way For Cisco To Sell More Training

This was a good one. The list of CE classes that was release at the beginning of the program included Cisco Live classes as well as Cisco Authorized training classes. Those were the only thing on the list as of right now. When some people saw the list, they jumped to the conclusion that the reason why the CE program exists is because Cisco wants to push their training courses. Let’s look at that.

Let’s say you want to start a global program that requires people to keep track of their training credits to turn them in for some kind of reward, whether it be money or credit for something else. Do you:

  1. Open the program for submissions of any kind and then hire a team to sort through them all to verify that they are legitmate
  2. Use a small list of verified submissions that can be audited at any time internally and are known to be of good quality based on existing metrics

I can only imagine that you would pick #2 every time. Remember that the CCIE CE program is barely a month old. It was announced so people could start taking advantage of it at Cisco Live. The list of classes included on the list was small on purpose. They were Cisco affiliated classes on purpose. The CCIE team can audit these classes easily with internal metrics. They can drop in on them and ensure the content is high quality and appropriate for learners. They can revoke classes deemed too easy or add advanced classes at any time.

The list of training classes looks the way it does because Cisco thinks that these are classes that CCIEs would learn from. They weren’t picked at random to get class sizes higher or to make more profit for Cisco. These classes are something that people would benefit from. And if you’re going to be taking the class anyway or are looking to take a class on a subject, wouldn’t you rather take one that you can get extra credit for?

Myth #2: The CCIE CE Program Was Designed to Sell More Cisco Live Conference Passes

Another chuckle-worthy conclusion about the CCIE CE program. People assumed that because Cisco Live courses were included in the acceptable courses for CE credits, Cisco must obviously be trying to push people to register for more Cisco Live courses, right?

It is true that the CCIE CE program was announced right before Cisco Live 2017. I personally think that was so the CCIEs attending the conference could get credit toward any classes they had booked already. Yes, the courses count. And yes, the longer 4-hour and 8-hour Techtorial classes count for more credits than the 1-hour sessions. But, there is a limit to how many classes count for credit at Cisco Live in total. And there is a cap of 70 credits per cycle on Cisco Live credits in total.

Even if Cisco wanted to use the CCIE CE program to push Cisco Live attendance, this isn’t the best way to do it. The Cisco Live option was to reward those that went anyway for things like advanced training classes and the CCIE NetVet lunch with the CEO. If Cisco wanted to make the CCIE dependent on Cisco Live, they could easily go back to the model of a specific conference just for CCIE recert as they did in the past. They could also just require a specific number of 3000-level classes be taken to recertify, again as in the past, instead of awarding points for other things like Techtorials. Thanks to Terry Slattery for helping me out with these last two points.

Additionally, tying CCIE CE credits to Cisco Live is a horrible way to push conference attendance. Most of the “cool stuff” happening at Cisco Live right now is happening in the DevNet Zone. Many people that I talked to ahead of the conference this year are strongly considering getting Explorer or Social passes next year and spending the whole time in the DevNet Zone instead of the conference proper. If Cisco wanted to push Cisco Live conference pass purchases, they would lock the DevNet Zone behind a more expensive pass.

Myth #3: There Are No Third Party CCIE CE Credits Because Cisco Hates Competition

This myth is currently a half truth. Yes, there are no third party CCIE CE options as of July 2017. Let’s go back to myth #1 and take a look at things. Why would Cisco open the program to the whole world and deal with all the hassle of auditing every potential source of CE credits just after launching the program? Sure, there are a lot of great providers out there. But, for every Narbik bootcamp there’s a bunch of shady stuff going on that isn’t on the up-and-up. But investigating the difference requires time and manpower, which aren’t easy to come by.

Ask yourself a simple question: Do you think Cisco will never have third party options? I can almost guarantee you the answer is no. Based on conversations I had with CCIE program people at Cisco Live this year, I would speculate that the CCIE CE program will expand in the future to encompass more training options, including third parties. I would bet the first inclusions will be certified trainers offering official courses. The next step will be auditing of classes for inclusion, like bootcamps and other semi-official classes. Expansion will be slow, but the classes that make the grade will help enhance the program.

What won’t be included? Youtube videos. Training webinars that are just cleverly disguised promotional pitches. Anything that is given without any way to track down the author and verify their knowledge level. And, as much as it pains me, I can almost guarantee that blog posts won’t count either. Cisco wants to be able to verify that you learned something and that you put in the effort. The only way to do that is through class attendance auditing and verification. Not through Youtube views or blog post counters.


Tom’s Take

For a program that’s less than a month old, there were a lot of people rushing to pass judgement on the hard work put into it. To pronounce the death of something that has endured for more than 20 years is a bit presumptuous. Is the current version of the CCIE CE program perfect? Nope. However, it’s better than the lack of a CE program we had three months ago. It’s also a work-in-progress that will only get better over time. It’s a program that Cisco is going to put significant investment into across the entire certification portfolio.

Rather than tearing down the hard work of so many people for the sake of ego stroking, let’s look at what was delivered and help the CCIE program managers build a bigger, better offering that helps us all in the long run. Cisco wants their CCIEs to succeed and go far in the networking world. And that’s no myth.