Certifications Are About Support

You may have seen this week that VMware has announced they are removing the mandatory recertification requirement for their certification program. This is a huge step from VMware. The VCP, VCAP, and VCDX are huge certifications in the virtualization and server industry. VMware has always wanted their partners and support personnel to be up-to-date on the latest and greatest software. But, as I will explain, the move to remove the mandatory recertification requirement says more about the fact that certifications are less about selling and more about supporting.

The Paper Escalator

Recertification is a big money maker for companies. Sure, you’re spending a lot money on things like tests and books. But those aren’t usually tied to the company offering the certification. Instead, the testing fees are given to the testing center, like Pearson, and the book fees go to the publisher.

The real money maker for companies is the first-party training. If the company developing the certification is also offering the training courses you can bet they’re raking in the cash. VMware has done this for years with the classroom requirement for the VCP. Cisco has also started doing in with their first-party CCIE training. Cisco’s example also shows how quality first-party content can drive out the third parties in the industry by not even suggesting to prospective candidates that this is another option to get their classroom materials.

I’ve railed against the VCP classroom requirement before. I think forcing your candidates to take an in-person class as a requirement for certification is silly and feels like it’s designed to make money and not make good engineers. Thankfully, VMware seems to agree with me in the latest release of info. They’re allowing the upgrade path to be used for their recertification process, which doesn’t necessarily require attendance in a classroom offering. I’d argue that it’s important to do so, especially if you’re really out of date with the training. But not needing it for certification is a really nice touch.

Keeping the Lights On

The other big shift with this certification change from VMware is the tacit acknowledgement that people aren’t in any kind of rush to upgrade their software right after the newest version is released. Ask any system administrator out there and they’ll tell you to wait for a service pack before you upgrade anything. System admins for VMware are as cautious as anyone, if not moreso. Too often, new software updates break existing functionality or cause issues that can’t be fixed without a huge time investment.

How is this affected by certification? Well, if I spent all my time learning VMware 5.x and I got my VCP on it because my company was using it you can better believe that my skill set is based around VCP5. If my company doesn’t decide to upgrade to 6.x or even 7.x for several years, my VCP is still based on 5.x technology. It shouldn’t expire just because I never upgraded to 6.x. The skills that I have are focused on what I do, not what I’m studying. If my company finally does decide to move to 6.x, then I can study for and receive my VCP on that version. Not before.

Companies love to make sure their evangelists and resellers are all on the latest version of their certifications because they see certifications as a sales tool. People certified in a technology will pick that solution over any others because they are familiar with it. Likewise, the sales process benefits from knowledgable sales people that understand the details behind your solution. It’s a win-win for both sides.

What this picture really ignores is the fact that a larger number of non-reseller professionals are actually using the certification as a study guide to support their organization. Perhaps they get certified as a way to get better support terms or a quicker response to support calls. Maybe they just learned so much about the product along the way that they want to show off what they’ve been doing. No matter what the reason, it’s very true that these folks are not in a sales role. They’re the support team keeping the lights on.

Support doesn’t care about upgrading at the drop of a hat. Instead, they are focused on keeping the existing stuff running as long as possible. Keeping users happy. Keeping executives happy. Keeping people from asking questions about availability or services. That’s not something that looks good on a bill of materials. But it’s what we all expect. Likewise, support isn’t focused on new things if the old things keep running. Certification, for them, is more about proving you know something instead of proving you can sell something.


Tom’s Take

I’ve had so many certifications that I don’t even remember them all. I got some of them because we needed it to sell a solution to a customer. I got others to prove I knew some esoteric command in a forgotten platform. But, no matter what else came up, I was certified on that platform. Windows 2000, NetWare 6.x, you name it. I was certified on that collection of software. I never rushed to get my certification upgraded because I knew what the reality of things really was. I got certified to keep the lights on for my customers. I got certified to help the people that believed in my skills. That’s the real value of a certification to me. Not sales. Just keeping things running another month.

The Magic of the CCIE

I stumbled across a great Reddit thread this week: Is the CCIE as impossible as it seems? There are a lot of great replies on that thread about people passing and the “good old days” of Banyan Vines, Appletalk, and more. It’s also a fascinating look into how the rest of the networking industry sees exams like the CCIE and JNCIE. Because those of us that have the numbers seem to be magicians to some.

Sleight of CLI Hand

Have you ever seen the cups and balls magic trick? Here’s an excellent example of it from the recently departed Ricky Jay:

Impressive, right? It’s amazing to behold a master craftsman at work. Every time I watch that video I’m amazed. I know he’s doing sleight of hand. But I can’t catch it. Now, watch this same video but with annotations turned on. SPOILER ALERT – The annotations will tell you EXACTLY where the tricks are done:

Is it more impressive now that you know how the tricks are done? Check out this demonstration from Penn and Teller that shows you exactly how they do the tricks as well:

Okay, so it’s a little less mystifying now that you’ve seen how all the sleight of hand happens. But it’s still impressive because, as a professional, you can appreciate how the execute their tradecraft. Knowing that it’s not magic doesn’t mean it’s not an impressive feat. It must means you appreciate something different about the performance.

Let’s apply that to the CCIE. When you’re just starting out in networking, every piece of knowledge is new. Everything you learn is something you didn’t know before. Subnet masks, routing tables, and even just addressing an interface are new skills that you acquire and try to understand. It’s like learning how to take a coin from someone’s ear. It’s simple but it provides the building blocks for future tricks.

When you reach the level of studying for the CCIE lab, it does look like a daunting task. If you’ve followed Cisco’s guidelines you probably have your CCNP or equivalent knowledge. However, there is still a lot you don’t know. If you don’t believe that, go pick up Jeff Doyle’s Routing TCP/IP Volume 1 book. That book taught me I still had a lot to learn about networking.

But, as I slogged through the CCIE, I realized that I was acquiring skills. Just like the magicians that practice the cups and balls every day to get it right, I was picking up the ability to address interfaces quickly and see potential routing loops before I made them like I did in my first lab attempt. Each thing I learned and practiced not only made me a better engineer but also made the CCIE seem less like a mountain and more like a hill that could be climbed.

And I truly realized this when I was thumbing through a copy of the CCIE Official Exam guide. Someone had given me a copy to take a look at and I was happy with the depth of knowledge that I found. I wanted to pass it along to another junior engineer because, as I said to myself, “If only I had this book when I started! I could have skipped over all those other books!”

Practice, Practice, Practice!

That’s where I went wrong. Because I jumped right to the end goal instead of realizing the process. Magicians don’t start out making the Statue of Liberty disappear. They start out pulling coins from your ear and finding your card in a deck. They build their basic skills and then move on to harder things. But they most grand tricks in the magician’s top hat all still use the basic skills: sleight of hand, misdirection, and preparation. To neglect those is to court folly on stage.

CCIEs are no different. Every person that asks me about the test asks “How hard is it to pass?” I usually respond with something like “Not hard if you study.” Some of the people I talk to pick up on the “not hard” part and get crushed by the lab their first time out. They even end up with a $1,500 soda for their efforts. The other people, the ones that focus on “study” in my answer, they are the people who pass on the first attempt or the ones that get it right pretty quickly thereafter.

The CCIE isn’t a test. It’s a course in studying. It’s the culmination of teaching yourself the minutia of protocols and how they interact. The exam itself is almost perfunctory. It tests specific combinations of things you might see in the real world. And if you ask any CCIE, the real world is often ten time stranger than the lab. But the lab makes you think about the things you’ve already learned in new ways and apply that knowledge to find ways to solve problems. The lab isn’t hard because it’s easy. The lab becomes easier when you practice enough to not think the knowledge is hard any longer. I think Bruce Lee said it best:

I fear not the man who has practiced 10,000 kicks once, but I fear the man who has practiced one kick 10,000 times.

Most people would agree that Bruce Lee was one of the best martial artists of all times. And even he practiced until his fingers bled and he body was exhausted. Because he knew that being the best wasn’t about passing an exam for a belt or about showing off for people. It was about knowing what you needed to know and practicing it until it was second nature.


Tom’s Take

The CCIE has a certain magical aura for sure. But it’s not magical in and of itself. It’s a test designed to ensure that the people that pass know their skills at a deep level. It’s a test designed to make you look deeper at a problem and exhaust all your options before throwing in the towel. The CCIE isn’t impossible any more than sawing someone in half is impossible. It’s all about how your practice and prepare for the show that makes the trick seem impressive.

Why Is The CCIE Lab Moving?

Cisco confirmed big CCIE rumor this week that the RTP lab was going to be moved to Richardson, TX.

The language Cisco used is pretty neutral. San Jose and RTP are being shut down as full time lab locations and everyone is moving to Richardson. We knew about this thanks to the detective work of Jeff Fry, who managed to figure this out over a week ago. Now that we know what is happening, why is it coming to pass?

They Don’t Build Them Like They Used To

Real estate is expensive. Anyone that’s ever bought a house will tell you that. Now, imagine that on a commercial scale. Many companies will get the minimum amount of building that they need to get by. Sometimes they’re bursting at the seams before they upgrade to a new facility.

Other companies are big about having lots of area. These are the companies that have giant campuses. Companies like Cisco, Dell EMC, Intel, and NetApp have multiple buildings spread across a wide area. It makes sense to do this when you’re a large company that needs the room to spread out. In Cisco’s case, each business unit had their own real estate. Wireless was in one building. Firewalls in another. Each part of the company had their own area to play in.

Cisco was a real estate maven for a while. They built out in anticipation of business. There was a story years ago of a buried concrete slab foundation in Richardson that was just waiting for the next big Cisco product to be developed so they could clear away the dirt and start construction. But, why not just build the building and be done with it?

Remember how I said that real estate is expensive? That expense doesn’t come completely from purchases. It comes from operations. You need to have utilities for the building. You need to have services for the building. You need to pay taxes on the building. And those things happen all the time. Even if you never have anyone in the building the electricity is still running. That’s one of the reasons why Cisco shuts down their offices between Christmas and New Year’s every year. And the taxes are still due. Hence the reason why the foundation in Richardson was buried.

Real estate is also not an infinite resource. Anyone that’s been to Silicon Valley knows that. They’re running out of room in the South Bay. And building the new 49ers stadium on the corner of Tasman Drive and Great America Parkway didn’t help either. Sports teams are as hungry for real estate as tech companies. The support structures that cropped up for the stadium ended up buying the Letter Buildings from Cisco, which is why the lab was moved from Building C to Building L years ago.

Home Is Where The Work Is

The other shifting demographic is that more workers are remote in today’s environment. A combination of factors have led people to be just as productive from their home office as their open-plan cubicle. Increased collaboration software coupled with changing job requirements means that people don’t have to go to their desk every day to be productive.

This is especially true now that companies like Cisco are putting more of a focus on software instead of hardware. In the good old days of hardware dominance you needed to go into the office to work on your chipset diagrams. You needed your desktop CAD program to draw the silicon traces on a switch. And you needed to visit the assembly lines and warehouses to see that everything was in order.

Today? It’s all code. Everything is written in an IDE and stored on a powerful laptop. You can work from anywhere. A green space outside your office window. A coffee shop. Your living room. The possibilities are endless. But that also means that you don’t need a permanent office desk. And if you don’t need a desk that means your company doesn’t need to pay for you to have one.

Now, instead of bustling buildings full of people working in their shared offices there are acres of empty open-plan cubicle farms lying fallow. People would rather work from Starbucks than go to the office. People would rather work in their pajamas than toil away in a cube. And so companies like Cisco are paying taxes and utilities for open spaces that don’t have anyone while the offices around the perimeter are filled with managers that are leading people that they don’t see.

CCIE Real Estate

But what does this all mean for the lab? Well, Cisco needs to downsize their big buildings in high-value real estate markets. They’re selling off buildings in San Jose as fast as the NFL will buy them. They are downsizing the workforce in RTP as well. The first hint of the CCIE move was David Blair trying to find a new job. As real estate becomes more and more costly to obtain, Cisco is going to need to expand in less expensive markets. The Dallas/Fort Worth (DFW) area is still one of the cheapest in the country.

DFW is also right in the middle of the country. It’s pretty much the same distance from everything. So people that don’t want to schedule a mobile lab can fly to Richardson and take the test there. RTP and San Jose are being transitioned to mobile lab facilities, which means people that live close to those areas can still take the test, just not on the schedule they may like. This allows Cisco to free up the space in those buildings for other purposes and consolidate their workforce down to areas that require less maintenance. They can also sell off unneeded buildings to other companies and take the profits for reinvestment in other places. Cutting costs and making money is what real estate is all about, even if you aren’t a real estate developer.


Tom’s Take

I’m sad to see the labs moving out of RTP and San Jose. Cisco has said they are going to frame the famous Wall of Pain in RTP as a tribute to the lab takers there. I have some fond memories of San Jose as well, but even those memories are from a building that Cisco doesn’t own any longer. The new reality of a software defined Cisco is that there isn’t as much of a need for real estate any more. People want to work remotely and not live in a cube farm. And when people don’t want an office, you don’t need to keep paying for them to have one. Cisco won’t be shutting everything down any time soon, but the CCIE labs are just the first part of a bigger strategy.

Editor’s Note: An earlier version of this post accidentally referred to David Mallory instead of David Blair. This error has been corrected.

How High Can The CCIE Go?

Congratulations to Michael Wong, CCIE #60064! And yes, you’re reading that right. Cisco has certified 30,000 new CCIEs in the last nine years. The next big milestone for CCIE nerds will be 65,536, otherwise known as CCIE 0x10000. How did we get here? And what does this really mean for everyone in the networking industry?

A Short Disclaimer

Before we get started here, a short disclaimer. I am currently on the Cisco CCIE Advisory Board for 2018 and 2019. My opinions here do not reflect those of Cisco, only me. No insider information has been used in the crafting of this post. Any sources are freely available or represent my own opinions.

Ticket To Ride

Why the push for a certified workforce? It really does make sense when you look at it in perspective. More trained people means more people that know how to implement your system properly. More people implementing your systems means more people that will pick that solution over others when they’re offered. And that means more sales. And hopefully also less support time spent by your organization based on the trained people doing the job right in the first place.

You can’t fault people for wanting to show off their training programs. CWNP just announced at Wi-Fi Trek 2018 that they’ve certified CWNE #300, Robert Boardman (@Robb_404). Does that mean that any future CWNEs won’t know what they’re doing compared to the first one, Devin Akin? Or does it mean that CWNP has hit critical mass with their certification program and their 900-page tome of wireless knowledge? I’d like to believe it’s the latter.

You can’t fault Cisco for their successes in getting people certified. Just like Novell and Microsoft, Cisco wants everyone installing their products to be trained. Which would you rather deal with? A complete novice who has no idea how the command line works? Or someone competent that makes simple mistakes that cause issues down the road? I know I’d rather deal with a semi-professional instead of a complete amateur.

The only way that we can get to a workforce that has pervasive knowledge of a particular type of technology is if the certification program expands. For everyone that claims they want to keep their numbers small you should have a bit of reflective doubt. Either they don’t want to spend the money to expand their program or they don’t have the ability to expand it. Because a rising tide lifts all boats. When everyone knows more about your solutions the entire community and industry benefit from that knowledge.

Tradition Is An Old Word

Another criticism of the CCIE today is that it doesn’t address the changing way we’re doing our jobs. Every month I hear people asking for a CCIE Automation or CCIE SDN or some thing like that. I also remember years ago hearing people clamoring for CCIE OnePK, so just take that with a grain of salt.

Why is the CCIE so slow to change? Think about it from the perspective of the people writing the test. It takes months to get single changes made to questions. it takes many, many months to get new topics added to the test via blueprints. And it could take at least two years (or more) to expand the number of topics tested by introducing a new track. So, why then would Cisco or any other company spend time introducing new and potentially controversial topics into one of their most venerable and traditional tests without vetting things thoroughly before finalizing them.

Cisco took some flak for introducing the CCIE Data Center with the Application Control Engine (ACE) module in version 1. Many critics felt that the solution was outdated and no one used it in real life. Yet it took a revision or two before it was finally removed. Imagine what would happen if something like that were to occur as someone was developing a new test.

Could you imagine the furor if Cisco had decided to build a CCIE OpenFlow exam? What would be tested? Which version would have been used? How will you test integration on non-Cisco devices? Which controller would you use? Why aren’t you testing on this esoteric feature in 1.1 that hasn’t officially been deprecated yet. Why don’t you just forget it because OpenFlow is a failure? I purposely picked a controversial topic to highlight how silly it would have been to build an OpenFlow test but feel free to attach that to the technology de jour, like IoT.


Tom’s Take

The CCIE is a bellwether. It changes when it needs to change. When the CCIE Voice became the CCIE Collaboration, it was an endorsement of the fact that the nature of communications was changing away from a focus on phones and more toward presence and other methods. When the CCIE Data Center was announced, Cisco formalized their plans to stay in the data center instead of selling a few servers and then exiting the market. The CCIE doesn’t change to suit the whims of everyone in the community that wants to wear a badge that’s shiny or has a buzzword on it. Just like the retired CCIE tracks like ISP Dial or Design, you don’t want to wear that yoke around your neck going into the future of technology.

I’m happy that Cisco has a force of CCIEs. I’m deeply honored to know quite a few of them going all the way back to Terry Slattery. I can tell you that every person that has earned their number has done so with the kind of study and intense concentration that is necessary to achieve this feat. Whether they get it through self-study, bootcamp practice, or good old fashioned work experience you can believe that, no matter what their number might be, they’re there because they want to be there.

Should We Build A Better BGP?

One story that seems to have flown under the radar this week with the Net Neutrality discussion being so dominant was the little hiccup with BGP on Wednesday. According to sources, sources inside AS39523 were able to redirect traffic from some major sites like Facebook, Google, and Microsoft through their network. Since the ISP in question is located inside Russia, there’s been quite a lot of conversation about the purpose of this misconfiguration. Is it simply an accident? Or is it a nefarious plot? Regardless of the intent, the fact that we live in 2017 and can cause massive portions of Internet traffic to be rerouted has many people worried.

Routing by Suggestion

BGP is the foundation of the modern Internet. It’s how routes are exchanged between every autonomous system (AS) and how traffic destined for your favorite cloud service or cat picture hosting provider gets to where it’s supposed to be going. BGP is the glue that makes the Internet work.

But BGP, for all of the greatness that it provides, is still very fallible. It’s prone to misconfiguration. Look no further than the Level 3 outage last month. Or the outage that Google caused in Japan in August. And those are just the top searches from Google. There have been a myriad of problems over the course of the past couple of decades. Some are benign. Some are more malicious. And in almost every case they were preventable.

BGP runs on the idea that people configuring it know what they’re doing. Much like RIP, the suggestion of a better route is enough to make BGP change the way that traffic flows between systems. You don’t have to be a evil mad genius to see this in action. Anyone that’s ever made a typo in their BGP border router configuration will tell you that if you make your system look like an attractive candidate for being a transit network, BGP is more than happy to pump a tidal wave of traffic through your network without regard for the consequences.

But why does it do that? Why does BGP act so stupid sometimes in comparison to OSPF and EIGRP? Well, take a look at the BGP path selection mechanism. CCIEs can probably recite this by heart. Things like Local Preference, Weight, and AS_PATH govern how BGP will install routes and change transit paths. Notice that these are all set by the user. There are not automatic conditions outside of the route’s origin. Unlike OSPF and EIGRP, there is no consideration for bandwidth or link delay. Why?

Well, the old Internet wasn’t incredibly reliable from the WAN side. You couldn’t guarantee that the path to the next AS was the “best” path. It may be an old serial link. It could have a lot of delay in the transit path. It could also be the only method of getting your traffic to the Internet. Rather than letting the routing protocol make arbitrary decisions about link quality the designers of BGP left it up to the person making the configuration. You can configure BGP to do whatever you want. And it will do what you tell it to do. And if you’ve ever taken the CCIE lab you know that you can make BGP do some very interesting things when you’re faced with a challenge.

BGP assumes a minimum level of competency to use correctly. The protocol doesn’t have any built in checks to avoid doing stupid things outside of the basics of not installing incorrect routes in the routing table. If you suddenly start announcing someone else’s AS with better metrics then the global BGP network is going to think you’re the better version of that AS and swing traffic your way. That may not be what you want. Given that most BGP outages or configurations of this type only last a couple of hours until the mistake is discovered, it’s safe to say that fat fingers cause big BGP problems.

Buttoning Down BGP

How do we fix this? Well, aside from making sure that anyone touching BGP knows exactly what they’re doing? Not much. Some Regional Internet Registrars (RIRs) require you to preconfigure new prefixes with them before they can be brought online. As mentioned in this Reddit thread, RIPE is pretty good about that. But some ISPs, especially ones in the US that work with ARIN, are less strict about that. And in some cases, they don’t even bring the pre-loaded prefixes online at the correct time. That can cause headaches when trying to figure out why your networks aren’t being announced even though your config is right.

Another person pointed out the Mutually Agreed Norms for Routing Security (MANRS). These look like some very good common sense things that we need to be doing to ensure that routing protocols are secure from hijacks and other issues. But, MANRS is still a manual setup that relies on the people implementing it to know what they’re doing.

Lastly, another option would be the Resource Public Key Infrastructure (RPKI) service that’s offered by ARIN. This services allows people that own IP Address space to specify which autonomous systems can originate their prefixes. In theory, this is an awesome idea that gives a lot of weight to trusting that only specific ASes are allowed to announce prefixes. In practice, it requires the use of PKI cryptographic infrastructure on your edge routers. And anyone that’s ever configured PKI on even simple devices knows how big of a pain that can be. Mixing PKI and BGP may be enough to drive people back to sniffing glue.


Tom’s Take

BGP works. It’s relatively simple and gets the job done. But it is far too trusting. It assumes that the people running the Internet are nerdy pioneers embarking on a journey of discovery and knowledge sharing. It doesn’t believe for one minute that bad people could be trying to do things to hijack traffic. Or, better still, that some operator fresh from getting his CCNP isn’t going to reroute Facebook traffic through a Cisco 2524 router in Iowa. BGP needs to get better. Or we need to make some changes to ensure that even if BGP still believes that the Internet is a utopia someone is behind it to ensure those rose colored glasses don’t cause it to walk into a bus.

Mythbusting the CCIE Continuing Education Program

It’s been about a month since the CCIE Continuing Education program was announced ahead of Cisco Live. There was a fair amount of discussion about it both on this blog as well as other places, like Jeff Fry’s post. Overall, the response has been positive. However, there are a few questions and ideas about the program that are simply not true. And no, this is not The Death Of The CCIE Program (just Google it). So, let’s take a look at this edition of Mythbusters for the CCIE CE program.

Myth : The CE Program Is Just A Way For Cisco To Sell More Training

This was a good one. The list of CE classes that was release at the beginning of the program included Cisco Live classes as well as Cisco Authorized training classes. Those were the only thing on the list as of right now. When some people saw the list, they jumped to the conclusion that the reason why the CE program exists is because Cisco wants to push their training courses. Let’s look at that.

Let’s say you want to start a global program that requires people to keep track of their training credits to turn them in for some kind of reward, whether it be money or credit for something else. Do you:

  1. Open the program for submissions of any kind and then hire a team to sort through them all to verify that they are legitmate
  2. Use a small list of verified submissions that can be audited at any time internally and are known to be of good quality based on existing metrics

I can only imagine that you would pick every time. Remember that the CCIE CE program is barely a month old. It was announced so people could start taking advantage of it at Cisco Live. The list of classes included on the list was small on purpose. They were Cisco affiliated classes on purpose. The CCIE team can audit these classes easily with internal metrics. They can drop in on them and ensure the content is high quality and appropriate for learners. They can revoke classes deemed too easy or add advanced classes at any time.

The list of training classes looks the way it does because Cisco thinks that these are classes that CCIEs would learn from. They weren’t picked at random to get class sizes higher or to make more profit for Cisco. These classes are something that people would benefit from. And if you’re going to be taking the class anyway or are looking to take a class on a subject, wouldn’t you rather take one that you can get extra credit for?

Myth : The CCIE CE Program Was Designed to Sell More Cisco Live Conference Passes

Another chuckle-worthy conclusion about the CCIE CE program. People assumed that because Cisco Live courses were included in the acceptable courses for CE credits, Cisco must obviously be trying to push people to register for more Cisco Live courses, right?

It is true that the CCIE CE program was announced right before Cisco Live 2017. I personally think that was so the CCIEs attending the conference could get credit toward any classes they had booked already. Yes, the courses count. And yes, the longer 4-hour and 8-hour Techtorial classes count for more credits than the 1-hour sessions. But, there is a limit to how many classes count for credit at Cisco Live in total. And there is a cap of 70 credits per cycle on Cisco Live credits in total.

Even if Cisco wanted to use the CCIE CE program to push Cisco Live attendance, this isn’t the best way to do it. The Cisco Live option was to reward those that went anyway for things like advanced training classes and the CCIE NetVet lunch with the CEO. If Cisco wanted to make the CCIE dependent on Cisco Live, they could easily go back to the model of a specific conference just for CCIE recert as they did in the past. They could also just require a specific number of 3000-level classes be taken to recertify, again as in the past, instead of awarding points for other things like Techtorials. Thanks to Terry Slattery for helping me out with these last two points.

Additionally, tying CCIE CE credits to Cisco Live is a horrible way to push conference attendance. Most of the “cool stuff” happening at Cisco Live right now is happening in the DevNet Zone. Many people that I talked to ahead of the conference this year are strongly considering getting Explorer or Social passes next year and spending the whole time in the DevNet Zone instead of the conference proper. If Cisco wanted to push Cisco Live conference pass purchases, they would lock the DevNet Zone behind a more expensive pass.

Myth #3: There Are No Third Party CCIE CE Credits Because Cisco Hates Competition

This myth is currently a half truth. Yes, there are no third party CCIE CE options as of July 2017. Let’s go back to myth and take a look at things. Why would Cisco open the program to the whole world and deal with all the hassle of auditing every potential source of CE credits just after launching the program? Sure, there are a lot of great providers out there. But, for every Narbik bootcamp there’s a bunch of shady stuff going on that isn’t on the up-and-up. But investigating the difference requires time and manpower, which aren’t easy to come by.

Ask yourself a simple question: Do you think Cisco will never have third party options? I can almost guarantee you the answer is no. Based on conversations I had with CCIE program people at Cisco Live this year, I would speculate that the CCIE CE program will expand in the future to encompass more training options, including third parties. I would bet the first inclusions will be certified trainers offering official courses. The next step will be auditing of classes for inclusion, like bootcamps and other semi-official classes. Expansion will be slow, but the classes that make the grade will help enhance the program.

What won’t be included? Youtube videos. Training webinars that are just cleverly disguised promotional pitches. Anything that is given without any way to track down the author and verify their knowledge level. And, as much as it pains me, I can almost guarantee that blog posts won’t count either. Cisco wants to be able to verify that you learned something and that you put in the effort. The only way to do that is through class attendance auditing and verification. Not through Youtube views or blog post counters.


Tom’s Take

For a program that’s less than a month old, there were a lot of people rushing to pass judgement on the hard work put into it. To pronounce the death of something that has endured for more than 20 years is a bit presumptuous. Is the current version of the CCIE CE program perfect? Nope. However, it’s better than the lack of a CE program we had three months ago. It’s also a work-in-progress that will only get better over time. It’s a program that Cisco is going to put significant investment into across the entire certification portfolio.

Rather than tearing down the hard work of so many people for the sake of ego stroking, let’s look at what was delivered and help the CCIE program managers build a bigger, better offering that helps us all in the long run. Cisco wants their CCIEs to succeed and go far in the networking world. And that’s no myth.

There Won’t Be A CCIE: SDN. Here’s Why

There’s a lot of work that’s been done recently to bring the CCIE up to modern network standards. Yusuf and his team are working hard to incorporate new concepts into the written exam. Candidates are broadening their horizons and picking up new ideas as they learn about industry stalwarts like OSPF and spanning tree. But the biggest challenge out there is incorporating the ideas behind software defined networking (SDN) into the exam. I don’t believe that this will ever happen. Here’s why.

Take This Broken Network

If you look at the CCIE and what it’s really testing, the exam is really about troubleshooting and existing network integration. The CCIE introduces and tests on concepts like link aggregation, routing protocol redistribution, and network service implementation. These are things that professionals are expected to do when they walk in the door, either as a consultant or as someone advising on the incorporation of a new network.

The CCIE doesn’t deal with the design of a network from the ground up. It doesn’t task someone with coming up with the implementation of a greenfield network from scratch. The CCIE exam, especially the lab component, only tests a candidate on their ability to work on something that has already exists. That’s been one of the biggest criticisms of the CCIE for a very long time. Since the knowledge level of a CCIE is at the highest level, they are often drafted to design networks rather than implementing them.

That’s the reason why the CCDE was created. CCDEs create networks from nothing. Their coursework focuses on taking requirements and making a network out of it. That’s why their practical exam focuses less on command lines and more on product knowledge and implementation details. The CCDE is where people that build networks prove they know their trade.

The Road You Must Design For

When you look at the concepts behind SDN, it’s not really built for troubleshooting and implementation without thought. Yes, automation does help implementation. Orchestration helps new devices configure themselves on the fly. API access allows us to pull all kinds of useful information out of the network for the purposes of troubleshooting and management. But each and every one of these things is not in the domain of the CCIE.

Can SDN solve the thorny issues behind redistributing EIGRP into OSPF? How about creating Multiple Spanning Tree instances for odd numbered VLANs? Will SDN finally help me figure out how to implement Frame Relay Traffic Shaping without screwing up the QoS policies? The answer to almost every one of these questions is no.

SDNs major advantages can only be realized with forethought and guidelines. Orchestration and automation make sense when implemented in pods or with new greenfield deployments. Once they have been tested and proven, these concepts can be spread across the entire network and used to ease design woes.

Does it make more sense to start using Ansible and Jinja at the beginning? Or halfway through a deployment? Would you prefer to create Python scripts to poll against APIs after you’ve implemented a different network monitoring system (NMS)? Or would it make more sense to do it right from the start?

CCIEs may see SDN in practice as they start using things like APIC-EM to roll out polices in the network, but CCDEs are the real SDN gatekeepers. They alone can make the decisions to incorporate these ideas from the very beginning to leverage capabilities to ease deployment and make troubleshooting easier. Even though CCIEs won’t see SDN, they will reap the benefits from it being baked in to everything they do.


Tom’s Take

Rather than asking when the CCIE is going to get SDN-ified, a better question would be “Should the CCIE worry?” The answer, as explained above, is no. SDN isn’t something that a CCIE needs to study for. CCDEs, on the other hand, will be hugely impacted by SDN and it will make a big difference to them in the long run. Rather than forcing CCIEs into a niche role that they aren’t necessarily suited for, we should instead let them do what they do best. We should incorporate SDN concepts into the CCDE and let them do what they do best and make the network a better place for CCIEs. Everyone will be better in the long run.

The Rising Tide of CCIE Written Costs

CCIELogo

In CCIE news this week, Cisco has raised the price of their exams across the board. The CCNA has moved up to $325, and the CCIE Written moves from $400 to $450. It goes without saying that there is quite a bit of outcry in the community. Why is the price of the CCIE Written exam surging so high?

No Such Thing As A Free Test

The most obvious answer is that the amount of work going in to development of the exam has increased. The number of people working behind the scenes to create a better exam has caused the amount of outlay to go up, hence the need to recover those costs. This is the simplest explanation of all the cost increases.

As Cisco pours more and more technology into the tests, the amount of hands and fingers touching them has gone down. At the same time, the quality of the eyeballs that do look at the exam has gone up. It’s a lot like going to a specialist doctor. The quality of the care you receive for your condition is high, but the costs associated with that doctor are higher than a regular general practice doctor. Cisco’s headcount is now focused on keeping exam quality high. That kind of expertise is always more expensive per capita, even if the number of those people is fewer.

The odd thing here is that even if the costs of the people doing the work are going up, the amount that the test is increasing doesn’t seem to correlate. It’s been less than two years since the formal introduction of the current version of the CCIE written exam at the then-unheard of price point of $400. We’re two and a half years removed from the CCIE 4.0 Written exam and it’s lofty $350 price point. Has the technology changed so much in less than three years?

The Great Barrier Test

Going back to the introduction of the 5.0 version of the CCIE Written, there was also a retake policy change introduced. Cisco wanted to create a “backoff timer” to reduce the amount of times that a person could take the exam before needing to wait. The change still allowed you to take the second attempt after 30 days, but then the third attempt must wait an additional 90 days after that. So, instead of being able to get three exam attempts in 60 days, those same three attempts would have taken 120 days.

This change was rolled back about six months ago due to outcry from the community. CCIEs trying to recertify were stymied by the exam and forced to wait longer and longer to pass it, with their certification hanging in the balance. With the increased timeouts and limit of four retakes per year, some long time CCIEs were in danger of exhausting their attempts and watching their certification slide away without any recourse to fix it.

Now, the increased price behind the CCIE Written could indeed be attributed to the increased overhead. But it could also be an attempt to keep people from rushing in to take the test every 30 days. Making a policy change to keep people out the exam is one way to do it. But making the exam financially painful to continually fail is another. If you’re willing to drop $1350 in three months to try and pass then you either have money to burn or you’re desperate to pass.

In addition, a higher exam fee would cause test takers to be absolutely certain of their knowledge level before attempting the exam. Creating an initial barrier to entry that will make people think twice before scheduling an exam on a whim does create a situation where the first-time pass rate will improve significantly. This will also help drive funding to certification materials and classes, as candidates will want to know that they will pass before stepping into a certification exam center.


Tom’s Take

I’d really like to think that Cisco is just trying to cover their overhead with the recent price increases. Everything goes up in price. Some things go up faster than others. But the conspiracy theorist in me wonders if Cisco isn’t trying to use the increased price of the exam to help raise the pass rates and discourage folks from rushing the test repeatedly to see the exam question pool. $450 is a tough pill to swallow even if you pass. I think we’re going to see a lot more people taking advantage of the free Cisco Live exam as well as the half price cert exams there. And I sincerely hope the rumored options for recertification take flight soon. Because I don’t know how ready I am to go all out to study when there’s that much money on the line.

Fixing The CCIE Written – A Follow Up

955951_28854808

I stirred up quite the hornet’s nest last week, didn’t I? I posted about how I thought the CCIE Routing and Switching Written Exam needed to be fixed. I got 75 favorites on Twitter and 40 retweets of my post, not to mention the countless people that shared it on a variety of forums and other sites. Since I was at Cisco Live, I had a lot of people coming up to me saying that they agreed with my views. I also had quite a few people that weren’t thrilled with my perspective. Thankfully, I had the chance to sit down with Yusuf Bhaiji, head of the CCIE program, and chat about things. I wanted to share some thoughts here.

Clarity Of Purpose

One of the biggest complaints that I’ve heard is that I was being “malicious” in my post with regards to the CCIE. I was also told that it was a case of “sour grapes” and even that the exam was as hard as it was on purpose because the CCIE is supposed to be hard. Mostly, I felt upset that people were under the impression that my post was designed to destroy, harm, or otherwise defame the CCIE in the eyes of the community. Let me state for the record what my position is:

I still believe the CCIE is the premier certification in networking. I’m happy to be a CCIE and love the program.

Why did I write the post? Not because I couldn’t pass the written. Not because I wanted people to tell me that I was wrong and being mean to them. I wrote the post because I saw a problem and wanted to address it. I felt that the comments being made by so many people that had recently taken the test needed to be collected and discussed. Sure, making light of these kinds of issues in a public forum won’t make people happy. But, as I said to the CCIE team, would you rather know about it or let it fester quietly?

Yusuf assured me that the CCIE program holds itself to the highest standards. All questions are evaluated by three subject matter experts (SMEs) for relevance and correctness before being included in the exam. If those three experts don’t sign off, the question doesn’t go in. There are also quite a few metrics built into the testing software that give the CCIE team feedback on questions and answer choices. Those programs can index all manner of statistics to figure out if questions are creating problems for candidates. Any given test can produce pages worth of valuable information for the people creating the test and trying to keep it relevant.

Another point that was brought up was the comment section on the exam. If you have any problem with a question, you need to fill out the comment form. Yes, I know that taking time out of the test to provide feedback can cause issues. It also interrupts your flow of answering questions. But if you even think for an instant that the question is unfair or misleading or incorrect, you have to leave a detailed comment to make sure the question is flagged properly for review. Which of the following comments means more to you?

  • Trivia question

or

  • This question tests on an obscure command and isn’t valid for a CCIE-level test.

I can promise I know which one is going to be evaluated more closely. And yes, every comment that has purpose is reviewed. The exam creators can print off every comment ever left on a question. The more detailed the comment, the more likely to trigger a review. So please make sure to leave a comment if you think there is a problem with the question.

Clarity Of Vision

Some of the conversations that I had during Cisco Live revolved around the relevance of the questions on the test to a CCIE candidate. Most of the people that I talked to were CCIEs already and using the test for recertification. A few came to me to talk about the relevance of the test questions to candidates that are qualifying for the lab.

While I’m not able to discuss any of the specific plans for the future of the program, I will say that there are ideas in place that could make this distinction matter less. Yusuf told me that the team will be releasing more details as soon as they are confirmed.

The most important point is that the issues that I have with the CCIE Written exam are fixable. I also believe that criticism without a suggestion solution is little more than whining. So I decided to put my money where my mouth is with regard to the CCIE written exam.

I volunteered to fix it.

I stepped up and offered my time as an SME to review the questions on the written exam for relevance, correctness, and grammar. That’s not a light undertaking. There are a ton of questions in the pool that need to be examined. So for every person that agreed with my post or told me that they thought the exam needed to be fixed, I’m putting you all on the spot as well.

It’s time for us as a community of CCIEs to do our part for the exam. Yusuf told me the easiest way to take part in the program is to visit the following URL:

http://www.cisco.com/go/certsme

Sign up for the SME program. Tell them that you want to help fix the CCIE. Maybe you only have to look at 5-10 questions. If the hundred or so people that agreed with me volunteered today, the entire test question pool could be analyzed in a matter of weeks. We could do our part to ensure that people taking the exam have the best possible test in front of them.

But I also challenge you to do more. Don’t just correct grammar or tell them they spelled “electricity” wrong in the question. Challenge them. Ask yourself if this is a question a CCIE candidate should know the answer to. There’s a chance that you could make a difference there. But you can’t do that unless you step up the plate.


Tom’s Take

I had at least ten people tell me that they would do whatever it took to fix the CCIE test last week after I talked to the CCIE cert team. They were excited and hopeful that the issues they saw with the test could be sorted out. I’ll admit that I stepped out on a pretty big limb here by doing this in public as opposed to over email or through official channels. And I do admit that I didn’t clarify my intent to build the program up as opposed to casting the whole exam team and process in a bad light.

Mea culpa.

But, my motivation succeed in getting people to talk about the CCIE written. There are many of you that are ready to do your part to help. Please, go sign up at the link above to join the SME program. Maybe you’ll never look at a single question, Maybe you’ll look at fifty. The point is that you step up and tell Cisco that you’re willing. If even fifteen people come forward and agree to help then that message will sound loud and clear that each and every one of us is proud of being a CCIE and want the program to continue long past the time when we’re retired and telling our grandchildren about the good old days of hard but fair tests.

If you have any questions about participating in the program or you want to reach out to me with your thoughts, don’t hesitate to contact me. Let’s put the power of community behind this!

The CCIE Routing And Switching Written Exam Needs To Be Fixed

CCIELogo

The former logo listed in this post was removed by request of Cisco

I’m having a great time at Cisco Live this year talking to networking professionals about the state of things. Most are optimistic about where their jobs are going to fit in with networking and software and the new way of doing things. But there is an undercurrent of dissatisfaction with one of the most fundamental pieces of network training in the world. The discontent is palpable. From what I’ve heard around Las Vegas this week, it’s time to fix the CCIE Written Exam.

Whadda Ya Know?!?

The CCIE written is the bellwether of network training. It’s a chance for network engineers that use Cisco gear to prove they have what it takes to complete a difficult regimen of training to connect networks of impressive size. It’s also a rite of passage to show others that you know how to study, prep, and complete a difficult practical examination without losing your cool. But all that hard work starts with a written test.

The CCIE written has always been a tough test. It’s the only barrier to entry to the CCIE lab. Because the CCIE has never had prerequisites and likely never will due to long standing tradition, the only thing standing in the way of you ability to sit the grueling lab test is a 100 question multiple choice exam that gauges your ability to understand networking at a deep technical level.

But within the last year or so, the latest version of the CCIE written exam has begun to get very bad reviews from all takers of the test. There are quite a few people that have talked about how bad the test is for candidates. Unlike a lot of “sour grapes” cases of people railing against a test they failed, the feedback for the CCIE written is entirely different. It tends to fall into a couple of categories:

The Test Is Poorly Written

The most resounding critique of the exam is that it is a poorly constructed and executed test. The question quality is subpar. There are spelling mistakes throughout and test questions that have poor answer selections. Having spent a large amount of time helping construct the CCNA exam years ago, I can tell you that you will spend the bulk of your time creating wrong answers as distractors to the right ones. Guidelines say that a candidate should have no better than a 25% chance to guess the correct answer from all the choices. If you’ve ever taken a math test that has four multiple choice answers with three being correct for various mistakes in working the problem, you know just how insidious proper distractors can be (and math teachers too).

The CCIE written is riddled with bad distractors according to reports. It also has questions that don’t have a true proper answer or a set of answers that are all technically correct with no way to select them all. That frustrates test takers and makes it very difficult to study for the exam. The editing and test mechanics errors must be rectified quickly in order to restore confidence to the people taking the test.

The Test Doesn’t Cover The Material

Once people stop telling me how bad the test is constructed, they start telling me that the questions are bad on a conceptual level as well. No NDAs are violated during these discussions to protect everyone involved, but the general opinion is that the test has skewed in the wrong direction. Cisco seems to be creating a test that focuses more on the Cisco and less on the Internetworking part of the CCIE.

The test has never been confused for being a vendor-neutral exam. Any look at the blueprint will tell you that there a plenty of proprietary protocols and implementation methods there. But the older versions of the exam did do a good job of teaching you how to build a network that could behave itself with other non-Cisco sections. Redistributing EIGRP and OSPF is a prime example. But the focus of the new exam seems to be skewed toward very specific Cisco proprietary protocols and the minutia around how they operate. I’ve always thought that knowing the hello and dead timers of OSPF NBMA areas is a huge time sink and really only justified for test takers, but I also see why knowing that would be important in multi-vendor operations. But knowing the same thing for an EIGRP DMVPN seems a bit pointless.

The other problem is that, by the admission of most test takers, the current CCIE Written Exam study guide doesn’t cover the areas of the blueprint that are potentially on the test. I feel very sorry for my friend Narbik Kocharians here. He worked very hard to create a study guide that would help test takers pass the exam with the knowledge necessary to do well on the lab. And having a test over a completely different area than his guide makes him look bad in the eyes of testers without good cause. It’s like a college class when the professor tells you to study the book but gives you a test over his or her lectures. It’s not fair because you studied what you were told and failed because they tested something else.

CCIEs Feel There Are Better Recert Options

This is the most damaging problem in my mind. About half the test takers for the CCIE written are candidates looking to qualify for the lab. That requires them to take the written exam for their specific track. But the other half of the test takers are CCIEs that have passed the lab and are looking to recertify. For these professionals, any CCIE written exam is valid for recertification.

Many CCIE candidates look to broaden their horizons by moving to different track to keep their CCIE current while they study for service provider, data center, or even collaboration as a topic area of study. For them, the CCIE is a stepping stone to keep the learning process going. But many CCIEs I’ve spoken to in the past few months are starting to take other exams not because they want to learn new things, but because the CCIE Routing and Switch written exam is such a terrible test.

Quite a few CCIEs are using the CCDE written to recertify. They feel it is a better overall test even though it doesn’t test the material to the level that the CCIE R&S written exam does. They would even be willing to take the chance of getting a question on an area of technology that they know nothing about to avoid having to deal with poor questions in their areas of study. Still more CCIEs are choosing to become Emeritus and “retire” so as to avoid the pain of the written exam. While this has implications for partner status and a host of other challenges for practicing engineers, you have to wonder how bad things must be to make retirement of your CCIE number look like a better option.


Tom’s Take

I took the CCIE R&S written last year at Cisco Live. I was so disgusted with the exam that I immediately switched to the CCDE written and recertified my number while simultaneously vowing never to take the R&S written again. From what I’ve heard this year, the test quality is still slipping with no relief in sight. It’s a sad state of affairs when you realize that the flagship test for Cisco engineers is so horribly broken that those same engineers believe it can’t be fixed. They feel that all the comments and feedback in the world are ignored and their expertise in taking exams is pushed aside for higher cut scores and a more exclusive number of candidates. The dark side of it all is the hope that there isn’t an agenda to push official training materials or other kinds of shortcuts that would help candidates while charging them more and/or locking out third party training providers that work hard to help people study for the lab.

Cisco needs to fix this problem now. They need to listen to feedback and get their written problems under control. If they don’t, they may soon find the only people taking the R&S written test are the same kinds of dumpers and cheaters they think they are trying to keep out with a poorly constructed test.

NOTE: I have published an update to this post here: Fixing The CCIE Written – A Follow Up