Studying for a big exam takes time and effort. I spent the better part of 3 years trying to get my CCIE with constant study and many, many attempts. And I was lucky that the CCIE Routing and Switching exam is offered 5 days a week across multiple sites in the world. But what happens when the rug gets pulled out from under your feet?

Not Appearing In This Testing Center

The Cisco Certified Design Expert (CCDE) is a very difficult exam. It takes all of the technical knowledge of the CCIE and bends it in a new direction. There are fun new twists like requirements determination, staged word problems, and whole new ways to make a practical design exam. Russ White made a monster of a thing all those years ago and the team that continues to build on the exam has set a pretty high bar for quality. So high, in fact, that gaining the coveted CCDE number with its unique styling is a huge deal for the majority of people I know that have it, even those with multiple CCIEs.

The CCDE is also only offered 3-4 times a year. The testing centers are specialized Pearson centers that can offer enhanced security. You don’t have to fly to RTP or San Jose for your exam, but you can’t exactly take it at the local community college either. It’s the kind of thing that you work toward and get ready for, not the kind of spur-of-the-moment sales test that you have to take today to certify on a new product.

So, what happens if the test gets canceled? Because that’s exactly what happened two weeks ago. All those signed up to take the May 17, 2017 edition of the exam were giving one week’s notice that the exam had been canceled. Confusion reigned everywhere. Why had Cisco done this? What was going to happen? Would it be rescheduled or would these candidates be forced to sit the August exam dates instead? What was going on? This was followed by lots of rumors and suggestions of impropriety.

According to people that should know, the February 2017 exam had a “statistically signification pass rate increase”. For those that don’t know, Cisco tracks the passing exam scores of all their tests. They have people trained in the art of building tests and setting a specific passing rate. And they keep an eye on it. When too many people start passing the exam, it’s a sign that the difficulty needs to be increased. When too few people are able to hit the mark, a decision needs to be made about reducing difficulty or examining why the pass rate is so low. Those are the kinds of decisions that are made every day when the data comes back about tests.

For larger exams like the CCDE and CCIE that have fewer test taking opportunities, the passing rate is a huge deal. If 50% of the people that take the CCIE pass, it doesn’t say very much for the “expert” status of the exam. Likewise, if only 0.5% of test takers pass the CCDE on a given attempt, it’s an indicator that the test is too hard or overly complex and less about skill demonstration and more about being “really, really tough”. The passing rate is a big deal to Cisco.

According to those reports, there was a huge spike in the passing rate for the February CCDE. Big enough to make Cisco shut down the May attempts to find out why. If you have a few percentage points variance in passing or failing rates now and then, it’s not a huge deal. But if you have a huge increase in the number of certified individuals for one of the most challenging exams ever created, you need to find out why. That’s why Cisco went into damage control mode for May and maybe even for August.

The Jenga Problem

So, if your still with me at this point, you’ve probably figured out that there’s a good possibility that someone got their hands on a copy of the CCDE exam. That’s why Cisco had to stop the most recent date in order to ensure that whatever is out in the wild isn’t going to cause issues with the exam passing rates going forward. It makes sense from a test giver perspective to plug the leak and ensure the integrity of the exam.

Yes, it does suck for the people that are taking the exam. That’s a lot of time and effort wasted. One of the things I kept hearing from people was, “Why doesn’t Cisco just change the test?” Sadly, changes to the CCDE are almost impossible at this point in the game.

Even written exams with hundreds of test bank questions are difficult to edit and refresh. I know because I’ve been involved the process for many of them in the past. Getting new things added and old things removed takes time, effort, and cooperation among many people on a team. Imagine the nightmare of trying to get that done for a test where every question is hand-crafted and consists of multiple parts. Where something that is involved in Question 2 of a scenario has an impact on Question 8.

Remember, exams like the CCIE and CCDE are infamous for word choices mattering a lot in the decision making process of the candidate. Changing any of the words is a huge deal. Now, think about having to change a question or two. Or a scenario or two out of four. Or a whole exam revision. Now, do it all in two weeks. You can see what the CCDE team was faced with and why they had to make the decision they did. Paying customers are angry, but the possibility that the integrity of a complicated exam is compromised is worth more to Cisco in the long run.

Tom’s Take

What’s going to happen? It’s difficult to say. Cisco has to find out what happened and stop it immediately. Then, they have to assess what can be done to salvage the exam as it exists today. Scenarios must be created to replace known bad versions. Evaluation of those new scenarios could take a while. And in the interim, Cisco can’t just shut down the testing and certification process. It would be the end of the CCDE and cause a huge backlash in the elite community that exists to spread the good word of Cisco throughout networking circles. Cisco isn’t acting from a position of confusion, but instead from a position of caution. The wrong step at the wrong time could be disastrous.