My friend Joshua Williams (@802DotMe) texted me today with a great quote that I wanted to share with you that made me think about certifications:
You’ve probably already thought through this extensively, and maybe even written about it, but after sitting through another 8 hour practical exam yesterday I’m more convinced than ever that expert level exams from technical companies are more analogous to a gimmicky Food Network TV show than real world application of technical acumen. They don’t care so much about my skill level as they do about what kind of meal I can prepare in 30 minutes using Tialapia, grapes, and Dr. Pepper syrup with my salt shaker taken away halfway through.
I laughed because it’s true. And then I thought about it more and realized he’s way more than right. We know for a fact that companies love to increase the level of challenge in their exams from novice to expert. It’s a way to weed out the people that aren’t committed to learning about something. However, as the questions and tasks get harder it becomes much more difficult to get a good sense of how candidates are going to perform.
Boiling Water Isn’t Hard?
When you look at something like the CCNA, they’re trying to make sure you know how networks actually work. The simulations and lab exercises are pretty basic. Can you configure RIP correctly? Do you know the command to enable a switch port? There isn’t a need to get crazy with it. Using Joshua’s analogy from above, it’s not unlike a show like Worst Cooks in America, where the basics are the challenge that needs to be overcome. Not everyone is a superstar chef. Sometimes getting the building blocks right is more than half the battle.
As you move up the ladder, the learning gets harder. You dive deep into protocols and see how technologies build on each other. You need to configure BGP, but you also need to have some kind of other IGP running to distribute the routes. You need to remember that this spice goes in while the dish is cooking and this other goes on at the end so the flavor isn’t destroyed. I would liken this to a “fun” challenge cooking show, where the expert Food Network Chef faces off against someone that isn’t in the food business at a high professional level. Maybe they run a diner or are a short-order cook in a hotel restaurant. They aren’t looking to create their own signature dish. They know enough to cook what tastes good. But ask them to make hollandaise sauce or make pufferfish sashimi and they’re out.
Which brings us to the highest level of learning. The expert certification tracks. These are the crowing achievements of a career. They are the level that you have to be at to prove you know the technology inside and out. How do you test that, exactly? Microsoft had a great way of doing it back in the day with some of the mastery programs. You went to Redmond and you spent a couple of months learning the technology with the people that wrote it. It was very similar to a doctor’s internship in a hospital. You did the work with people that knew what you needed to know. They corrected you and helped you grown your knowledge. Even though you were an expert you understood what needed to be done and how to get there. At the end you took an exam to cover what you had learned and you earned your mastery.
Other certification programs don’t do that. Instead, they try to trip you up with tricky scenarios and make you make mistakes if you’re not paying attention. This is the Iron Chef round. You know your stuff, eh? Face off against this hard challenge. And by the way, here’s your curveball: You have to use this crazy extra ingredient. A show like Chopped does this a lot too. You need to make a meal using chicken, soy sauce, and candy corn. Are they testing your ability to prepare food? Or trying to figure out how creative you can be with a set of constraints that don’t make sense?
The theory behind this kind of challenge is sound on paper. You never know what you’re going to walk into and what you’ll be forced to fix. I’ve had some real interesting problems that I’ve needed to solve over my career. But in every crazy case I never had to deal with the kinds of constrained setups that you get in lab-based exams. Configure this protocol, but don’t use these options. Make this connection work this way using one of these options but know that picking the wrong one will wreck your configuration in about two hours. Make trout-flavored ice cream. You name it and it’s a huge challenge for no good reason.
In theory, this is a great way to challenge your experts. In practice, it’s silly because you’re putting up barriers they will never see. Worse yet, you force them to start looking for the crazy constraints that don’t exist. One of my favorites is the overarching constraint in the CCIE lab that you are not allowed to use a static route to anything unless explicitly allowed in the question. Why? Because static routes don’t scale? Because they create administrative overhead? Or is it because a single static route fixes the problem and doesn’t require you to spend an hour tagging routes when redistribution happens? Static routes cut the Gordian Knot in the lab. So they can’t be allowed. Because that would make things too easy.
We need to move away from trivia and Iron Chef-style certifications. Instead of making our people dependent on silly tricks or restricting them from specific tools in their kit, we need to ensure their knowledge is at the right level. You would never ask a chef to cook an entire meal and not be able to use a saucepan. Why would you take away things like static routes or access lists from a network engineer’s arsenal? Instead of crafting the perfect tricky scenario to trap your candidates, spend the time instead teaching them what they need to know. Because once someone learns that trout is a horrible ice cream flavor we all win.
Thanks to Josh Williams for this great post idea!
I think one expert level exam that might be somewhat of an exception to this is CCDE, where it’s more about understanding the requirements, the technology capabilities and constraints and then be able to navigate the different design options and caveats to get a design that works. It’s not perfect, but I think it mimics the reality of what many Network Architects experience in their real lives.
I believe you are right on that one. I also know that Russ White developed the certification that way specifically to avoid trivia questions and test design in a real-world manner. I would be interested to see what happens at the next revision of the exam.
Techs writing sloppy questions and lack of good review is another cert test challenge (and pain point for me). From experience, I do know how hard it is to write good questions. I once got hit with a question I wrote for a course, and had no idea what the right answer was of the five possible answers. I’ve had a lot of $VENDOR cert tests where the question was ambiguous or under-specified: true given some unstated assumptions, or true but only on firewalls not routers, that kind of thing. Not so good when 20% of the questions are like that! (I count them when testing: tick marks for how many did I have no clue on, how many were ambiguous or lacking context, etc.)
This problem also comes up in some online courseware from various vendors, and most recently, in mandatory HR and security training materials. Out of curiosity, recently in one I wrote down my answers, did quick retake until I guessed the right answer on the sticky question(s). In some cases, the desired answer was sort of right but kind of a bizarre choice, not what I would have chosen from the top 10 things relating to the question. Another failing there is asking questions where the answer was not actually in the course materials. Guessing time!
Way back in 1996 when I passed my CCIE Lab Exam it was much more practical than what I hear about today. We were told that all the configs were based on actual customer problems that Cisco TAC had received, and I’m inclined to believe that. There were very few artificial constraints, and we were allowed to bring external material into the lab with us since it was the way the real world would be (though there was little time to use such resources). As long as your config met the requirements of the exam task you got the points. Ah, the good ol’ days…
Totally agree, it’s a different world now than the mid 90’s too – knowing some of the obscure nerd knobs had some value when I couldn’t just Google what I was trying to do on my phone. Now that level of retention is worth less and being able to analyze requirements and apply common sense worth more. CCDE and VCDX are better at recognizing this, but panel based certs are very hard to scale.
My disappointment is the same as yours, Tom… I’m working on a new “anti-certification” now, if I can just get my butt in gear and get my team to energize … 🙂