Iron Chef: Certification Edition

My friend Joshua Williams (@802DotMe) texted me today with a great quote that I wanted to share with you that made me think about certifications:

You’ve probably already thought through this extensively, and maybe even written about it, but after sitting through another 8 hour practical exam yesterday I’m more convinced than ever that expert level exams from technical companies are more analogous to a gimmicky Food Network TV show than real world application of technical acumen. They don’t care so much about my skill level as they do about what kind of meal I can prepare in 30 minutes using Tialapia, grapes, and Dr. Pepper syrup with my salt shaker taken away halfway through.

I laughed because it’s true. And then I thought about it more and realized he’s way more than right. We know for a fact that companies love to increase the level of challenge in their exams from novice to expert. It’s a way to weed out the people that aren’t committed to learning about something. However, as the questions and tasks get harder it becomes much more difficult to get a good sense of how candidates are going to perform.

Boiling Water Isn’t Hard?

When you look at something like the CCNA, they’re trying to make sure you know how networks actually work. The simulations and lab exercises are pretty basic. Can you configure RIP correctly? Do you know the command to enable a switch port? There isn’t a need to get crazy with it. Using Joshua’s analogy from above, it’s not unlike a show like Worst Cooks in America, where the basics are the challenge that needs to be overcome. Not everyone is a superstar chef. Sometimes getting the building blocks right is more than half the battle.

As you move up the ladder, the learning gets harder. You dive deep into protocols and see how technologies build on each other. You need to configure BGP, but you also need to have some kind of other IGP running to distribute the routes. You need to remember that this spice goes in while the dish is cooking and this other goes on at the end so the flavor isn’t destroyed. I would liken this to a “fun” challenge cooking show, where the expert Food Network Chef faces off against someone that isn’t in the food business at a high professional level. Maybe they run a diner or are a short-order cook in a hotel restaurant. They aren’t looking to create their own signature dish. They know enough to cook what tastes good. But ask them to make hollandaise sauce or make pufferfish sashimi and they’re out.

Which brings us to the highest level of learning. The expert certification tracks. These are the crowing achievements of a career. They are the level that you have to be at to prove you know the technology inside and out. How do you test that, exactly? Microsoft had a great way of doing it back in the day with some of the mastery programs. You went to Redmond and you spent a couple of months learning the technology with the people that wrote it. It was very similar to a doctor’s internship in a hospital. You did the work with people that knew what you needed to know. They corrected you and helped you grown your knowledge. Even though you were an expert you understood what needed to be done and how to get there. At the end you took an exam to cover what you had learned and you earned your mastery.

Other certification programs don’t do that. Instead, they try to trip you up with tricky scenarios and make you make mistakes if you’re not paying attention. This is the Iron Chef round. You know your stuff, eh? Face off against this hard challenge. And by the way, here’s your curveball: You have to use this crazy extra ingredient. A show like Chopped does this a lot too. You need to make a meal using chicken, soy sauce, and candy corn. Are they testing your ability to prepare food? Or trying to figure out how creative you can be with a set of constraints that don’t make sense?

Ala Config!

The theory behind this kind of challenge is sound on paper. You never know what you’re going to walk into and what you’ll be forced to fix. I’ve had some real interesting problems that I’ve needed to solve over my career. But in every crazy case I never had to deal with the kinds of constrained setups that you get in lab-based exams. Configure this protocol, but don’t use these options. Make this connection work this way using one of these options but know that picking the wrong one will wreck your configuration in about two hours. Make trout-flavored ice cream. You name it and it’s a huge challenge for no good reason.

In theory, this is a great way to challenge your experts. In practice, it’s silly because you’re putting up barriers they will never see. Worse yet, you force them to start looking for the crazy constraints that don’t exist. One of my favorites is the overarching constraint in the CCIE lab that you are not allowed to use a static route to anything unless explicitly allowed in the question. Why? Because static routes don’t scale? Because they create administrative overhead? Or is it because a single static route fixes the problem and doesn’t require you to spend an hour tagging routes when redistribution happens? Static routes cut the Gordian Knot in the lab. So they can’t be allowed. Because that would make things too easy.

Tom’s Take

We need to move away from trivia and Iron Chef-style certifications. Instead of making our people dependent on silly tricks or restricting them from specific tools in their kit, we need to ensure their knowledge is at the right level. You would never ask a chef to cook an entire meal and not be able to use a saucepan. Why would you take away things like static routes or access lists from a network engineer’s arsenal? Instead of crafting the perfect tricky scenario to trap your candidates, spend the time instead teaching them what they need to know. Because once someone learns that trout is a horrible ice cream flavor we all win.

Thanks to Josh Williams for this great post idea!

Security and Salt

One of the things I picked up during the quarantine is a new-found interest in cooking. I’ve been spending more time researching recipes and trying to understand how my previous efforts to be a four-star chef have fallen flat. Thankfully, practice does indeed make perfect. I’m slowly getting better , which is to say that my family will actually eat my cooking now instead of just deciding that pizza for the fourth night in a row is a good choice.

One of the things I learned as I went on was about salt. Sodium Chloride is a magical substance. Someone once told me that if you taste a dish and you know it needs something but you’re not quite sure what that something is, the answer is probably salt. It does a lot to tie flavors together. But it’s also a fickle substance. It has the power to make or break a dish in very small amounts. It can be the difference between perfection and disaster. As it turns out, it’s a lot like security too.

Too Much is Exactly Enough

Security and salt are alike in the first way because you need the right amount to make things work. You have to have a minimum amount of both to make something viable. If you don’t have enough salt in your dish you won’t be able to taste it. But you also won’t be able to pull the flavors in the dish together with it. So you have to work with a minimum. Whether its a dash or salt or a specific minimum security threshold, you have to have enough to matter otherwise it’s the same as not having it at all.

To The Salt Mines

Likewise, the opposite effect is also detrimental. If you need to have the minimum amount to be effective, the maximum amount of both salt and security is bad. We all know what happens when we put too much salt into a dish. You can’t eat it at all. While there are tricks to getting too much salt out of a dish they change the overall flavor profile of whatever you’re making. Even just a little too much salt is very apparent depending on the dish you’re trying to make. Likewise, too much security is a deterrent to getting actual work done. Restrictive controls get in the way of productivity and ultimately lead to people trying to work out solutions that don’t solve the problem but instead try to bypass the control.

Now you may be saying to yourself, “So, the secret is to add just the right amount of security, right?” And you would be correct. But what is the right amount? Well, it’s not unlike trying to measure salt by sight instead of using a measuring device. Have you ever seen a chef or TV host pour an amount of salt into their hands and say it needs “about that much”? Do you know how they know how much salt to add? It’s not rocket science. Instead, it’s the tried-and-true practice of practice. They know about how much salt a dish needs for a given cooking time or flavor profile. They may have even made the dish a few times in order to understand when it might need more or less salt. They know that starches need more salt and delicate foods need less. Most importantly, they measured how much salt they can hold in their cupped hand. So they know what a teaspoon and tablespoon of salt look like in their palm.

How is this like security? Most Infosec professionals know inherently how to make things more secure. Their experience and their training tell them how much security to add to a system to make it more secure without putting too much in place to impede the operations of the system. They know where to put an IPS to provide maximum coverage without creating too many false positives. And they can do that because they have the experience to know how to do it right without guessing. Because the odds are good they’ve done it wrong at least one time.

The last salty thing to remember is that even when you have the right amounts down to a science you’re still going to need to figure out how to make it perfect. Potato soup is a notoriously hard dish to season properly. As mentioned above, starchy foods tend to soak up salt. You can fix a salty dish by putting a piece of a potato in it to soak up the salt. But is also means that it’s super hard to get it right when everything in your dish soaks up salt. But the best chefs can get it right. Because they know where to start and they know to test the dish before they do more. They know they need to start from a safe setup and push out from there without ruining everything. They know that no exact amount is the same between two dishes and the only way to make sure it’s right is to test until you get it right. Then make notes so you know how to make it better the next time.

Tom’s Take

Salt is one of my downfalls. I tend to like things salty, so I put too much in when I taste things. It’s never too salty for me unless my mouth shrinks up like a desiccated dish. That’s why I also have to rely on my team at home to help me understand when something is just right for them so I don’t burn out their taste buds either. Security is the same. You need a team that understands everything from their own perspective so they can help you get it right all over. You can’t take salt out of a dish without a massive crutch. And you can’t really reduce too much security without causing issues like budge overruns or costly meetings to decide what to remove. It’s better to get your salt and your security right in the first place.