The Score Is High. Who’s Holding On?


Checklist

If you haven’t had the chance to read Jeff Fry’s treatise on why the CCIE written should be dropped, do it now. He raises some very valid points about relevancy and continuing education and how the written exam is approaching irrelvancy as a prerequisite for lab candidates. I’d like to approach another aspect of this whole puzzle, namely the growing need to get that extra edge to pass the cut score.

Cuts Like A Knife

Every standardized IT test has a cut score, or the minimum necessary score required to pass. There is a surprising amount of work that goes into calculating a cut score for a standardized test. Too low and you end up with unqualified candidates being certified. Too high and you have a certification level that no one can attain.

The average cut score for a given exam level tends to rise as time goes on. This has a lot to do with the increasing depth of potential candidates as well as the growing average of scores from those candidates. Raising the score with each revision of the test guarantees you have the best possible group representing that certification. It’s like having your entire group be members of the honor roll.

A high cut score ensures that unqualified personnel are washed out of the program quickly. If you take a test with a cut score of 800 and you score a 500, you quickly know that you need to study quite a bit more before you’re ready to attempt the exam again. You might even say to yourself that you don’t know the material in any kind of depth to continue your quest for certification.

What happens if you’re just below the cut score? If you miss the mark by one question or less? How much more studying can you do? What else do you need to know? Sure, you can look at the exam and realize that there are many, many more questions you can answer correctly to hit the right score. But what if the passing score is absurdly high?

Horseshoes and Hand Grenades

I believe the largest consumer of purloined test questions is not the candidate that is completely clueless about a subject. Instead, the largest market of these types of services is the professional that has either missed the mark by a small margin or is afraid they will not pass even after hours of exhaustive study.

Rising cut scores lead to panic during exams. Why was a 790 good enough last time but now I need an 850 to pass? It’s easy to start worrying that your final score may fall in between that gray area that will leave lacking on the latest attempt. What happens if you miss the mark with all of the knowlege that you have obtained?

Those are the kinds of outcomes that drive people to invest in “test aids”. The lure is very compelling. Given the choice between failing an exam that costs $400 or spending a quarter of that to have a peek at what might be on the test, what is stopping the average test taker besides morality? What if your job depended on passing that exam? Now that multi-hundred dollar exam becomes a multi-thousand dollar decision.

Now we’re not talking about a particular candidate’s desire to fleece a potential employer or customer about knowledge. We’re talking about good old fashioned fear. Fear of failure. Fear of embarassement. Fear of losing your livelyhood because of a test. And that fear is what drives people to break the rules to ensure success.

Cut Us Some Slack

The solution to this issue is complicated. How can you ensure the integrity of a testing program? Worse yet, how can you stem the rising tide of improper behavior when it comes to testing?

The first thing is to realize what drives this behavior. Should a test like the CCIE written have higher and higher cut scores to eliminate illicit behavior? Is that really the issue here? Or is it more about the rising cut score itself causing a feedback loop that drives the behavior?

Companies need to take a hard look at their testing programs to understand what is going on with candidates. Are people missing the mark widely? Or are they coming very close without passing? Are the passing scores in the 99th percentile? Or barely above the mark? Adjustments in the cut score should happen both up and down.

It’s easy to look at testing groups and say, “If you just stuided a bit harder, you’d hit this impossibly high mark.” It’s also very easy to look at scores and say, “We see that many of you are missing the mark by less than ten points. Let’s lower that and see how things go from here.”

Certification programs are very worried about diluting the pool of certified candidates. But is having more members of the group with scores within a question or two of passing preferable to having a group with absurdly high passing scores thanks to outside help?


Tom’s Take

I’ve taken exams with a 100% cut score. It’s infuriating to think that even a single wrong answer could cost you an entire exam. It’s even worse if you are financing the cost of your exam on your own. Fear of missing that mark can drive people to do lots of crazy things.

I’m not going to say that companies like Cisco need to lower the cut scores of exams to unrealistically low levels. That would cheapen the certifications that people have already earned. What needs to happen is that Cisco and other certification bodies need to learn what they are trying to accomplish with their programs and adjust all the parameters of the tests to accomplish those goals.

Perhaps raising the cut scores to more than 900 points isn’t the answer. Maybe instead more complex questions or more hands-on simulations are required to better test the knowledge of the candidates. These are better solutions that take time and research. They aren’t the false panacea of raising the passing score. The rising tide can’t be fixed by making the buoys float just a little higher.

 

Advertisements

2 thoughts on “The Score Is High. Who’s Holding On?

  1. I’ll offer one more reason for “test aids”: The questions themselves.

    The joke is that on the CCNA, instead of just asking “what direction does the sun rise in?” it is phrased something like “You are in San Francisco, facing a bridge. You feel something warm on your back. What is it?”

    The reality which inspired the joke is – to anyone whose taken standardized IT exams – self-evident. Many questions are assessing a relatively simple piece of knowledge. The test creator wants to phrase the question in such a way that it doesn’t give away the answer. But in doing so, they create a question so convoluted that only the test creator themselves (or the very lucky, or perhaps the slightly psychic) would understand which of the 4 equally-obscurely-phrased answers is the right one.

    In that scenario, the professional who has both experience AND has studied the material is incented to just find out what “their” correct answer is, to get the test over with and move on.

    Because on a Microsoft exam, we all know that the RIGHT answer was an $800 Cisco PIX, but the CORRECT answer was to install Windows 2003 running ISA

  2. Pingback: Reaction: Testing to the Cut Score (Certifications) - 'net work

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s