Monthly Archives: July 2015

The Score Is High. Who’s Holding On?

Posted on by
2

Checklist

If you haven’t had the chance to read Jeff Fry’s treatise on why the CCIE written should be dropped, do it now. He raises some very valid points about relevancy and continuing education and how the written exam is approaching irrelvancy as a prerequisite for lab candidates. I’d like to approach another aspect of this whole puzzle, namely the growing need to get that extra edge to pass the cut score.

Cuts Like A Knife

Every standardized IT test has a cut score, or the minimum necessary score required to pass. There is a surprising amount of work that goes into calculating a cut score for a standardized test. Too low and you end up with unqualified candidates being certified. Too high and you have a certification level that no one can attain.

The average cut score for a given exam level tends to rise as time goes on. This has a lot to do with the increasing depth of potential candidates as well as the growing average of scores from those candidates. Raising the score with each revision of the test guarantees you have the best possible group representing that certification. It’s like having your entire group be members of the honor roll.

A high cut score ensures that unqualified personnel are washed out of the program quickly. If you take a test with a cut score of 800 and you score a 500, you quickly know that you need to study quite a bit more before you’re ready to attempt the exam again. You might even say to yourself that you don’t know the material in any kind of depth to continue your quest for certification.

What happens if you’re just below the cut score? If you miss the mark by one question or less? How much more studying can you do? What else do you need to know? Sure, you can look at the exam and realize that there are many, many more questions you can answer correctly to hit the right score. But what if the passing score is absurdly high?

Horseshoes and Hand Grenades

I believe the largest consumer of purloined test questions is not the candidate that is completely clueless about a subject. Instead, the largest market of these types of services is the professional that has either missed the mark by a small margin or is afraid they will not pass even after hours of exhaustive study.

Rising cut scores lead to panic during exams. Why was a 790 good enough last time but now I need an 850 to pass? It’s easy to start worrying that your final score may fall in between that gray area that will leave lacking on the latest attempt. What happens if you miss the mark with all of the knowlege that you have obtained?

Those are the kinds of outcomes that drive people to invest in “test aids”. The lure is very compelling. Given the choice between failing an exam that costs $400 or spending a quarter of that to have a peek at what might be on the test, what is stopping the average test taker besides morality? What if your job depended on passing that exam? Now that multi-hundred dollar exam becomes a multi-thousand dollar decision.

Now we’re not talking about a particular candidate’s desire to fleece a potential employer or customer about knowledge. We’re talking about good old fashioned fear. Fear of failure. Fear of embarassement. Fear of losing your livelyhood because of a test. And that fear is what drives people to break the rules to ensure success.

Cut Us Some Slack

The solution to this issue is complicated. How can you ensure the integrity of a testing program? Worse yet, how can you stem the rising tide of improper behavior when it comes to testing?

The first thing is to realize what drives this behavior. Should a test like the CCIE written have higher and higher cut scores to eliminate illicit behavior? Is that really the issue here? Or is it more about the rising cut score itself causing a feedback loop that drives the behavior?

Companies need to take a hard look at their testing programs to understand what is going on with candidates. Are people missing the mark widely? Or are they coming very close without passing? Are the passing scores in the 99th percentile? Or barely above the mark? Adjustments in the cut score should happen both up and down.

It’s easy to look at testing groups and say, “If you just stuided a bit harder, you’d hit this impossibly high mark.” It’s also very easy to look at scores and say, “We see that many of you are missing the mark by less than ten points. Let’s lower that and see how things go from here.”

Certification programs are very worried about diluting the pool of certified candidates. But is having more members of the group with scores within a question or two of passing preferable to having a group with absurdly high passing scores thanks to outside help?

Tom’s Take

I’ve taken exams with a 100% cut score. It’s infuriating to think that even a single wrong answer could cost you an entire exam. It’s even worse if you are financing the cost of your exam on your own. Fear of missing that mark can drive people to do lots of crazy things.

I’m not going to say that companies like Cisco need to lower the cut scores of exams to unrealistically low levels. That would cheapen the certifications that people have already earned. What needs to happen is that Cisco and other certification bodies need to learn what they are trying to accomplish with their programs and adjust all the parameters of the tests to accomplish those goals.

Perhaps raising the cut scores to more than 900 points isn’t the answer. Maybe instead more complex questions or more hands-on simulations are required to better test the knowledge of the candidates. These are better solutions that take time and research. They aren’t the false panacea of raising the passing score. The rising tide can’t be fixed by making the buoys float just a little higher.

 

Objectivity Never Rests

Posted on by
1

objectivity

Being an independent part of the IT community isn’t an easy thing. There is a lot of writing involved and an even greater amount of research. For every word you commit to paper there is at least an hour of taking phone calls and interviewing leaders in the industry about topics. The rewards can be legion. So can the pitfalls. Objectivity is key, yet that is something where entire communities appear to be dividing.

Us Or Them

Communities are complex organisms with their own flow and feel. What works well in one community doesn’t work well in another. Familiarity with one concept doesn’t immediately translate to another. However, one thing that is universal across all communities is the polarization between extremes.

For instance, in the networking community this polarization is best characterized by the concept of “ABC – Anything But Cisco”. Companies make millions selling Cisco equipment every year. Writers and speakers can make a very healthy career from covering Cisco technologies. And yet there are a large number of companies and people that choose to use other options. They write about Juniper or install Brocade. They spend time researching Cumulus Linux or Big Switch Networks.

Knowing a little about many things is a great thing. There is no way I could have done my old VAR job had I only known Cisco gear. But when that specialization is taken to an extreme, you get the mentality that anything or anyone involved in the opposite camp must be wrong on principal. It does happne that some choose to ignore all other things at their own peril. Still others are branded as “haters” not because they truly hate a position but because others have taken comments and pushed them beyond their meaning to an extreme to serve as a comparison point.

Think a bit about the following situations that have been mentioned to me in recent months and look at what the perception is in certain communities:

  • Cisco vs. Not Cisco
  • Cisco vs. VMware
  • Cisco vs. Whitebox
  • VMware vs. OpenStack
  • VMware vs. Docker
  • EMC vs. Not EMC

The list could go on for many more entries. The point is that people have drawn “battle lines” in the industry around companies and concepts to provide contrast for positions.

Objectivity In Motion

How does the independent influencer cope with all these challenges to objectivity? It’s not unlike navigating a carpet full of Lego bricks with no shoes on.

The first important step is to avoid the trap of being pigeonholed as a “hater”. That’s easier than it sounds. Simply covering one technology or vendor isn’t going to cause you to fall into that trap. If someone writes a lot about Juniper, I simply assume they spend the majority of their time with Juniper gear. The only time they cross the line into the territory of anti-Someone is through calculated commentary to that effect.

The other important step in reference to the above is to keep your commentary on point. Petty comments like “that’s a stupid idea” or “no one in their right mind would do it like that” aren’t constructive and lead to labels of “hater”. The key to criticism is to keep it constructive. Why is it a stupid idea? Why would someone choose to do it differently? These are ways to provide contrast without relying on generalizing to get your point across.

The third and most important way to avoid losing objectivty is to keep the discussion focused on things and ideas and not people. As soon as you start attacking people and crticizing them your objectivity will always be called into question. For example, a few years ago I wrote a review of a short book that Greg Ferro (@EtherealMind) wrote about blogging. I disagreed with many of his points based on my own experiences. In my post, I never attacked Greg or called his blogging ability into question. Instead, I addressed his points and provided my own perspective. Greg and I have had many beers since then without wanting to choke each other, so I think we’re still friends. But more importantly, we’re still objective about blogging even though we have different opinions.

Tom’s Take

Objectivity is hard to gain and easy to lose. It’s also easy to have it taken from you by people that feel you’ve lost it. It wouldn’t be a stretch to look at my last blog post about Meraki and assume that I “hate” them based on my comments. But if you read through what I wrote, I never say that I hate the company or the people. Instead, I disagree with a choice they have made with their software. I still feel my objectivity is intact. If Meraki decides tomorrow to implement some of my ideas or something similar, I will be more than happy to tell everyone about it.

You can never stop looking at your own objectivity. When you get complacent you have lost. You need to constantly ask yourself why you are writing or speaking about something and how objective you are. If you are the first person to question your own objectivity it will be much easier to answer those that question it later.

Meraki Will Never Be A Large Enterprise Solution

Posted on by
29

Cisco-Cloud-Networking-Meraki

Thanks to a couple of recent conversations, I thought it was time to stir the wireless pot a little. First was my retweet of an excellent DNS workaround post from Justin Cohen (@CanTechIt). One of the responses I got from wireless luminary Andrew von Nagy (@RevolutionWifi):

http://twitter.com/revolutionwifi/status/618167906313076737

This echoed some of the comments that I heard from Sam Clements (@Samuel_Clements) and Blake Krone (@BlakeKrone) during this video from Cisco Live Milan in January:

During that video, you can hear Sam and Blake asking for a few features that aren’t really supported on Meraki just yet. And it all comes down to a simple issue.

Should It Just Work?

Meraki has had a very simple guiding philosophy since the very beginning. Things should be easy to configure and work without hassle for their customers. It’s something we see over and over again in technology. From Apple to Microsoft, the focus has shifted away from complexity and toward simplicity. Gone are the field of radio buttons and obscure text fields. In their place we find simple binary choics. “Do You Want To Do This Thing? YES/NO”.

Meraki believes that the more complicated configuration items confuse users and lead to support issues down the road. And in many ways they are absolutely right. If you’ve ever seen someone freeze up in front of a Coke Freestyle machine, you know how easy it is to be overwhelmed by the power of choice.

In a small business or small enterprise environment, you just need things to work. A business without a dedicated IT department doesn’t need to spend hours figuring out how to disable 802.11b data rates to increase performance. That SMB/SME market has historically been the one that Meraki sells into better than anyone else. The times are changing though.

Exceptions Are Rules?

Meraki’s acquistion by Cisco has raised their profile and provided a huge new sales force to bring their hardware and software to the masses. The software in particular is a tipping point for a lot of medium and large enterprises. Meraki makes it easy to configure and manage large access point deployments. And nine times out of ten their user interface provides everything a person could need for configuration.

Notice that was “nine times out of ten”. In an SME, that one time out of ten that something more was needed could happen once or twice in the lifetime of a deployment. In a large enterprise, that one time out of ten could happen once a month or even once a week. With a huge number of clients accessing the system for long periods of time, the statistical probability that an advanced feature will need to be configured does approach certainty quickly.

Meraki doesn’t have a way to handle these exceptions currently. They have an excellent feature request system in their “Make A Wish” feedback system, but the tipping point required for a feature to be implemented in a new release doesn’t have a way to be weighted for impact. If two hundred people ask for a feature and the average number of access points in their networks is less than five, it reflects differently than if ten people ask for a feature with an average of one thousand access points per network. It is important to realize that enterprises can scale up rapidly and they should carry a heavier weight when feature requests come in.

That’s not to say that Meraki should go the same route as Cisco Unified Communications Manager (CUCM). Several years ago, I wrote about CSCsb42763 which is a bug ID that enables a feature by typing that code into an obscure text field. It does enable the feature, but you have no idea what or how or why. In fact, if it weren’t for Google or a random call to TAC, you’d never even know about the feature. This is most definitely not the way to enable advanced features.

Making It Work For Me

Okay, the criticism part is over. Now for the constructive part. Because complaining without offering a solution is just whining.

Meraki can fix their issues with large enterprises by offering a “super config mode” to users that have been trained. It’s actually not that far away from how they validate licenses today. If you are listed as an admin on the system and you have a Meraki Master ID under your profile then you get access to the extra config mode. This would benefit both enterprise admins as well as partners that have admin accounts on customer systems.

This would also be a boon for the Meraki training program. Sure, having another piece of paper is nice. But what if all that hard work actually paid off with better configuration access to the system? Less need to call support instead of just getting slightly better access to engineers? If you can give people what they need to fix my problem without calling for support they will line up outside your door to get it.

If Meraki isn’t willing to take that giant leap just yet, another solution would be to weight the “Make A Wish” suggestions based on the number of APs covered by the user. They might even do this now. But it would be nice to know as a large enterprise end user that my feature requests are being taken under more critical advisement than a few people with less than a dozen APs. Scale matters.

Tom’s Take

Yes, the headline is a bit of clickbait. I don’t think it would have had quite the same impact if I’d titled it “How Meraki Can Fix Their Enterprise Problems”. You, the gentle reader, would have looked at the article either way. But the people that need to see this wouldn’t have cared unless it looked like the sky was falling. So I beg your forgiveness for an indulgence to get things fixed for everyone.

I use Meraki gear at home. It works. I haven’t even configured even 10% of what it’s capable of doing. But there are times when I go looking for a feature that I’ve seen on other enterprise wireless systems that’s just not there. And I know that it’s not there on purpose. Meraki does a very good job reaching the customer base that they have targeted for years. But as Cisco starts pushing their solutions further up the stack and selling Meraki into bigger and more complex environments, Meraki needs to understand how important it is to give those large enterprise users more control over their systems. Or “It Just Works” will quickly become “It Doesn’t Work For Me”.