Why Is The CCIE Lab Moving?

Cisco confirmed big CCIE rumor this week that the RTP lab was going to be moved to Richardson, TX.

The language Cisco used is pretty neutral. San Jose and RTP are being shut down as full time lab locations and everyone is moving to Richardson. We knew about this thanks to the detective work of Jeff Fry, who managed to figure this out over a week ago. Now that we know what is happening, why is it coming to pass?

They Don’t Build Them Like They Used To

Real estate is expensive. Anyone that’s ever bought a house will tell you that. Now, imagine that on a commercial scale. Many companies will get the minimum amount of building that they need to get by. Sometimes they’re bursting at the seams before they upgrade to a new facility.

Other companies are big about having lots of area. These are the companies that have giant campuses. Companies like Cisco, Dell EMC, Intel, and NetApp have multiple buildings spread across a wide area. It makes sense to do this when you’re a large company that needs the room to spread out. In Cisco’s case, each business unit had their own real estate. Wireless was in one building. Firewalls in another. Each part of the company had their own area to play in.

Cisco was a real estate maven for a while. They built out in anticipation of business. There was a story years ago of a buried concrete slab foundation in Richardson that was just waiting for the next big Cisco product to be developed so they could clear away the dirt and start construction. But, why not just build the building and be done with it?

Remember how I said that real estate is expensive? That expense doesn’t come completely from purchases. It comes from operations. You need to have utilities for the building. You need to have services for the building. You need to pay taxes on the building. And those things happen all the time. Even if you never have anyone in the building the electricity is still running. That’s one of the reasons why Cisco shuts down their offices between Christmas and New Year’s every year. And the taxes are still due. Hence the reason why the foundation in Richardson was buried.

Real estate is also not an infinite resource. Anyone that’s been to Silicon Valley knows that. They’re running out of room in the South Bay. And building the new 49ers stadium on the corner of Tasman Drive and Great America Parkway didn’t help either. Sports teams are as hungry for real estate as tech companies. The support structures that cropped up for the stadium ended up buying the Letter Buildings from Cisco, which is why the lab was moved from Building C to Building L years ago.

Home Is Where The Work Is

The other shifting demographic is that more workers are remote in today’s environment. A combination of factors have led people to be just as productive from their home office as their open-plan cubicle. Increased collaboration software coupled with changing job requirements means that people don’t have to go to their desk every day to be productive.

This is especially true now that companies like Cisco are putting more of a focus on software instead of hardware. In the good old days of hardware dominance you needed to go into the office to work on your chipset diagrams. You needed your desktop CAD program to draw the silicon traces on a switch. And you needed to visit the assembly lines and warehouses to see that everything was in order.

Today? It’s all code. Everything is written in an IDE and stored on a powerful laptop. You can work from anywhere. A green space outside your office window. A coffee shop. Your living room. The possibilities are endless. But that also means that you don’t need a permanent office desk. And if you don’t need a desk that means your company doesn’t need to pay for you to have one.

Now, instead of bustling buildings full of people working in their shared offices there are acres of empty open-plan cubicle farms lying fallow. People would rather work from Starbucks than go to the office. People would rather work in their pajamas than toil away in a cube. And so companies like Cisco are paying taxes and utilities for open spaces that don’t have anyone while the offices around the perimeter are filled with managers that are leading people that they don’t see.

CCIE Real Estate

But what does this all mean for the lab? Well, Cisco needs to downsize their big buildings in high-value real estate markets. They’re selling off buildings in San Jose as fast as the NFL will buy them. They are downsizing the workforce in RTP as well. The first hint of the CCIE move was David Blair trying to find a new job. As real estate becomes more and more costly to obtain, Cisco is going to need to expand in less expensive markets. The Dallas/Fort Worth (DFW) area is still one of the cheapest in the country.

DFW is also right in the middle of the country. It’s pretty much the same distance from everything. So people that don’t want to schedule a mobile lab can fly to Richardson and take the test there. RTP and San Jose are being transitioned to mobile lab facilities, which means people that live close to those areas can still take the test, just not on the schedule they may like. This allows Cisco to free up the space in those buildings for other purposes and consolidate their workforce down to areas that require less maintenance. They can also sell off unneeded buildings to other companies and take the profits for reinvestment in other places. Cutting costs and making money is what real estate is all about, even if you aren’t a real estate developer.


Tom’s Take

I’m sad to see the labs moving out of RTP and San Jose. Cisco has said they are going to frame the famous Wall of Pain in RTP as a tribute to the lab takers there. I have some fond memories of San Jose as well, but even those memories are from a building that Cisco doesn’t own any longer. The new reality of a software defined Cisco is that there isn’t as much of a need for real estate any more. People want to work remotely and not live in a cube farm. And when people don’t want an office, you don’t need to keep paying for them to have one. Cisco won’t be shutting everything down any time soon, but the CCIE labs are just the first part of a bigger strategy.

Editor’s Note: An earlier version of this post accidentally referred to David Mallory instead of David Blair. This error has been corrected.

How High Can The CCIE Go?

Congratulations to Michael Wong, CCIE #60064! And yes, you’re reading that right. Cisco has certified 30,000 new CCIEs in the last nine years. The next big milestone for CCIE nerds will be 65,536, otherwise known as CCIE 0x10000. How did we get here? And what does this really mean for everyone in the networking industry?

A Short Disclaimer

Before we get started here, a short disclaimer. I am currently on the Cisco CCIE Advisory Board for 2018 and 2019. My opinions here do not reflect those of Cisco, only me. No insider information has been used in the crafting of this post. Any sources are freely available or represent my own opinions.

Ticket To Ride

Why the push for a certified workforce? It really does make sense when you look at it in perspective. More trained people means more people that know how to implement your system properly. More people implementing your systems means more people that will pick that solution over others when they’re offered. And that means more sales. And hopefully also less support time spent by your organization based on the trained people doing the job right in the first place.

You can’t fault people for wanting to show off their training programs. CWNP just announced at Wi-Fi Trek 2018 that they’ve certified CWNE #300, Robert Boardman (@Robb_404). Does that mean that any future CWNEs won’t know what they’re doing compared to the first one, Devin Akin? Or does it mean that CWNP has hit critical mass with their certification program and their 900-page tome of wireless knowledge? I’d like to believe it’s the latter.

You can’t fault Cisco for their successes in getting people certified. Just like Novell and Microsoft, Cisco wants everyone installing their products to be trained. Which would you rather deal with? A complete novice who has no idea how the command line works? Or someone competent that makes simple mistakes that cause issues down the road? I know I’d rather deal with a semi-professional instead of a complete amateur.

The only way that we can get to a workforce that has pervasive knowledge of a particular type of technology is if the certification program expands. For everyone that claims they want to keep their numbers small you should have a bit of reflective doubt. Either they don’t want to spend the money to expand their program or they don’t have the ability to expand it. Because a rising tide lifts all boats. When everyone knows more about your solutions the entire community and industry benefit from that knowledge.

Tradition Is An Old Word

Another criticism of the CCIE today is that it doesn’t address the changing way we’re doing our jobs. Every month I hear people asking for a CCIE Automation or CCIE SDN or some thing like that. I also remember years ago hearing people clamoring for CCIE OnePK, so just take that with a grain of salt.

Why is the CCIE so slow to change? Think about it from the perspective of the people writing the test. It takes months to get single changes made to questions. it takes many, many months to get new topics added to the test via blueprints. And it could take at least two years (or more) to expand the number of topics tested by introducing a new track. So, why then would Cisco or any other company spend time introducing new and potentially controversial topics into one of their most venerable and traditional tests without vetting things thoroughly before finalizing them.

Cisco took some flak for introducing the CCIE Data Center with the Application Control Engine (ACE) module in version 1. Many critics felt that the solution was outdated and no one used it in real life. Yet it took a revision or two before it was finally removed. Imagine what would happen if something like that were to occur as someone was developing a new test.

Could you imagine the furor if Cisco had decided to build a CCIE OpenFlow exam? What would be tested? Which version would have been used? How will you test integration on non-Cisco devices? Which controller would you use? Why aren’t you testing on this esoteric feature in 1.1 that hasn’t officially been deprecated yet. Why don’t you just forget it because OpenFlow is a failure? I purposely picked a controversial topic to highlight how silly it would have been to build an OpenFlow test but feel free to attach that to the technology de jour, like IoT.


Tom’s Take

The CCIE is a bellwether. It changes when it needs to change. When the CCIE Voice became the CCIE Collaboration, it was an endorsement of the fact that the nature of communications was changing away from a focus on phones and more toward presence and other methods. When the CCIE Data Center was announced, Cisco formalized their plans to stay in the data center instead of selling a few servers and then exiting the market. The CCIE doesn’t change to suit the whims of everyone in the community that wants to wear a badge that’s shiny or has a buzzword on it. Just like the retired CCIE tracks like ISP Dial or Design, you don’t want to wear that yoke around your neck going into the future of technology.

I’m happy that Cisco has a force of CCIEs. I’m deeply honored to know quite a few of them going all the way back to Terry Slattery. I can tell you that every person that has earned their number has done so with the kind of study and intense concentration that is necessary to achieve this feat. Whether they get it through self-study, bootcamp practice, or good old fashioned work experience you can believe that, no matter what their number might be, they’re there because they want to be there.

Should We Build A Better BGP?

One story that seems to have flown under the radar this week with the Net Neutrality discussion being so dominant was the little hiccup with BGP on Wednesday. According to sources, sources inside AS39523 were able to redirect traffic from some major sites like Facebook, Google, and Microsoft through their network. Since the ISP in question is located inside Russia, there’s been quite a lot of conversation about the purpose of this misconfiguration. Is it simply an accident? Or is it a nefarious plot? Regardless of the intent, the fact that we live in 2017 and can cause massive portions of Internet traffic to be rerouted has many people worried.

Routing by Suggestion

BGP is the foundation of the modern Internet. It’s how routes are exchanged between every autonomous system (AS) and how traffic destined for your favorite cloud service or cat picture hosting provider gets to where it’s supposed to be going. BGP is the glue that makes the Internet work.

But BGP, for all of the greatness that it provides, is still very fallible. It’s prone to misconfiguration. Look no further than the Level 3 outage last month. Or the outage that Google caused in Japan in August. And those are just the top searches from Google. There have been a myriad of problems over the course of the past couple of decades. Some are benign. Some are more malicious. And in almost every case they were preventable.

BGP runs on the idea that people configuring it know what they’re doing. Much like RIP, the suggestion of a better route is enough to make BGP change the way that traffic flows between systems. You don’t have to be a evil mad genius to see this in action. Anyone that’s ever made a typo in their BGP border router configuration will tell you that if you make your system look like an attractive candidate for being a transit network, BGP is more than happy to pump a tidal wave of traffic through your network without regard for the consequences.

But why does it do that? Why does BGP act so stupid sometimes in comparison to OSPF and EIGRP? Well, take a look at the BGP path selection mechanism. CCIEs can probably recite this by heart. Things like Local Preference, Weight, and AS_PATH govern how BGP will install routes and change transit paths. Notice that these are all set by the user. There are not automatic conditions outside of the route’s origin. Unlike OSPF and EIGRP, there is no consideration for bandwidth or link delay. Why?

Well, the old Internet wasn’t incredibly reliable from the WAN side. You couldn’t guarantee that the path to the next AS was the “best” path. It may be an old serial link. It could have a lot of delay in the transit path. It could also be the only method of getting your traffic to the Internet. Rather than letting the routing protocol make arbitrary decisions about link quality the designers of BGP left it up to the person making the configuration. You can configure BGP to do whatever you want. And it will do what you tell it to do. And if you’ve ever taken the CCIE lab you know that you can make BGP do some very interesting things when you’re faced with a challenge.

BGP assumes a minimum level of competency to use correctly. The protocol doesn’t have any built in checks to avoid doing stupid things outside of the basics of not installing incorrect routes in the routing table. If you suddenly start announcing someone else’s AS with better metrics then the global BGP network is going to think you’re the better version of that AS and swing traffic your way. That may not be what you want. Given that most BGP outages or configurations of this type only last a couple of hours until the mistake is discovered, it’s safe to say that fat fingers cause big BGP problems.

Buttoning Down BGP

How do we fix this? Well, aside from making sure that anyone touching BGP knows exactly what they’re doing? Not much. Some Regional Internet Registrars (RIRs) require you to preconfigure new prefixes with them before they can be brought online. As mentioned in this Reddit thread, RIPE is pretty good about that. But some ISPs, especially ones in the US that work with ARIN, are less strict about that. And in some cases, they don’t even bring the pre-loaded prefixes online at the correct time. That can cause headaches when trying to figure out why your networks aren’t being announced even though your config is right.

Another person pointed out the Mutually Agreed Norms for Routing Security (MANRS). These look like some very good common sense things that we need to be doing to ensure that routing protocols are secure from hijacks and other issues. But, MANRS is still a manual setup that relies on the people implementing it to know what they’re doing.

Lastly, another option would be the Resource Public Key Infrastructure (RPKI) service that’s offered by ARIN. This services allows people that own IP Address space to specify which autonomous systems can originate their prefixes. In theory, this is an awesome idea that gives a lot of weight to trusting that only specific ASes are allowed to announce prefixes. In practice, it requires the use of PKI cryptographic infrastructure on your edge routers. And anyone that’s ever configured PKI on even simple devices knows how big of a pain that can be. Mixing PKI and BGP may be enough to drive people back to sniffing glue.


Tom’s Take

BGP works. It’s relatively simple and gets the job done. But it is far too trusting. It assumes that the people running the Internet are nerdy pioneers embarking on a journey of discovery and knowledge sharing. It doesn’t believe for one minute that bad people could be trying to do things to hijack traffic. Or, better still, that some operator fresh from getting his CCNP isn’t going to reroute Facebook traffic through a Cisco 2524 router in Iowa. BGP needs to get better. Or we need to make some changes to ensure that even if BGP still believes that the Internet is a utopia someone is behind it to ensure those rose colored glasses don’t cause it to walk into a bus.

Mythbusting the CCIE Continuing Education Program

It’s been about a month since the CCIE Continuing Education program was announced ahead of Cisco Live. There was a fair amount of discussion about it both on this blog as well as other places, like Jeff Fry’s post. Overall, the response has been positive. However, there are a few questions and ideas about the program that are simply not true. And no, this is not The Death Of The CCIE Program (just Google it). So, let’s take a look at this edition of Mythbusters for the CCIE CE program.

Myth #1: The CE Program Is Just A Way For Cisco To Sell More Training

This was a good one. The list of CE classes that was release at the beginning of the program included Cisco Live classes as well as Cisco Authorized training classes. Those were the only thing on the list as of right now. When some people saw the list, they jumped to the conclusion that the reason why the CE program exists is because Cisco wants to push their training courses. Let’s look at that.

Let’s say you want to start a global program that requires people to keep track of their training credits to turn them in for some kind of reward, whether it be money or credit for something else. Do you:

  1. Open the program for submissions of any kind and then hire a team to sort through them all to verify that they are legitmate
  2. Use a small list of verified submissions that can be audited at any time internally and are known to be of good quality based on existing metrics

I can only imagine that you would pick #2 every time. Remember that the CCIE CE program is barely a month old. It was announced so people could start taking advantage of it at Cisco Live. The list of classes included on the list was small on purpose. They were Cisco affiliated classes on purpose. The CCIE team can audit these classes easily with internal metrics. They can drop in on them and ensure the content is high quality and appropriate for learners. They can revoke classes deemed too easy or add advanced classes at any time.

The list of training classes looks the way it does because Cisco thinks that these are classes that CCIEs would learn from. They weren’t picked at random to get class sizes higher or to make more profit for Cisco. These classes are something that people would benefit from. And if you’re going to be taking the class anyway or are looking to take a class on a subject, wouldn’t you rather take one that you can get extra credit for?

Myth #2: The CCIE CE Program Was Designed to Sell More Cisco Live Conference Passes

Another chuckle-worthy conclusion about the CCIE CE program. People assumed that because Cisco Live courses were included in the acceptable courses for CE credits, Cisco must obviously be trying to push people to register for more Cisco Live courses, right?

It is true that the CCIE CE program was announced right before Cisco Live 2017. I personally think that was so the CCIEs attending the conference could get credit toward any classes they had booked already. Yes, the courses count. And yes, the longer 4-hour and 8-hour Techtorial classes count for more credits than the 1-hour sessions. But, there is a limit to how many classes count for credit at Cisco Live in total. And there is a cap of 70 credits per cycle on Cisco Live credits in total.

Even if Cisco wanted to use the CCIE CE program to push Cisco Live attendance, this isn’t the best way to do it. The Cisco Live option was to reward those that went anyway for things like advanced training classes and the CCIE NetVet lunch with the CEO. If Cisco wanted to make the CCIE dependent on Cisco Live, they could easily go back to the model of a specific conference just for CCIE recert as they did in the past. They could also just require a specific number of 3000-level classes be taken to recertify, again as in the past, instead of awarding points for other things like Techtorials. Thanks to Terry Slattery for helping me out with these last two points.

Additionally, tying CCIE CE credits to Cisco Live is a horrible way to push conference attendance. Most of the “cool stuff” happening at Cisco Live right now is happening in the DevNet Zone. Many people that I talked to ahead of the conference this year are strongly considering getting Explorer or Social passes next year and spending the whole time in the DevNet Zone instead of the conference proper. If Cisco wanted to push Cisco Live conference pass purchases, they would lock the DevNet Zone behind a more expensive pass.

Myth #3: There Are No Third Party CCIE CE Credits Because Cisco Hates Competition

This myth is currently a half truth. Yes, there are no third party CCIE CE options as of July 2017. Let’s go back to myth #1 and take a look at things. Why would Cisco open the program to the whole world and deal with all the hassle of auditing every potential source of CE credits just after launching the program? Sure, there are a lot of great providers out there. But, for every Narbik bootcamp there’s a bunch of shady stuff going on that isn’t on the up-and-up. But investigating the difference requires time and manpower, which aren’t easy to come by.

Ask yourself a simple question: Do you think Cisco will never have third party options? I can almost guarantee you the answer is no. Based on conversations I had with CCIE program people at Cisco Live this year, I would speculate that the CCIE CE program will expand in the future to encompass more training options, including third parties. I would bet the first inclusions will be certified trainers offering official courses. The next step will be auditing of classes for inclusion, like bootcamps and other semi-official classes. Expansion will be slow, but the classes that make the grade will help enhance the program.

What won’t be included? Youtube videos. Training webinars that are just cleverly disguised promotional pitches. Anything that is given without any way to track down the author and verify their knowledge level. And, as much as it pains me, I can almost guarantee that blog posts won’t count either. Cisco wants to be able to verify that you learned something and that you put in the effort. The only way to do that is through class attendance auditing and verification. Not through Youtube views or blog post counters.


Tom’s Take

For a program that’s less than a month old, there were a lot of people rushing to pass judgement on the hard work put into it. To pronounce the death of something that has endured for more than 20 years is a bit presumptuous. Is the current version of the CCIE CE program perfect? Nope. However, it’s better than the lack of a CE program we had three months ago. It’s also a work-in-progress that will only get better over time. It’s a program that Cisco is going to put significant investment into across the entire certification portfolio.

Rather than tearing down the hard work of so many people for the sake of ego stroking, let’s look at what was delivered and help the CCIE program managers build a bigger, better offering that helps us all in the long run. Cisco wants their CCIEs to succeed and go far in the networking world. And that’s no myth.

There Won’t Be A CCIE: SDN. Here’s Why

There’s a lot of work that’s been done recently to bring the CCIE up to modern network standards. Yusuf and his team are working hard to incorporate new concepts into the written exam. Candidates are broadening their horizons and picking up new ideas as they learn about industry stalwarts like OSPF and spanning tree. But the biggest challenge out there is incorporating the ideas behind software defined networking (SDN) into the exam. I don’t believe that this will ever happen. Here’s why.

Take This Broken Network

If you look at the CCIE and what it’s really testing, the exam is really about troubleshooting and existing network integration. The CCIE introduces and tests on concepts like link aggregation, routing protocol redistribution, and network service implementation. These are things that professionals are expected to do when they walk in the door, either as a consultant or as someone advising on the incorporation of a new network.

The CCIE doesn’t deal with the design of a network from the ground up. It doesn’t task someone with coming up with the implementation of a greenfield network from scratch. The CCIE exam, especially the lab component, only tests a candidate on their ability to work on something that has already exists. That’s been one of the biggest criticisms of the CCIE for a very long time. Since the knowledge level of a CCIE is at the highest level, they are often drafted to design networks rather than implementing them.

That’s the reason why the CCDE was created. CCDEs create networks from nothing. Their coursework focuses on taking requirements and making a network out of it. That’s why their practical exam focuses less on command lines and more on product knowledge and implementation details. The CCDE is where people that build networks prove they know their trade.

The Road You Must Design For

When you look at the concepts behind SDN, it’s not really built for troubleshooting and implementation without thought. Yes, automation does help implementation. Orchestration helps new devices configure themselves on the fly. API access allows us to pull all kinds of useful information out of the network for the purposes of troubleshooting and management. But each and every one of these things is not in the domain of the CCIE.

Can SDN solve the thorny issues behind redistributing EIGRP into OSPF? How about creating Multiple Spanning Tree instances for odd numbered VLANs? Will SDN finally help me figure out how to implement Frame Relay Traffic Shaping without screwing up the QoS policies? The answer to almost every one of these questions is no.

SDNs major advantages can only be realized with forethought and guidelines. Orchestration and automation make sense when implemented in pods or with new greenfield deployments. Once they have been tested and proven, these concepts can be spread across the entire network and used to ease design woes.

Does it make more sense to start using Ansible and Jinja at the beginning? Or halfway through a deployment? Would you prefer to create Python scripts to poll against APIs after you’ve implemented a different network monitoring system (NMS)? Or would it make more sense to do it right from the start?

CCIEs may see SDN in practice as they start using things like APIC-EM to roll out polices in the network, but CCDEs are the real SDN gatekeepers. They alone can make the decisions to incorporate these ideas from the very beginning to leverage capabilities to ease deployment and make troubleshooting easier. Even though CCIEs won’t see SDN, they will reap the benefits from it being baked in to everything they do.


Tom’s Take

Rather than asking when the CCIE is going to get SDN-ified, a better question would be “Should the CCIE worry?” The answer, as explained above, is no. SDN isn’t something that a CCIE needs to study for. CCDEs, on the other hand, will be hugely impacted by SDN and it will make a big difference to them in the long run. Rather than forcing CCIEs into a niche role that they aren’t necessarily suited for, we should instead let them do what they do best. We should incorporate SDN concepts into the CCDE and let them do what they do best and make the network a better place for CCIEs. Everyone will be better in the long run.

The Rising Tide of CCIE Written Costs

CCIELogo

In CCIE news this week, Cisco has raised the price of their exams across the board. The CCNA has moved up to $325, and the CCIE Written moves from $400 to $450. It goes without saying that there is quite a bit of outcry in the community. Why is the price of the CCIE Written exam surging so high?

No Such Thing As A Free Test

The most obvious answer is that the amount of work going in to development of the exam has increased. The number of people working behind the scenes to create a better exam has caused the amount of outlay to go up, hence the need to recover those costs. This is the simplest explanation of all the cost increases.

As Cisco pours more and more technology into the tests, the amount of hands and fingers touching them has gone down. At the same time, the quality of the eyeballs that do look at the exam has gone up. It’s a lot like going to a specialist doctor. The quality of the care you receive for your condition is high, but the costs associated with that doctor are higher than a regular general practice doctor. Cisco’s headcount is now focused on keeping exam quality high. That kind of expertise is always more expensive per capita, even if the number of those people is fewer.

The odd thing here is that even if the costs of the people doing the work are going up, the amount that the test is increasing doesn’t seem to correlate. It’s been less than two years since the formal introduction of the current version of the CCIE written exam at the then-unheard of price point of $400. We’re two and a half years removed from the CCIE 4.0 Written exam and it’s lofty $350 price point. Has the technology changed so much in less than three years?

The Great Barrier Test

Going back to the introduction of the 5.0 version of the CCIE Written, there was also a retake policy change introduced. Cisco wanted to create a “backoff timer” to reduce the amount of times that a person could take the exam before needing to wait. The change still allowed you to take the second attempt after 30 days, but then the third attempt must wait an additional 90 days after that. So, instead of being able to get three exam attempts in 60 days, those same three attempts would have taken 120 days.

This change was rolled back about six months ago due to outcry from the community. CCIEs trying to recertify were stymied by the exam and forced to wait longer and longer to pass it, with their certification hanging in the balance. With the increased timeouts and limit of four retakes per year, some long time CCIEs were in danger of exhausting their attempts and watching their certification slide away without any recourse to fix it.

Now, the increased price behind the CCIE Written could indeed be attributed to the increased overhead. But it could also be an attempt to keep people from rushing in to take the test every 30 days. Making a policy change to keep people out the exam is one way to do it. But making the exam financially painful to continually fail is another. If you’re willing to drop $1350 in three months to try and pass then you either have money to burn or you’re desperate to pass.

In addition, a higher exam fee would cause test takers to be absolutely certain of their knowledge level before attempting the exam. Creating an initial barrier to entry that will make people think twice before scheduling an exam on a whim does create a situation where the first-time pass rate will improve significantly. This will also help drive funding to certification materials and classes, as candidates will want to know that they will pass before stepping into a certification exam center.


Tom’s Take

I’d really like to think that Cisco is just trying to cover their overhead with the recent price increases. Everything goes up in price. Some things go up faster than others. But the conspiracy theorist in me wonders if Cisco isn’t trying to use the increased price of the exam to help raise the pass rates and discourage folks from rushing the test repeatedly to see the exam question pool. $450 is a tough pill to swallow even if you pass. I think we’re going to see a lot more people taking advantage of the free Cisco Live exam as well as the half price cert exams there. And I sincerely hope the rumored options for recertification take flight soon. Because I don’t know how ready I am to go all out to study when there’s that much money on the line.

Fixing The CCIE Written – A Follow Up

955951_28854808

I stirred up quite the hornet’s nest last week, didn’t I? I posted about how I thought the CCIE Routing and Switching Written Exam needed to be fixed. I got 75 favorites on Twitter and 40 retweets of my post, not to mention the countless people that shared it on a variety of forums and other sites. Since I was at Cisco Live, I had a lot of people coming up to me saying that they agreed with my views. I also had quite a few people that weren’t thrilled with my perspective. Thankfully, I had the chance to sit down with Yusuf Bhaiji, head of the CCIE program, and chat about things. I wanted to share some thoughts here.

Clarity Of Purpose

One of the biggest complaints that I’ve heard is that I was being “malicious” in my post with regards to the CCIE. I was also told that it was a case of “sour grapes” and even that the exam was as hard as it was on purpose because the CCIE is supposed to be hard. Mostly, I felt upset that people were under the impression that my post was designed to destroy, harm, or otherwise defame the CCIE in the eyes of the community. Let me state for the record what my position is:

I still believe the CCIE is the premier certification in networking. I’m happy to be a CCIE and love the program.

Why did I write the post? Not because I couldn’t pass the written. Not because I wanted people to tell me that I was wrong and being mean to them. I wrote the post because I saw a problem and wanted to address it. I felt that the comments being made by so many people that had recently taken the test needed to be collected and discussed. Sure, making light of these kinds of issues in a public forum won’t make people happy. But, as I said to the CCIE team, would you rather know about it or let it fester quietly?

Yusuf assured me that the CCIE program holds itself to the highest standards. All questions are evaluated by three subject matter experts (SMEs) for relevance and correctness before being included in the exam. If those three experts don’t sign off, the question doesn’t go in. There are also quite a few metrics built into the testing software that give the CCIE team feedback on questions and answer choices. Those programs can index all manner of statistics to figure out if questions are creating problems for candidates. Any given test can produce pages worth of valuable information for the people creating the test and trying to keep it relevant.

Another point that was brought up was the comment section on the exam. If you have any problem with a question, you need to fill out the comment form. Yes, I know that taking time out of the test to provide feedback can cause issues. It also interrupts your flow of answering questions. But if you even think for an instant that the question is unfair or misleading or incorrect, you have to leave a detailed comment to make sure the question is flagged properly for review. Which of the following comments means more to you?

  • Trivia question

or

  • This question tests on an obscure command and isn’t valid for a CCIE-level test.

I can promise I know which one is going to be evaluated more closely. And yes, every comment that has purpose is reviewed. The exam creators can print off every comment ever left on a question. The more detailed the comment, the more likely to trigger a review. So please make sure to leave a comment if you think there is a problem with the question.

Clarity Of Vision

Some of the conversations that I had during Cisco Live revolved around the relevance of the questions on the test to a CCIE candidate. Most of the people that I talked to were CCIEs already and using the test for recertification. A few came to me to talk about the relevance of the test questions to candidates that are qualifying for the lab.

While I’m not able to discuss any of the specific plans for the future of the program, I will say that there are ideas in place that could make this distinction matter less. Yusuf told me that the team will be releasing more details as soon as they are confirmed.

The most important point is that the issues that I have with the CCIE Written exam are fixable. I also believe that criticism without a suggestion solution is little more than whining. So I decided to put my money where my mouth is with regard to the CCIE written exam.

I volunteered to fix it.

I stepped up and offered my time as an SME to review the questions on the written exam for relevance, correctness, and grammar. That’s not a light undertaking. There are a ton of questions in the pool that need to be examined. So for every person that agreed with my post or told me that they thought the exam needed to be fixed, I’m putting you all on the spot as well.

It’s time for us as a community of CCIEs to do our part for the exam. Yusuf told me the easiest way to take part in the program is to visit the following URL:

http://www.cisco.com/go/certsme

Sign up for the SME program. Tell them that you want to help fix the CCIE. Maybe you only have to look at 5-10 questions. If the hundred or so people that agreed with me volunteered today, the entire test question pool could be analyzed in a matter of weeks. We could do our part to ensure that people taking the exam have the best possible test in front of them.

But I also challenge you to do more. Don’t just correct grammar or tell them they spelled “electricity” wrong in the question. Challenge them. Ask yourself if this is a question a CCIE candidate should know the answer to. There’s a chance that you could make a difference there. But you can’t do that unless you step up the plate.


Tom’s Take

I had at least ten people tell me that they would do whatever it took to fix the CCIE test last week after I talked to the CCIE cert team. They were excited and hopeful that the issues they saw with the test could be sorted out. I’ll admit that I stepped out on a pretty big limb here by doing this in public as opposed to over email or through official channels. And I do admit that I didn’t clarify my intent to build the program up as opposed to casting the whole exam team and process in a bad light.

Mea culpa.

But, my motivation succeed in getting people to talk about the CCIE written. There are many of you that are ready to do your part to help. Please, go sign up at the link above to join the SME program. Maybe you’ll never look at a single question, Maybe you’ll look at fifty. The point is that you step up and tell Cisco that you’re willing. If even fifteen people come forward and agree to help then that message will sound loud and clear that each and every one of us is proud of being a CCIE and want the program to continue long past the time when we’re retired and telling our grandchildren about the good old days of hard but fair tests.

If you have any questions about participating in the program or you want to reach out to me with your thoughts, don’t hesitate to contact me. Let’s put the power of community behind this!

The CCIE Routing And Switching Written Exam Needs To Be Fixed

CCIELogo

The former logo listed in this post was removed by request of Cisco

I’m having a great time at Cisco Live this year talking to networking professionals about the state of things. Most are optimistic about where their jobs are going to fit in with networking and software and the new way of doing things. But there is an undercurrent of dissatisfaction with one of the most fundamental pieces of network training in the world. The discontent is palpable. From what I’ve heard around Las Vegas this week, it’s time to fix the CCIE Written Exam.

Whadda Ya Know?!?

The CCIE written is the bellwether of network training. It’s a chance for network engineers that use Cisco gear to prove they have what it takes to complete a difficult regimen of training to connect networks of impressive size. It’s also a rite of passage to show others that you know how to study, prep, and complete a difficult practical examination without losing your cool. But all that hard work starts with a written test.

The CCIE written has always been a tough test. It’s the only barrier to entry to the CCIE lab. Because the CCIE has never had prerequisites and likely never will due to long standing tradition, the only thing standing in the way of you ability to sit the grueling lab test is a 100 question multiple choice exam that gauges your ability to understand networking at a deep technical level.

But within the last year or so, the latest version of the CCIE written exam has begun to get very bad reviews from all takers of the test. There are quite a few people that have talked about how bad the test is for candidates. Unlike a lot of “sour grapes” cases of people railing against a test they failed, the feedback for the CCIE written is entirely different. It tends to fall into a couple of categories:

The Test Is Poorly Written

The most resounding critique of the exam is that it is a poorly constructed and executed test. The question quality is subpar. There are spelling mistakes throughout and test questions that have poor answer selections. Having spent a large amount of time helping construct the CCNA exam years ago, I can tell you that you will spend the bulk of your time creating wrong answers as distractors to the right ones. Guidelines say that a candidate should have no better than a 25% chance to guess the correct answer from all the choices. If you’ve ever taken a math test that has four multiple choice answers with three being correct for various mistakes in working the problem, you know just how insidious proper distractors can be (and math teachers too).

The CCIE written is riddled with bad distractors according to reports. It also has questions that don’t have a true proper answer or a set of answers that are all technically correct with no way to select them all. That frustrates test takers and makes it very difficult to study for the exam. The editing and test mechanics errors must be rectified quickly in order to restore confidence to the people taking the test.

The Test Doesn’t Cover The Material

Once people stop telling me how bad the test is constructed, they start telling me that the questions are bad on a conceptual level as well. No NDAs are violated during these discussions to protect everyone involved, but the general opinion is that the test has skewed in the wrong direction. Cisco seems to be creating a test that focuses more on the Cisco and less on the Internetworking part of the CCIE.

The test has never been confused for being a vendor-neutral exam. Any look at the blueprint will tell you that there a plenty of proprietary protocols and implementation methods there. But the older versions of the exam did do a good job of teaching you how to build a network that could behave itself with other non-Cisco sections. Redistributing EIGRP and OSPF is a prime example. But the focus of the new exam seems to be skewed toward very specific Cisco proprietary protocols and the minutia around how they operate. I’ve always thought that knowing the hello and dead timers of OSPF NBMA areas is a huge time sink and really only justified for test takers, but I also see why knowing that would be important in multi-vendor operations. But knowing the same thing for an EIGRP DMVPN seems a bit pointless.

The other problem is that, by the admission of most test takers, the current CCIE Written Exam study guide doesn’t cover the areas of the blueprint that are potentially on the test. I feel very sorry for my friend Narbik Kocharians here. He worked very hard to create a study guide that would help test takers pass the exam with the knowledge necessary to do well on the lab. And having a test over a completely different area than his guide makes him look bad in the eyes of testers without good cause. It’s like a college class when the professor tells you to study the book but gives you a test over his or her lectures. It’s not fair because you studied what you were told and failed because they tested something else.

CCIEs Feel There Are Better Recert Options

This is the most damaging problem in my mind. About half the test takers for the CCIE written are candidates looking to qualify for the lab. That requires them to take the written exam for their specific track. But the other half of the test takers are CCIEs that have passed the lab and are looking to recertify. For these professionals, any CCIE written exam is valid for recertification.

Many CCIE candidates look to broaden their horizons by moving to different track to keep their CCIE current while they study for service provider, data center, or even collaboration as a topic area of study. For them, the CCIE is a stepping stone to keep the learning process going. But many CCIEs I’ve spoken to in the past few months are starting to take other exams not because they want to learn new things, but because the CCIE Routing and Switch written exam is such a terrible test.

Quite a few CCIEs are using the CCDE written to recertify. They feel it is a better overall test even though it doesn’t test the material to the level that the CCIE R&S written exam does. They would even be willing to take the chance of getting a question on an area of technology that they know nothing about to avoid having to deal with poor questions in their areas of study. Still more CCIEs are choosing to become Emeritus and “retire” so as to avoid the pain of the written exam. While this has implications for partner status and a host of other challenges for practicing engineers, you have to wonder how bad things must be to make retirement of your CCIE number look like a better option.


Tom’s Take

I took the CCIE R&S written last year at Cisco Live. I was so disgusted with the exam that I immediately switched to the CCDE written and recertified my number while simultaneously vowing never to take the R&S written again. From what I’ve heard this year, the test quality is still slipping with no relief in sight. It’s a sad state of affairs when you realize that the flagship test for Cisco engineers is so horribly broken that those same engineers believe it can’t be fixed. They feel that all the comments and feedback in the world are ignored and their expertise in taking exams is pushed aside for higher cut scores and a more exclusive number of candidates. The dark side of it all is the hope that there isn’t an agenda to push official training materials or other kinds of shortcuts that would help candidates while charging them more and/or locking out third party training providers that work hard to help people study for the lab.

Cisco needs to fix this problem now. They need to listen to feedback and get their written problems under control. If they don’t, they may soon find the only people taking the R&S written test are the same kinds of dumpers and cheaters they think they are trying to keep out with a poorly constructed test.

NOTE: I have published an update to this post here: Fixing The CCIE Written – A Follow Up

CCIE at 50k: Software Defined? Or Hardware Driven?

50kSticker

Congratulations to Ryan Booth (@That1Guy_15) on becoming CCIE #50117. It’s a huge accomplishment for him and the networking community. Ryan has put in a lot of study time so this is just the payoff for hard work and a job well done. Ryan has done something many dream of and few can achieve. But where is the CCIE program today? And where will it be in the future?

Who Wants To Be A CCIE?

A lot of virtual ink has been committed to opinions in the past couple of years about how the CCIE is become increasingly irrelevant in a world of software defined DevOps focused non-traditional networking teams. It has been said that the CCIE doesn’t teach modern networking concepts like programming or building networks in a world with no CLI access. While this is all true, I don’t think it diminishes the value of getting a CCIE.

The CCIE has never been about building a modern network. It has never been focused on creating anything other than a medium-sized enterprise network in the case of the routing and switching exam. It is not a test of best practices or of greenfield deployment scenarios. Instead, it has been a test of interoperability with an exisiting architecture. It tests the ability of the candidate to add devices and protocols to a stable existing network.

Other flavors of the CCIE test over different protocols or technologies, but the idea is still the same. The only one that even comes close to requiring programming is the CCIE Collaboration, which tests over the ability to customize Cisco Contact Center scripts. Otherwise, each test focuses on technology implementation and not architecture or operation.

Current logic dictates that people don’t want to take the CCIE because it doesn’t teach programming or API interaction. Yet candidates are showing up in droves. It’s almost as if the networks we have today are going to need to be maintained and built out over the coming years. These are the kinds of tasks that are well suited to a support-focused certification like the CCIE. The ideal CCIE candidate isn’t using Vagrant and Chef in a lab somewhere. They’re muddling through OSPF to RIP distribution somewhere in the dark corners of a network that got welded on after an acquisition.

Is Everyone A CCIE?

One thing I have noticed about the CCIE is the fact the number climb seems to have leveled off. It’s not the rapid explosion of certifications that it has been in the past, nor is it the eventual cliff of increased difficulty. Things seem to be marching more toward steady growth. I don’t know how much of that can be attributed to factors like the Cisco official CCIE training program or the upgrade to version 5 almost two years ago.

Lots of CCIEs doesn’t necessarily mean that the test has lost meaning. Microsoft had several thousand MCSEs by the time the certification became a punchline to countless call center jokes. Novell had a virtual army of Certified NetWare Engineers (CNEs) before software changes locked many of them into CNE 5 or CNE 6. Having a lot of certified individuals doesn’t devalue the certificaiton. It’s what people do with it that creates the reputation. Ask and Novell Certififed Directory Engineer (CDE) about the reputation garnered by a test and they can give you a lesson in hard exams that breed bright engineers.

Does that mean that we should brace ourselves for even more CCIEs in the future? It likely won’t be as bad as has been imagined. The written exam for version 5 has pointed out to me that Cisco is going to start closing ranks around technologies in the near future. The written exam serves as a testing ground for potential new topics on the exam. MPLS was a written topic long before it became a potential lab exam topic. The current written exam is full of technologies that make me think Cisco is starting to put more emphasis on the Cisco and less on the Internetworking in CCIE.

Cisco wants to have a legion of certified individuals that think about Cisco technology benefits. That’s why we’re starting to see a shift toward things like DMVPN and GETVPN in testing. In place of industry standard protocols, we get the Cisco improved versions. This locks candidates into the Cisco method of thinking and ensures that their go-to solutions will include some form of proprietary technology.

If this shift in thinking is really the start of the new way of certification testing, I worry for the future of the CCIE. Not because there are 50,000 CCIEs, but because the new inductees into the CCIE group will be focused on creating islands of Cisco in the sea of interoperable data center networks. That’s good for Cisco’s bottom line, but bad for the reputation of the CCIE. Could you imagine what would happen if a CCIE walked in and told you they couldn’t fix your MPLS VPN configuration issues because “I only know how to work on DMVPN”?


Tom’s Take

Every time someone I know passes the CCIE it makes me happy that they’ve completed a rigorous exam testing process. It tells me this person knows how to follow the lab instructions to create an interoperable enterprise network based on constraints. It also tells me that this person knows how to study material and doesn’t give up. Those are the kinds of people I would want in my networking group.

CCIEs are the perfect people to learn more modern network techniques like programmability and SDN. Not because they learned how to do it on their test. But because they are the kinds of people that learn well and will apply everything they have to picking up a new concept. But it needs to be pointed out here that Cisco must foster that kind of interoperable learning experience with CCIEs. Focusing too heavily on proprietary solutions to help create an army of unknowing Cisco SEs in the field will only serve to hurt Cisco in the future when that group of certified individuals must learn to work in the world of networking post-SD.

 

Get a CCIE, Don’t Be A CCIE

Getting a CCIE is considered to be the pinnacle of a person’s networking career.  It is the culmination of hundreds (if not thousands) of hours of study.  People pass the lab and celebrate with the relief that can only come from completing a milestone in life.  But it’s important for newly-minted CCIEs to realize that getting your number doesn’t mean you obtained hubris with it.

A great article that talks about something similar comes from Hunter Walk.  It’s Fine To Get an MBA, But Don’t Be An MBA shows many of the things I’m talking about.  With the MBA, it’s a bit different.  The MBA is a pure book learning environment with very little practical experience.  The CCIE is a totally practical exam that requires demonstration of knowledge.  However, both of these things share something in common.  People get very hung up on the knowledge from the certification and forget to keep an open mind about other ideas.  In essence, someone that is “Being a CCIE” is using their certification incorrectly.

Here are some points:

Get A CCIE to further your knowledge about networking and learn how system work. Don’t Be A CCIE and think that you’ve learned everything there is to know about networking.

Get A CCIE and work with your coworkers and peers to solve problems.  Don’t Be A CCIE and ignore everyone because you think you’re smarter than they are.

Get A CCIE and contribute to the community with knowledge and experience.  Don’t Be A CCIE and refuse to share because you can’t be bothered.

Get A CCIE and help your company to take on bigger and better networking projects.  Don’t Be A CCIE and assume you are indispensable.

Get A CCIE because you want to.  Don’t Be A CCIE and assume you’ve always been one.

A CCIE doesn’t change who you are.  It just serves to show people how dedicated you can be.  Don’t let five little numbers turn you into a bully or a know-it-all.  Realize you still have much to learn.  Understand that your position is now at the forefront of where networking is going, not where it has been.  When you know that being a CCIE is more than just a piece of paper, then you will have truly gotten your CCIE.