Tune In and Switch Off

As I sit here right now, the country of Egypt is a black hole on the Internet.  All 3,500 prefixes originated by Egypt’s four major ISPs have been withdrawn from the global BGP table.  There is no route into or out of the country, save the one ISP utilized by the Egyptian Stock Market, most likely in an effort to keep the country’s economy from collapsing.  This follows on the heels of other government interference in cybercommunications in Tunisia this past month and Iran last year.  Egypt, however, is the first country to completely darken the Internet in an effort to keep services such as Twitter and Facebook from coordinating resistance and allowing information to be disseminated to the world at large.  I learned a very long time ago that arguing about politics never leads anywhere.  What I would like to comment on, however, is the trend toward censoring information by disrupting network communication.

Egypt yanked all Internet access for its citizens in an effort to control information.  Tunisia has been accused of affecting Internet traffic for its citizens as well, blocking certain routes and causing outages on the Web.  Iran limited access to social media and even attempted to severely rate limit Internet traffic during the election protests last year.  This trend shows that governments are starting to realize the power that the Internet provides to disaffected groups of people.  No longer to “subversives” need to meet in underground basements or abandoned warehouses.  Those places have been replaced by chat rooms and e-mail.  Relying on one or two trustworthy individuals to get the word out by smuggling rolls of film to the mass media has been replaced with instant pictures being uploaded from a cell phone to Twitter or Flickr.  The speed with which protests can become revolutions has become frighteningly accelerated.  So too is the speed with which the affected government can slam the door shut on the ability for these revolutionaries to use the very media which they rely on to spread the word.  Egypt was able to successfully cut off access within a few hours of the first rumors of such a thing being contemplated.

For those of you that think that something like that could never happen here (here being the US), let me direct your attention to the Protecting Cyberspace as a National Asset Act.  This hotly debated bill would give the government more ability to combat large-scale cyber warfare and allow them to protect assets deemed vital to the national interest.  The biggest concern comes from a provision inserted that would give the president the ability to enact “emergency measures” to prevent a wide-reaching cyber attack.  This includes the power to shut down major networks for a period of up to 120 days.  After that time, Congress must either approve an extension, or the networks must be reactivated.  I won’t delve into some of the wilder conspiracy theories I’ve seen surrounding this bill, but the idea that our networks could be shut down without our consent to protect us is troubling.  According to my research, there is no provision that defines the situation that could cause a national shutdown.  The president, acting through the National Center for Cybersecurity and Communications (NCCC) Director, is supposed to inform the affected networks to enact their emergency measures and ensure the emergency actions represent the least disruptive means feasible to operations.  In other words, the NCCC director just has to tell you he shut you down and you should try to make things work as well as you can.

Using this as a possible scenario, assume some kind of external driver causes the president and the NCCC director to shut down a large portion of the Internet traffic.  It doesn’t have to be a revolution or something so sinister.  It could be a Stuxnet-type attack on critical power infrastructure.  Or maybe even a coordinated cyber attack like something out of a Tom Clancy novel.  In an attempt to deter the attack or mitigate the damage, let’s say the unprecedented step of withdrawing a large number of BGP prefixes is taken, similarly to what Egypt has done.  What kind of global chaos might this cause?  How many transit ASes exist in the US that would pass traffic around the world.  I’ve seen stories of how the World Trade Center attacks in 2001 caused a global Internet slowdown due to the amount of traffic that was passed through the networks located there.  That was two buildings.  Imagine withdrawing even half the traffic that flows through the US and networks located here.  What impact would that have?  The possibilities would be mind-boggling.  Even a carefully coordinated network shutdown would have far reaching impact that no one could foresee.  Chaos is funny like that.

The Internet, or cyberspace or whatever your term for it, is now something of a curiosity.  It exists on its own, independent of the laws of nations or man.  Those who seek to control information flow or restrict access find themselves quickly thwarted by the fact that packets and frames do not respect political boundaries.  For every attempt to shutdown The Pirate Bay, a simple move to different location allowed them to stay active.  Even when pressure was applied to the people behind the site, it was quickly seen that their creation had taken on a life of its own and would persist no matter what.  What of the Wikileaks saga, where the attempt to behead the organization by targeting its leader has only fanned its flames and most likely ensured its survival no matter what may happen to Julian Assange.  Those of us who live our lives in this electronic realm see differences in the way culture is developing.  There are lawless places in the Internet where mob rule is the law of the cyberland.  Information is never truly forgotten, merely pigeonholed away until it is needed again.  Attempts to impose political will upon the citizens of the Internet are usually met with force, protest, and in some cases, retribution.  I keep wondering when organizations are going to figure out that attempting to erase information is tantamount to daring the Internet to publicize it.  In the same way, attempting to shut down access the Internet and social media at large is a sure way to force people to circumvent these restrictions.  As we watched Egypt vanish from the cyber landscape last night, many of my friends remarked that it would only be a matter of time before someone challenged the blockade and won.  Someone could hack the edge routers and reestablish the BGP peering with the rest of the world and the floodgates would be opened again.  Whether or not that happens in the next few days remains to be seen.

As the world becomes more reliant on the Internet to provide information to everyone, we as cyber citizens must also remain vigilant to keep the information flowing freely.  The Internet by design lends itself to surviving major disruptions without totally crashing.  It is our responsibility to show the world that information wants to be learned and shared and no amount of meddling will change that.


I Am A Network Ninja

I am a network ninja.

I appear meek and uninteresting.  My stealth and guile are my weapons.  I have succeeded in my task if you never knew I fixed the network.  My khakis and polo shirt allow me to blend into the crowd of salespeople and marketing drones, yet hide my knowledge of BGP and MPLS.  I care not for the ritual troubleshooting combat of TAC engineers.  I use whatever methods I can to achieve victory over that which might harm my network.

My tools appear deceptive at first.  A laptop. A console cable. Simple business cards.  Yet, when in my hands, they become the weapons of legend.  Repeating Youtube videos to distract the masses while I work my network ninja magic.  A console cable to garrote those who question my skill with OSPF.  Business cards to use as shurikens to force back the account managers who dare to be technical and disrupt my troubleshooting chi.

My SNMP spies report every movement of the enemy to me.  I know what the battlefield will look like before the battle is even commenced.  With a flash of light from my LED iPhone camera flash, I mysteriously appear at your side, asking you why you are downloading a torrent on my network.  Before you can speak your lies, my honed reflexes have rate-limited your switchport.  I give you the choice of no choice.  Continue downloading at your peril, for your fate is in your own hands.  Before you can put up a fight explaining why you need a copy of Harry Potter before it’s released in theaters, I disappear with a puff of smoke, off to sow havoc in the server room.

I train my students without training.  I give them difficult configuration tasks and force them to gather information about switches by hand.  I make them learn from experience and recognize danger before they can perceive it.  I create spanning tree loops and redistribution quagmires so my students will never fear the hell of a network gone wrong.  When they realize that my difficult tasks and exacting manner have in fact taught them the way of network ninjitsu,  I send them into the world, knowing they will carry on my legacy and teach other network ninjas as they have been taught.

My deception is my strength.  When the C-level shoguns ask why the network is slow, I appear to give them many answers, yet I reveal none.  The questions for me become questions.  I ask him for his opinion of what might be wrong.  I feign ignorance in his presence, knowing how to use his ego and strength as a weapon of my own.  As he explains how the network needs his experience, I wait for the opening. When confusion reigns and all appears cloudy, I strike.  I turn off the shogun’s Internet radio and appear demure, claiming all should be well thanks to his enlightenment.  As I retreat to my network dojo, the shogun feels content, knowing he fixed the problems and showed his network serf a thing or two about the way things work.  My greatest work is fixing the network without fixing the network.

When others speak of the mysterious forces that weave magic and make the network bend in ways they never dreamed of, I shall laugh and tell them they must be dreaming.  There is no such thing as a network ninja.

Yet, I AM a network ninja…

These /8s Are Now Diamonds

The end times are upon us.  According to many reliable sources, the allocation of IPv4 addresses is quickly reaching it’s conclusion.  Of the final seven /8s available to be allocated by IANA, APNIC is about to exercise its option on two of them.  When that happens, the final 5 will be allocated as planned to each of the 5 Regional Internet Registrars (RIRs) and completely deplete the source pool of allocatable IPv4 addresses.  Now, before Chicken Little starts screaming about what this means, let’s take a step back and examine things.

I liken the total allocation addresses from IANA to the RIRs to a resource exhaustion.  For the sake of discussion, lets pretend the IPv4 addresses are diamonds.  The announcement of of total allocation would be like De Beers announcing that all of the diamonds in the earth’s crust have been mined and there are no more available.  That doesn’t necessarily mean that all the engagement rings and tennis bracelets for sale will disappear tomorrow.  What it means is the primary source for this resource is now depleted.  Just like we can’t manufacture any more IPv4 addresses, in this example we can’t mine any more diamonds.  So what happens next?

Usually, when a resource starts becoming scarce, the cost to acquire that resource will be driven up.  In this particular case, people like Greg Ferro have already suggested that there will be a “run” on addresses.  Again, this is a behavior that is typically seen when the announcement is made that a resource is becoming hard to come by.  I think that the RIRs will start putting policies in place to prevent ISPs and other parties from requesting more IPv4 addresses than they currently need.  That will prevent the pool that the RIRs currently possess from becoming depleted faster than necessary.  It will also hopefully stave off the resale of these addresses on the black market, which is a distant possibility but possible nonetheless.  Just like in our fictional diamond example, the price of the diamonds at the jewelry store will go up, and most stores will implement policies restricting the sale of large numbers of stones to single parties, so as to prevent hoarding and help keep prices high.  This will also stave off the onset of a diamond secondary market, where speculators will sell stockpiles of stones for exorbitant prices.

So, now that IANA has run out of addresses, what’s next?  Well, the next countdown becomes the date the first RIR runs out of it’s allocation.  Right now, that’s projected to be APNIC sometime in October 2011.  APNIC has been burning through its addressing at blinding speed, and so they find themselves at the head of the IPv4 exhaustion line.  Once APNIC runs out of addresses, the only thing they can offer their customers going forward is IPv6 address space.  For some customers, this will be rather unsavory.  These types of customers will feel that IPv6 hasn’t penetrated deep enough into the market.  They’ll pay any price for those precious v4 prefixes.  So, I imagine that APNIC and the other RIRs will hold a small portion of IPv4 address blocks in reserve for those customers that are willing to pay big bucks for them.  For the customers without the pocketbook or that don’t care about the address space they receive, they’ll get IPv6 and likely won’t think twice about it.  Just like in our diamond example, when the first jewelry supply runs out of stones purchased from De Beers the price will start going up.  Perhaps they’ll offer lab-created diamonds of similar quality.  But there will be customers that feel the lab-created stones are inferior and those same customers will pay a significant amount of money to get the “real” stones.  In these cases, the smart jewelry supplier will hold back some of the best gems in order to get a much better price for them.

Once the first RIR runs out of address, things will accelerate from there.  Depending on the level of IPv6 preparedness in the market, you may start seeing customers hosting equipment in locations where RIRs still have address space to assign.  I would sincerely hope that by the end of 2011 that most everyone has either begun their IPv6 prep in earnest or completed it with flying colors.  Otherwise, I predict there will be a migration of data centers to locations served by AfriNIC, which is the RIR with the largest block of unallocated /8s.  The final exhaustion of IPv4 addresses isn’t predicted to occur until July 2012.  That gives customers and plenty of time to decide how to implement IPv6 rather than moving data centers around to mop up what little IPv4 address space remains.  In our fictional example, as the jewelery suppliers start running out of diamonds to give to their customers, their customers will begin shopping around to find suppliers that still have stock, even if they have to start importing stock from supplies located overseas.

Once the last RIR runs out of addresses to give to its customers, then it will be a matter of time before the last of the IPv4 addresses are allocated to the final end users by the ISPs and other middle men.  Customers that deal directly with ARIN and RIPE and the other RIRs will be out of luck, instead needing to move to IPv6 to continue growing their Internet presence.  Hopefully by the time this occurs in late 2012, IPv6 will be firmly entrenched and the drive to allocate the final IPv4 address space will be greatly lessened.  With end users concentrating on their shiny new IPv6 address blocks, the last of the IPv4 addresses can be handed out to those truly in need.  After that, the Internet can go forward operating on a dual-stack of IPv4 and IPv6 until the last of the IPv4-only hosts go dark, leaving us totally on IPv6.  I doubt that day will ever truly come, but it’s a possibility that’s out there.  And in our fictional diamond example, people will still show off their fancy jewelry, but the world at large will start to turn to the next precious stone like rubies or sapphires.  While diamonds will never truly be gone, the demand for that which can no longer be obtained will be lessened greatly, only pursued by those with the resources to expend to obtain something so expensive.

The final and total allocation of IPv4 isn’t something that’s going to happen overnight.  It will be a death by degrees.  Just like boiling a frog one degree at a time, there was a time we might not have known what was going on until it was too late.  Thankfully, enough people have been getting the word out to cause everyone to start making their plans early and get ready for what is coming.  This was no more apparent to me as when I contacted a local technology group putting on a conference to request a presentation slot to speak about IPv4 exhaustion and IPv6 planning.  The person on the other end of the phone was a rather technical person, yet I had to spend some time explaining just what IPv6 was and why getting the word out was so important.  So, to those of you that have influence on communication and/or blogging and tweeting avenues, keep talking about what’s going on with IPv4 depletion as we approach the true end of the address space.  That way, we don’t find ourselves scrambling for diamonds at the last minute or wondering if we need to upgrade to Diamondv6.

For Want of a Nail

I have been weighed.  And measured.  And my configuring skills have been found…lacking.  Welcome, everyone, to the post-lab wrap up post.  Jan 20th looked like a good day for me.  I found all the faults in the troubleshooting section.  The config section looked very beatable to me.  I double checked all the configs with the hour and a half of extra time I had available.  I found some dumb mistakes that were immediately corrected.  And when I left the lab, I had a very good feeling about this one.  Alas, 58 minutes later, the score report I received indicated that I wasn’t as good as I had expected.

Without gory, rule-breaking details, I really thought I did better.  I had reachability in my lab without violating any of the constraints.  My paper was filled with two check marks next to all by one task that I was working on when time was called.  I re-read each question and physically put the point of my pencil on each word on the screen to make sure I didn’t read over anything and miss a subtle nuance that could sink me.  In the end, I think those nuances are what might have sunk me.  It’s not enough to have full reachability if you miss a little phrase that tells you to avoid a certain method or use a specific technique.  While I feel that I had accounted for all of those possibilities, the score report doesn’t think so.  And since the cold reality of the score report is more official than my high hopes, it looks like this trip to sunny San Jose was a bust as well.

After posting my results on Twitter, along with the condolences there were a lot of questions about whether or not I should as for a lab re-grade.  For those not familiar, the lab is partially graded by a script.  In cases where the script returns some strange results a proctor will look over your lab and double check certain tricky tasks.  Historically, I’ve gotten my scores after 3-5 hours.  The fact that I got this report while sitting at In-and-Out burger confused me.  Perhaps the first pass of the grading script didn’t have any issues with my configs.  One or two people even suggested that as a good sign that I might have even passed from the script alone.  So, as I stared at the percentages in front of me, I contemplated something I had never tried before.  I wanted them to take another look at my exam.  I wanted another proctor to take the configurations saved from my equipment and load them up onto new routers and regrade my exam by hand.  This process is not without it’s downsides, though.  First, it costs $250 for them to even do it.  If you get re-graded to a passing score, there is no refund.  Secondly, there is as much possibility of you losing points as there is passing.  This is because the human eyes of the proctor may catch something incorrect that the script missed.  So, you may have only been 3 points from passing, only to now find that you are 8 points away due to some other mistakes.  However, if you are very close to the mark, as in ~2-3%, there is no reason not to take the chance on the regrade.  After all, it’s still cheaper than a new $1400 lab attempt, right?  Third, the regrade attempt takes up to three weeks.  So if you’re hoping for a fast turnaround before deciding to take another lab attempt, you’re going to have a cool your jets for the better part of a month.  On the bright side, Cisco is usually very understanding and will refund the $1400 if you’ve booked another lab attempt and your re-read is honored and changed to a passing score.

Turns out, Cisco won’t even look at tests that are not “statistically close to the historical passing rate.” English?  If you aren’t within about 5% of the passing score, you don’t even have the option to have it regraded.  And, at least in Cisco’s eyes, I was wide of the mark this time.  I’ve filed a case with support to at least be granted the option for the regrade.  We’ll see what happens.  In the meantime, I’m getting back on the horse.  I’ve already had conversations with my employer about when and if there will be another attempt on their dime.  I’m not packing away any lab equipment for the time being.  I’m getting right back into the thick of things to keep it all fresh.  In fact, my wife and I went to the final closing party for a local pub that we’ve been going to since college.  As I sat there surrounded by all the merrymakers and noise of a bar full of people, all I could think of was how much I wanted to get back home and start labbing up some multicast questions.  It seems that the lab has transformed me and sharpened my focus into laser-like precision.  I think about how I’m going to configure tasks and how I’m going to move efficiently from one task to the next.  I pace myself based on the amount of work I can get done in 6 hours with a 40-minute lunch break.  I review flash cards and GNS3 configs during my free time.

No matter what, I’m going to go back and try this again.  I feel like the top of Everest is in sight and I just need one last push to reach the summit.  I’ve come too far to fail now.  More studying, more refining of my task config skills, more “stick time” for this airplane of router and switch configuration.  For want of a nail, this lab was lost.  But the next time around, the only nail will be the lab being nailed by me.

Avoiding Proctor Proctology

Other than perhaps professional sports referees, I can think of no more maligned group of individuals than the proctors in the CCIE labs.  The myths about them are legend.  They screw up your rack at lunch.  They do everything they can to make you fail.  They act obtuse when you ask questions because they don’t like passing new CCIEs.  For the most part, all the mystery and nastiness that surrounds the proctors is just a bunch of hokum.  They are people just like you and I.  They do their jobs just like you and I.  It just so happens that their job is closely identified with something that most people study for hundreds of hours to achieve, and when those people fail, they look to cast blame on others.

I’ve been to the lab once or twice.  I’ve met several proctors both there and at Cisco events.  People who only need to be known by first names.  People like Howard and Tom and Stefan.  And each time I encountered them outside the lab setting, they impressed me with their poise and charm and sense of humor.  Just like any position of authority, the CCIE lab proctor has a role to play once they step inside the walls of the lab.  They must project an  aura of authority and calm.  They must provide guidance where confusion reigns.  And above all, they must protect the integrity of the program.  When most people deal with them in this setting, they come off cold and uncaring.  Just like a judge or a police officer, it’s important to remember that the position of proctor is a role that must be played.  Keeping some things in mind before you step through the badged door will go a long way to making your proctor experience a smooth one.

1.  Be nice. Yes, it sounds silly, but it’s a very important thing to keep in mind.  People are amped up when the walk into the lab.  They are stressed and wired and strung out.  Some people retreat into a shell when stressed.  Some people lash out and are irritable.  What’s important to keep in mind is that when dealing with the proctor, you should keep a calm and even demeanor.  People tend to respond in kind with the emotions they are presented.  If you walk over to ask a question and are short and snippy, expect a short answer in return.  However, if you are nice and pleasant, it can go a long way toward getting a favorable answer quickly.  When you go to lunch, don’t be afraid to engage the proctor in some light conversation.  Talk about the weather or the food or a sports team.  Anything but the lab.  Engaging them outside the walls will give you a feel for how they react to things and can help you judge if they will be helpful when it comes time to utilize them.  Besides, it never hurts to be friendly.

2.  Ask ‘yes’ or ‘no’ questions. The most popular complaint I hear about proctors is that they never answer the question that you ask them.  Before you stand up to go ask for clarification, carefully word your question so that it can be answered with a simple ‘yes’ or ‘no’.  For instance, rather than asking “How should I configure this frame relay connection?”, instead ask “There are two ways two configure this frame relay connection, point-to-point and point-to-multipoint.  The question isn’t clear about which I should use.  I’m leaning toward configuring it as a point-to-point.  Would I be correct in this assumption?”  The proctor may still choose not to answer your question, but you’re more likely to get a binary yes/no response out of them.  By showing you understand the technology and you aren’t just trolling for an answer, you’ll appeal to their interpretive skills.  Remember that the proctor won’t give you an answer, no matter what.  They have too much riding on their job, reputation, and the CCIE program to ‘bend’ the rules for one candidate.

3.  Don’t assume a problem is something you can’t fix. Want to piss off a proctor really fast?  Walk up and say, “I can’t get OSPF to come up.  My rack must have a cabling problem.”  I promise that after the eye roll and gruff answer, you’ll fail the lab.  Candidates who are weak in troubleshooting skills tend to blame the physical layers first because it’s the one layer they can’t touch in the lab.  In the old 2-day lab, cabling could be an issue to resolve.  But in the 1-day lab, with cabling extracted from the candidate’s control, there is a 1-in-1,000 chance that the cabling is at fault.  Think about it like this: the lab equipment supports 3-5 candidates 5 days a week for 50 weeks a year.  Cabling issues will be caught and dealt with quickly.  Simple things like bad ports or bad cables will be caught when the lab racks are booted first thing in the morning.  You going to the proctor and claiming that OSPF is broken because your routers are wired backwards will only serve to irritate the proctor.  What will happen next is that the proctor will ask you to wait by his/her desk or make you wait in the RTP conference room.  They will go to your rack and start troubleshooting OSPF.  They’ll find out you misconfigured a network statement or forgot to enable authentication.  They’ll check to make absolutely sure it’s not a layer 1 problem, all while you are isolated away from your terminal.  Then, after 15-30 minutes, they’ll come get you and tell you, “It’s not a physical problem.”  That’s it.  No other information.  You won’t get to see what they did to find your problem.  You won’t get any hints about how to fix the issue.  And you won’t get those 15-30 minutes back.  Period.  On the other hand, if you go to the proctor with a long list of reasons why it has to be a layer 1 problem, like BERT or TDR output or a down/down physical interface that won’t come up at all, they’ll be more receptive due to your troubleshooting efforts.  And if they find a layer 1 fault during their troubleshooting, you’ll probably get the time back from their efforts and the time it takes to replace the faulty unit.

4.  Don’t cheat. Well, duh.  You’d think that should be a given.  But people still try to pull stupid things all the time.  And not just the braindumping.  Writing things down and then trying to sneak them out on the scratch paper, of which every scrap must be accounted for.  Trying to use a cell phone to discreetly look up answers.  Looking at other screens (for all the good it’ll do you).  Just don’t cheat, or even give the appearance you’re cheating.  That should keep the proctors from getting quite nasty with you.

In the end, just remember that these guys and gals are doing their jobs as defined by the CCIE program.  They work hard and do everything within their guidelines to help you pass.  They aren’t out to screw you, and if you keep your head about you and use some common sense and manners, they’ll be the best resource you’ll find in the lab.

Hooray for Bruno!

Twenty-four hours after Cisco ‘tweaked’ the CCIE lab troubleshooing section a little, we finally get a little bit of feedback about what they have wrought on the weary CCIE candidates.  This thread over on the Cisco Learning Network serves as the official announcement.  Further down the list is a response to some questions from concerned folks about what this could mean from Bruno van de Werve, who appears to be part of the CCIE program inside Cisco.  Bruno was also responsible for the excellent CCIE lab web interface video.  So, what did Bruno share with us?

1.  This is a virtual switch based on a router image, so the ports will be shutdown and not configured as switchports up front.  This about having a switch module in a router in Dynamips/GNS3 and I’m sure this is what the L2IOU is like.

2.  The image appears to be based more toward the 3550 rather than the 3560.  This comes from comments about the port being in dynamic desirable instead of the 3560 method.

3.  All interfaces are Ethernet, not FastEthernet or GigabitEthernet.  They are also arranged in groups of 4, e.g. e 0/0, e 0/1 and so on.  Due to the software limitations, you can only do interface range commands across the 4 ports of the module, so for instance setting up more than 4 trunks at once would require multiple interface range commands.  As an aside here, Bruno says that they are most likely not going to have more than 4 trunks per switch right now.

4.  These switches are based off of 12.2 mainline IOS with some L2 switching features added in.  They also lack any of the hardware ASICs necessary to do advanced features.  So, no Cat-QoS for instance (His words, not mine.  Cat-QoS, really???)

5.  There are only two switches at present in the TS section, and they are in the same IGP domain.  Bruno elaborates that they might add more switches later as features are added to the L2IOU image.

6.  There will still only be 10 questions in the TS section (his words).  So, at best you can probably expect 1-2 L2 questions for now.  He did say that there is a possibility that the questions could influence other tickets, so be on guard if you have a L2 question up front that it might affect an OSPF question later.

7.  There are no planned changes to the configuration section of the lab at this time.  No, they aren’t going to remove MPLS.  They aren’t screwing your config section totally.  Also, in a reply to one of the first commenters, Bruno said,

“At the moment, there won’t be any changes to the configuration section of the lab, so the two L2 troubleshooting questions will still be there.”

Ahem.  As my old econ professor in college used to say during test reviews…”Hint, hint, hint, oh hint.  You might see this on the exam.”  Guess what folks? Bruno just gave you a big hint.  You are still going to see L2 troubleshooting on the config section.  And if the historical trend is true, it’s going to be the first part of your lab.  “There are XXX faults in your initial lab configuration.  Correct these faults for 1 point each.  Be aware that these faults could impact configuration in other sections of the lab, and if they are not resolved and impact another working solution, you will not receive points for the non-working solution.”  So put your thinking caps on.

My Take

L2IOU was introduced to the troubleshooting section to crack down on the braindumpers.  Cisco knew they needed to add it to keep the cheaters off-balance, but the feature set of the IOU emulator wasn’t ready when the TS section went live.  I’m sure a team has been working on it night and day trying to get enough features built into it to make it a usable tool for the TS section.  It would be virtually impossible to have a physical setup for troubleshooting.  30+ routers would be a large rack, and now that you start adding physical switches in on top of it, it become unmanageable.  IOU made the most sense for the L3 section, but L2 is just as important for testing a candidate’s knowledge.  However, there wasn’t an emulator capable of perfectly virtualizing a switch.   And there still isn’t.  The fact that L2IOU got pushed out the door in the state that Bruno describes it in means that someone had to be done to stop the braindump hemmorage.  That’s the only reason I can think of to introduce a totally foreign software-emulated L2 device that by all accounts doesen’t resemble the feature set of the devices in the lab that you are already forced to troubleshoot a mere two hours after the beginning of the lab.  So why not put the additional troubleshooting questions in the config section where they can be done on real hardware?  Especially since there are only two L2IOU devices currently?

Because Cisco has never nailed down the exact nature of the TS section, it gives them more latitude in changing it quickly.  Adding ten new TS questions takes a lot less time than re-engineering a whole lab set.  Just like the OEQs before, the TS section is designed to weed out the weak candidates relying on someone else’s leaked materials rather than the tried-and-true studying methods that should be employed.

I still don’t like the change.  Not enough lead time.  A ham-handed attempt to fend off what appears to me to be a growing portion of lab candidates.  Seeing as how they appear to be phasing things in slowly and not increasing the impact of L2 until more features get baked in, I guess I can live with being a guinea pig for now.  But I sincerely hope that I pass my lab on this attempt so I don’t have to beta test new TS section features again.

Layer 2 – Electric Boogaloo

In alignment with CCIE level requirements, Cisco is adding L2 switching features to the CCIE R&S Troubleshooting exam through L2 IOS software on Unix (L2IOU) virtual environment. The new feature will be available starting January 17, 2011. The CCIE R&S exam consists of 2 sections b the troubleshooting (TS) section which runs two hours, and the Configuration (Config) section which is six hours. The Config lab utilizes actual physical devices in racks, whereas, the TS lab uses a virtual environment under IOU. IOU offers a very realistic simulation of router (L3+) features in the TS lab but until now had no L2 switch capability. With the addition of the new L2IOU, the TS lab will now include both L2 and L3 capabilities in the virtual environment.

Official Cisco Announcemnt (Thanks to Rob Routt for the text)

So, it appears that Cisco has finally added layer 2 (L2) troubleshooting to the first troubleshooting section of the lab.  Hmmm…

Ladies and gentlemen, a rant…

What?!?! Really?  Why did this have to happen four days before my exam!  Okay, it’s not fair to think that you’re singling anyone out with these changes, but when you added the open-ended questions (OEQ) in 2009, you did it the week before my second lab.  And that little change with the OEQs cost me one lab.  With luck like this, anyone going to Cisco Live Las Vegas 2011 should head down to the casino with me and bet the exact opposite of what I say. You’ll make millions.

I know the standard response.  Anything is fair game for troubleshooting on the TS section.  If I’m a prepared CCIE candidate, I should have no fear about this addition.  Everything will be fine, I’ll nail it.  All of these things are true, but it still doesn’t make me feel any better.  This is like going in to take a driver’s test only to find that the steering wheel is on the wrong side of the car!  Does it affect the test?  Not really, but it is a head scratcher.

If all I have to worry about is layer 3 faults, I don’t have to consider spanning tree or trunk ports or Q-in-Q tunneling or QoS mismatches or backup interfaces or Etherchannel misconfigurations or any one of a number of things.  No, adding L2 to the TS lab doesn’t make it necessarily harder, but it does broaden the range of things I have to think about when I start looking at a problem.  Irritating, this is.

Why the need to cram so much stuff into the beginning?  You know, you always did well with the first TS question of the config section – Hey, we broke three or four things in your lab that you need to fix before you get started.  You don’t get any points if they aren’t fixed, and since they probably are going to break other things, you won’t get points for other questions either.  Guess breaking things on five routers and four switches isn’t nearly as fun as doing it to 30 routers and who-knows-how-many switches.  I’m sure that the setup of the TS lab allows them to break lots and lots of interesting things in ways they can’t simulate in the config section.  But if that’s how things are going to be going forward, they need to remove any TS from the config section too, if it still exists.  After all, there are no sim questions on the CCIE written because they are covered in other sections.  Why should there be troubleshooting on the lab config section when you’ve so eloquently covered it in the TS section?  I figure that once the L2 TS stuff gets baked in, we might actually see this happen.

And another thing…one business day notice?  Really?  Come on now.  That’s just unfair.  Used to be, changes to the lab required six months of notification before going live to allow students the opportunity to study up on them and be prepared.  How do you think the folks taking the lab bright and early Monday morning are going to feel?  I guess that adding a whole new topic to the TS section doesn’t constitute a ‘major’ change anymore.  Or perhaps because you intentionally left the description of the TS section vague, it allows you the opportunity to ‘clarify’ it with little to no warning.

I’ve got a question, Cisco.  Are the braindumpers really hurting you that bad?  You can tell me all you want that the OEQs and the TS section were designed to better assess a candidate’s knowledge of every facet of networking.  With respect: bullshit.  The OEQs were a direct outgrowth of the interview process implemented at a specific testing site to catch people memorizing leaked lab questions and getting through with no difficulties.  The TS section replaced the OEQs once the process was refined to the point where they were no longer needed.  Have people started memorizing the TS section too?  Is it bad enough that we’re going to have a new exam every week in an effort to catch the unworthy candidates?  Perhaps we should just move to a totally interview-based exam with no access to CCO or documentation of any kind?  After all, the perfect CCIE candidate should be able to configure and troubleshoot a lab without access to a keyboard or a mouse or even a monitor!

Okay, I think that’s enough ranting for one morning.  I’m at T-minus 141 hours and counting.  That should be just enough time to brush up on my L2 TS skills in addition to all the other stuff I’m cramming for.  You know, I’m just about to the point where I’m not going to pass this lab because it’s a great career move or because it’s something I feel I need to do or even because it would show me as being at the pinnacle of my networking career.  I’m going to pass this lab just to spite Cisco and some of the dumb last-minute decisions their program mangers make.  Because I don’t know if I can handle CCIE Lab 3: CCIE With A Vengeance.