Category Archives: CCIE

My Thoughts on IOU-For-Learning

Posted on by
Reply

This week, Learning@Cisco announced a new program designed to help those people out there that want a virtualized router platform upon which to study for the CCNA and CCNP.  While the idea behind an emulated IOS platform is one that has been desired for a long time, what Cisco released today isn’t quite what we’ve been clamoring for.  The new programs use the now-famous IOS on Unix (IOU) setup that has been used internally at Cisco for a while now and was made famous by Jeremy Gaddis in this post.  This is also the same platform that is used in the troubleshooting section of the CCIE Routing & Switching Lab.

The new program is completely hosted by Cisco.  All of your access to the IOU environment is done via web and SSH.  You, as the end user, have no access to the files that comprise IOU.  Since the emulator is presented as a component of a learning package, there is no opportunity to modify the topologies presented.  They are canned and align with the courseware you purchase.  This is great for people that are just starting out in the networking world that have no access to the proper gear to learn how to enable telnet sessions and address an interface.  By limiting the access you have to a topology, you get rid of some of the confusion that surrounds tools such as GNS3, namely the dearth of options that tend to confuse the first-time users.

I have a couple of problems with what Cisco’s released so far:

1.  IOU isn’t a true layer 2 emulator.  The software that comprises IOU is great at simulating IOS running on a router.  That’s because it’s essentially an IOS image that has been modified to run on a different “hardware” platform.  So long as all you are worried about is working with routers, IOU is a great resource.  However, if you really want to dive into the second layer of the OSI model, you’re going to come up short rather quickly.  Basic layer 2 configuration is fine for a CCENT/CCNA type of student, but by the time you reach the CCNP level of switching, you’re going to find the interface of IOU wholly unsuitable.  Since IOU emulates a router, it has to emulate switching as it would be on a router with an ESM switch module.  That means that anything that relies on an ASIC to function, such as QoS, is right out the window.  Which means that some of the more esoteric and hard-to-learn parts of using IOS on a switch remain off-limits.  I’ve been able to use 16-port switching modules in GNS3 to emulate switches for some of my studies, but I quickly reached the limits of this configuration with things like advanced spanning tree configuration or specialized tasks like Storm Control.  I think that Cisco needs to put a little more effort into providing an emulated environment for switching.  Finding a way to emulate the ASICs of the QoS functions would make those learning VoIP QoS on 3560/3750 switches much happier.

2.  There’s still no proof-of-concept for engineers.  As luck would have it, I have a small lab at $employer to test some of the things customers ask me about.  It’s been cobbled together with bits and pieces of cast off equipment over the years.  Where I run into trouble are those cases where the customer has a setup that I can’t quite reconstruct with the equipment I have.  What would be nice is a kind of emulation environment that allows me to reconstruct this setup quickly.  This is the perfect scenario for something like IOU.  Being able to quickly reconstruct a customer’s environment or duplicate your own environment for things like change control and internal testing would be a dynamite idea.  By utilizing a Cisco UCS cluster with the right topology files, I could have my WAN configuration duplicated and run several sample configs for maintenance window changes quickly with the capability to roll them back if something horrible breaks.  That’s where the true power of having an emulator lies for the advanced engineer.

3.  Strict control of IOU cuts out the “gray market”.  It’s no big shock that Cisco has taken the stance with the 360 Program that you’re either with us or you’re the “gray market”.  Vendors like Internetwork Expert (INE) and IPExpert have their own courseware and rack space designed to aid their students.  These racks use real routers and switches to allow students the ability to do practical studying.  However, these kinds of study aids are prohibitively expensive for a training provider to get into.  Now, imagine if you could fire up and virtual rack of routers and switches for your students at the touch of a button.  The barrier to entry becomes much lower to those companies wishing to get involved in the training market.  The possibility then exists that you could have some bad apples in the bunch that might dilute the training offered to students and put a black mark against your name.  By holding all the cards in the IOU discussion, Cisco ensures that the technology never leaves their house, so any training partners wishing to leverage the power behind the emulated IOS platform must abide by Cisco’s rules if they want to keep playing.  Cisco can then force training partners to use 360 materials or the equivalent for CCNP/CCNA/CCENT training.  That forces the non-Cisco approved partners out of the space sooner rather than later.

Tom’s Take

Cisco’s getting to the educational platform party ahead of some of the other network vendors, like HP and Juniper, but they’re doing it with baby steps.  High level engineers have been hoping for a truly unlimited emulator for testing things for quite a while now.  I think they’re still going to be waiting for a while to come.  This new learning program is leveraging IOU to replace aging programs like the Boson Network Simulator or the NetSim products.  By tailoring it toward the entry-to-mid learner, it allows them to work out the kinks in the presentation while still keeping control over the platform for the time being.  I’ve heard that they will expand this idea to encompass security offerings and one day the CCIE as well.  I think that the IOU Learning Platform will be integrated into the 360 program and will only be offered as a part of the materials that you receive from your subscription to it.  I seriously doubt that even a CCIE-level student will have unfettered access to IOU in their own lab, since the possibility of a non-crippled version of IOU being readily available creates too many complications for Cisco support.  It’s already fairly easy to get a copy of IOU if you know where to look.  Imagine what would happen if a copy from a CCIE candidate got out into the wild without fixed configurations or limitations that you face in the hosted CCNA version?  I applaud Cisco for the steps they’ve taken in the right direction for allowing students access to emulated educational software.  Now it’s time to observe what happens and meet the needs of those of us on the other end of the scale.

If you think that Cisco needs to offer a full IOS platform for educational purposes, please head over to Greg Ferro’s site and put your digital signature on the educational IOS petition.  The more signatures that are gathered, the more pressure that can be brought to bear on Cisco to show them the will of the engineer.

For Want of a Nail

Posted on by
2

I have been weighed.  And measured.  And my configuring skills have been found…lacking.  Welcome, everyone, to the post-lab wrap up post.  Jan 20th looked like a good day for me.  I found all the faults in the troubleshooting section.  The config section looked very beatable to me.  I double checked all the configs with the hour and a half of extra time I had available.  I found some dumb mistakes that were immediately corrected.  And when I left the lab, I had a very good feeling about this one.  Alas, 58 minutes later, the score report I received indicated that I wasn’t as good as I had expected.

Without gory, rule-breaking details, I really thought I did better.  I had reachability in my lab without violating any of the constraints.  My paper was filled with two check marks next to all by one task that I was working on when time was called.  I re-read each question and physically put the point of my pencil on each word on the screen to make sure I didn’t read over anything and miss a subtle nuance that could sink me.  In the end, I think those nuances are what might have sunk me.  It’s not enough to have full reachability if you miss a little phrase that tells you to avoid a certain method or use a specific technique.  While I feel that I had accounted for all of those possibilities, the score report doesn’t think so.  And since the cold reality of the score report is more official than my high hopes, it looks like this trip to sunny San Jose was a bust as well.

After posting my results on Twitter, along with the condolences there were a lot of questions about whether or not I should as for a lab re-grade.  For those not familiar, the lab is partially graded by a script.  In cases where the script returns some strange results a proctor will look over your lab and double check certain tricky tasks.  Historically, I’ve gotten my scores after 3-5 hours.  The fact that I got this report while sitting at In-and-Out burger confused me.  Perhaps the first pass of the grading script didn’t have any issues with my configs.  One or two people even suggested that as a good sign that I might have even passed from the script alone.  So, as I stared at the percentages in front of me, I contemplated something I had never tried before.  I wanted them to take another look at my exam.  I wanted another proctor to take the configurations saved from my equipment and load them up onto new routers and regrade my exam by hand.  This process is not without it’s downsides, though.  First, it costs $250 for them to even do it.  If you get re-graded to a passing score, there is no refund.  Secondly, there is as much possibility of you losing points as there is passing.  This is because the human eyes of the proctor may catch something incorrect that the script missed.  So, you may have only been 3 points from passing, only to now find that you are 8 points away due to some other mistakes.  However, if you are very close to the mark, as in ~2-3%, there is no reason not to take the chance on the regrade.  After all, it’s still cheaper than a new $1400 lab attempt, right?  Third, the regrade attempt takes up to three weeks.  So if you’re hoping for a fast turnaround before deciding to take another lab attempt, you’re going to have a cool your jets for the better part of a month.  On the bright side, Cisco is usually very understanding and will refund the $1400 if you’ve booked another lab attempt and your re-read is honored and changed to a passing score.

Turns out, Cisco won’t even look at tests that are not “statistically close to the historical passing rate.” English?  If you aren’t within about 5% of the passing score, you don’t even have the option to have it regraded.  And, at least in Cisco’s eyes, I was wide of the mark this time.  I’ve filed a case with support to at least be granted the option for the regrade.  We’ll see what happens.  In the meantime, I’m getting back on the horse.  I’ve already had conversations with my employer about when and if there will be another attempt on their dime.  I’m not packing away any lab equipment for the time being.  I’m getting right back into the thick of things to keep it all fresh.  In fact, my wife and I went to the final closing party for a local pub that we’ve been going to since college.  As I sat there surrounded by all the merrymakers and noise of a bar full of people, all I could think of was how much I wanted to get back home and start labbing up some multicast questions.  It seems that the lab has transformed me and sharpened my focus into laser-like precision.  I think about how I’m going to configure tasks and how I’m going to move efficiently from one task to the next.  I pace myself based on the amount of work I can get done in 6 hours with a 40-minute lunch break.  I review flash cards and GNS3 configs during my free time.

No matter what, I’m going to go back and try this again.  I feel like the top of Everest is in sight and I just need one last push to reach the summit.  I’ve come too far to fail now.  More studying, more refining of my task config skills, more “stick time” for this airplane of router and switch configuration.  For want of a nail, this lab was lost.  But the next time around, the only nail will be the lab being nailed by me.

Avoiding Proctor Proctology

Posted on by
Reply

Other than perhaps professional sports referees, I can think of no more maligned group of individuals than the proctors in the CCIE labs.  The myths about them are legend.  They screw up your rack at lunch.  They do everything they can to make you fail.  They act obtuse when you ask questions because they don’t like passing new CCIEs.  For the most part, all the mystery and nastiness that surrounds the proctors is just a bunch of hokum.  They are people just like you and I.  They do their jobs just like you and I.  It just so happens that their job is closely identified with something that most people study for hundreds of hours to achieve, and when those people fail, they look to cast blame on others.

I’ve been to the lab once or twice.  I’ve met several proctors both there and at Cisco events.  People who only need to be known by first names.  People like Howard and Tom and Stefan.  And each time I encountered them outside the lab setting, they impressed me with their poise and charm and sense of humor.  Just like any position of authority, the CCIE lab proctor has a role to play once they step inside the walls of the lab.  They must project an  aura of authority and calm.  They must provide guidance where confusion reigns.  And above all, they must protect the integrity of the program.  When most people deal with them in this setting, they come off cold and uncaring.  Just like a judge or a police officer, it’s important to remember that the position of proctor is a role that must be played.  Keeping some things in mind before you step through the badged door will go a long way to making your proctor experience a smooth one.

1.  Be nice. Yes, it sounds silly, but it’s a very important thing to keep in mind.  People are amped up when the walk into the lab.  They are stressed and wired and strung out.  Some people retreat into a shell when stressed.  Some people lash out and are irritable.  What’s important to keep in mind is that when dealing with the proctor, you should keep a calm and even demeanor.  People tend to respond in kind with the emotions they are presented.  If you walk over to ask a question and are short and snippy, expect a short answer in return.  However, if you are nice and pleasant, it can go a long way toward getting a favorable answer quickly.  When you go to lunch, don’t be afraid to engage the proctor in some light conversation.  Talk about the weather or the food or a sports team.  Anything but the lab.  Engaging them outside the walls will give you a feel for how they react to things and can help you judge if they will be helpful when it comes time to utilize them.  Besides, it never hurts to be friendly.

2.  Ask ‘yes’ or ‘no’ questions. The most popular complaint I hear about proctors is that they never answer the question that you ask them.  Before you stand up to go ask for clarification, carefully word your question so that it can be answered with a simple ‘yes’ or ‘no’.  For instance, rather than asking “How should I configure this frame relay connection?”, instead ask “There are two ways two configure this frame relay connection, point-to-point and point-to-multipoint.  The question isn’t clear about which I should use.  I’m leaning toward configuring it as a point-to-point.  Would I be correct in this assumption?”  The proctor may still choose not to answer your question, but you’re more likely to get a binary yes/no response out of them.  By showing you understand the technology and you aren’t just trolling for an answer, you’ll appeal to their interpretive skills.  Remember that the proctor won’t give you an answer, no matter what.  They have too much riding on their job, reputation, and the CCIE program to ‘bend’ the rules for one candidate.

3.  Don’t assume a problem is something you can’t fix. Want to piss off a proctor really fast?  Walk up and say, “I can’t get OSPF to come up.  My rack must have a cabling problem.”  I promise that after the eye roll and gruff answer, you’ll fail the lab.  Candidates who are weak in troubleshooting skills tend to blame the physical layers first because it’s the one layer they can’t touch in the lab.  In the old 2-day lab, cabling could be an issue to resolve.  But in the 1-day lab, with cabling extracted from the candidate’s control, there is a 1-in-1,000 chance that the cabling is at fault.  Think about it like this: the lab equipment supports 3-5 candidates 5 days a week for 50 weeks a year.  Cabling issues will be caught and dealt with quickly.  Simple things like bad ports or bad cables will be caught when the lab racks are booted first thing in the morning.  You going to the proctor and claiming that OSPF is broken because your routers are wired backwards will only serve to irritate the proctor.  What will happen next is that the proctor will ask you to wait by his/her desk or make you wait in the RTP conference room.  They will go to your rack and start troubleshooting OSPF.  They’ll find out you misconfigured a network statement or forgot to enable authentication.  They’ll check to make absolutely sure it’s not a layer 1 problem, all while you are isolated away from your terminal.  Then, after 15-30 minutes, they’ll come get you and tell you, “It’s not a physical problem.”  That’s it.  No other information.  You won’t get to see what they did to find your problem.  You won’t get any hints about how to fix the issue.  And you won’t get those 15-30 minutes back.  Period.  On the other hand, if you go to the proctor with a long list of reasons why it has to be a layer 1 problem, like BERT or TDR output or a down/down physical interface that won’t come up at all, they’ll be more receptive due to your troubleshooting efforts.  And if they find a layer 1 fault during their troubleshooting, you’ll probably get the time back from their efforts and the time it takes to replace the faulty unit.

4.  Don’t cheat. Well, duh.  You’d think that should be a given.  But people still try to pull stupid things all the time.  And not just the braindumping.  Writing things down and then trying to sneak them out on the scratch paper, of which every scrap must be accounted for.  Trying to use a cell phone to discreetly look up answers.  Looking at other screens (for all the good it’ll do you).  Just don’t cheat, or even give the appearance you’re cheating.  That should keep the proctors from getting quite nasty with you.

In the end, just remember that these guys and gals are doing their jobs as defined by the CCIE program.  They work hard and do everything within their guidelines to help you pass.  They aren’t out to screw you, and if you keep your head about you and use some common sense and manners, they’ll be the best resource you’ll find in the lab.

Hooray for Bruno!

Posted on by
2

Twenty-four hours after Cisco ‘tweaked’ the CCIE lab troubleshooing section a little, we finally get a little bit of feedback about what they have wrought on the weary CCIE candidates.  This thread over on the Cisco Learning Network serves as the official announcement.  Further down the list is a response to some questions from concerned folks about what this could mean from Bruno van de Werve, who appears to be part of the CCIE program inside Cisco.  Bruno was also responsible for the excellent CCIE lab web interface video.  So, what did Bruno share with us?

1.  This is a virtual switch based on a router image, so the ports will be shutdown and not configured as switchports up front.  This about having a switch module in a router in Dynamips/GNS3 and I’m sure this is what the L2IOU is like.

2.  The image appears to be based more toward the 3550 rather than the 3560.  This comes from comments about the port being in dynamic desirable instead of the 3560 method.

3.  All interfaces are Ethernet, not FastEthernet or GigabitEthernet.  They are also arranged in groups of 4, e.g. e 0/0, e 0/1 and so on.  Due to the software limitations, you can only do interface range commands across the 4 ports of the module, so for instance setting up more than 4 trunks at once would require multiple interface range commands.  As an aside here, Bruno says that they are most likely not going to have more than 4 trunks per switch right now.

4.  These switches are based off of 12.2 mainline IOS with some L2 switching features added in.  They also lack any of the hardware ASICs necessary to do advanced features.  So, no Cat-QoS for instance (His words, not mine.  Cat-QoS, really???)

5.  There are only two switches at present in the TS section, and they are in the same IGP domain.  Bruno elaborates that they might add more switches later as features are added to the L2IOU image.

6.  There will still only be 10 questions in the TS section (his words).  So, at best you can probably expect 1-2 L2 questions for now.  He did say that there is a possibility that the questions could influence other tickets, so be on guard if you have a L2 question up front that it might affect an OSPF question later.

7.  There are no planned changes to the configuration section of the lab at this time.  No, they aren’t going to remove MPLS.  They aren’t screwing your config section totally.  Also, in a reply to one of the first commenters, Bruno said,

“At the moment, there won’t be any changes to the configuration section of the lab, so the two L2 troubleshooting questions will still be there.”

Ahem.  As my old econ professor in college used to say during test reviews…”Hint, hint, hint, oh hint.  You might see this on the exam.”  Guess what folks? Bruno just gave you a big hint.  You are still going to see L2 troubleshooting on the config section.  And if the historical trend is true, it’s going to be the first part of your lab.  “There are XXX faults in your initial lab configuration.  Correct these faults for 1 point each.  Be aware that these faults could impact configuration in other sections of the lab, and if they are not resolved and impact another working solution, you will not receive points for the non-working solution.”  So put your thinking caps on.

My Take

L2IOU was introduced to the troubleshooting section to crack down on the braindumpers.  Cisco knew they needed to add it to keep the cheaters off-balance, but the feature set of the IOU emulator wasn’t ready when the TS section went live.  I’m sure a team has been working on it night and day trying to get enough features built into it to make it a usable tool for the TS section.  It would be virtually impossible to have a physical setup for troubleshooting.  30+ routers would be a large rack, and now that you start adding physical switches in on top of it, it become unmanageable.  IOU made the most sense for the L3 section, but L2 is just as important for testing a candidate’s knowledge.  However, there wasn’t an emulator capable of perfectly virtualizing a switch.   And there still isn’t.  The fact that L2IOU got pushed out the door in the state that Bruno describes it in means that someone had to be done to stop the braindump hemmorage.  That’s the only reason I can think of to introduce a totally foreign software-emulated L2 device that by all accounts doesen’t resemble the feature set of the devices in the lab that you are already forced to troubleshoot a mere two hours after the beginning of the lab.  So why not put the additional troubleshooting questions in the config section where they can be done on real hardware?  Especially since there are only two L2IOU devices currently?

Because Cisco has never nailed down the exact nature of the TS section, it gives them more latitude in changing it quickly.  Adding ten new TS questions takes a lot less time than re-engineering a whole lab set.  Just like the OEQs before, the TS section is designed to weed out the weak candidates relying on someone else’s leaked materials rather than the tried-and-true studying methods that should be employed.

I still don’t like the change.  Not enough lead time.  A ham-handed attempt to fend off what appears to me to be a growing portion of lab candidates.  Seeing as how they appear to be phasing things in slowly and not increasing the impact of L2 until more features get baked in, I guess I can live with being a guinea pig for now.  But I sincerely hope that I pass my lab on this attempt so I don’t have to beta test new TS section features again.

Layer 2 – Electric Boogaloo

Posted on by
2

In alignment with CCIE level requirements, Cisco is adding L2 switching features to the CCIE R&S Troubleshooting exam through L2 IOS software on Unix (L2IOU) virtual environment. The new feature will be available starting January 17, 2011. The CCIE R&S exam consists of 2 sections b the troubleshooting (TS) section which runs two hours, and the Configuration (Config) section which is six hours. The Config lab utilizes actual physical devices in racks, whereas, the TS lab uses a virtual environment under IOU. IOU offers a very realistic simulation of router (L3+) features in the TS lab but until now had no L2 switch capability. With the addition of the new L2IOU, the TS lab will now include both L2 and L3 capabilities in the virtual environment.

Official Cisco Announcemnt (Thanks to Rob Routt for the text)

So, it appears that Cisco has finally added layer 2 (L2) troubleshooting to the first troubleshooting section of the lab.  Hmmm…

Ladies and gentlemen, a rant…

What?!?! Really?  Why did this have to happen four days before my exam!  Okay, it’s not fair to think that you’re singling anyone out with these changes, but when you added the open-ended questions (OEQ) in 2009, you did it the week before my second lab.  And that little change with the OEQs cost me one lab.  With luck like this, anyone going to Cisco Live Las Vegas 2011 should head down to the casino with me and bet the exact opposite of what I say. You’ll make millions.

I know the standard response.  Anything is fair game for troubleshooting on the TS section.  If I’m a prepared CCIE candidate, I should have no fear about this addition.  Everything will be fine, I’ll nail it.  All of these things are true, but it still doesn’t make me feel any better.  This is like going in to take a driver’s test only to find that the steering wheel is on the wrong side of the car!  Does it affect the test?  Not really, but it is a head scratcher.

If all I have to worry about is layer 3 faults, I don’t have to consider spanning tree or trunk ports or Q-in-Q tunneling or QoS mismatches or backup interfaces or Etherchannel misconfigurations or any one of a number of things.  No, adding L2 to the TS lab doesn’t make it necessarily harder, but it does broaden the range of things I have to think about when I start looking at a problem.  Irritating, this is.

Why the need to cram so much stuff into the beginning?  You know, you always did well with the first TS question of the config section – Hey, we broke three or four things in your lab that you need to fix before you get started.  You don’t get any points if they aren’t fixed, and since they probably are going to break other things, you won’t get points for other questions either.  Guess breaking things on five routers and four switches isn’t nearly as fun as doing it to 30 routers and who-knows-how-many switches.  I’m sure that the setup of the TS lab allows them to break lots and lots of interesting things in ways they can’t simulate in the config section.  But if that’s how things are going to be going forward, they need to remove any TS from the config section too, if it still exists.  After all, there are no sim questions on the CCIE written because they are covered in other sections.  Why should there be troubleshooting on the lab config section when you’ve so eloquently covered it in the TS section?  I figure that once the L2 TS stuff gets baked in, we might actually see this happen.

And another thing…one business day notice?  Really?  Come on now.  That’s just unfair.  Used to be, changes to the lab required six months of notification before going live to allow students the opportunity to study up on them and be prepared.  How do you think the folks taking the lab bright and early Monday morning are going to feel?  I guess that adding a whole new topic to the TS section doesn’t constitute a ‘major’ change anymore.  Or perhaps because you intentionally left the description of the TS section vague, it allows you the opportunity to ‘clarify’ it with little to no warning.

I’ve got a question, Cisco.  Are the braindumpers really hurting you that bad?  You can tell me all you want that the OEQs and the TS section were designed to better assess a candidate’s knowledge of every facet of networking.  With respect: bullshit.  The OEQs were a direct outgrowth of the interview process implemented at a specific testing site to catch people memorizing leaked lab questions and getting through with no difficulties.  The TS section replaced the OEQs once the process was refined to the point where they were no longer needed.  Have people started memorizing the TS section too?  Is it bad enough that we’re going to have a new exam every week in an effort to catch the unworthy candidates?  Perhaps we should just move to a totally interview-based exam with no access to CCO or documentation of any kind?  After all, the perfect CCIE candidate should be able to configure and troubleshoot a lab without access to a keyboard or a mouse or even a monitor!

Okay, I think that’s enough ranting for one morning.  I’m at T-minus 141 hours and counting.  That should be just enough time to brush up on my L2 TS skills in addition to all the other stuff I’m cramming for.  You know, I’m just about to the point where I’m not going to pass this lab because it’s a great career move or because it’s something I feel I need to do or even because it would show me as being at the pinnacle of my networking career.  I’m going to pass this lab just to spite Cisco and some of the dumb last-minute decisions their program mangers make.  Because I don’t know if I can handle CCIE Lab 3: CCIE With A Vengeance.

Frame (Relay) of Reference

Posted on by
6

A while back I wrote a CCIE lab-related post about RIP and why it’s still on the lab.  Most of the questions that I see lately revolve around another old technology and the curiosity of why it’s still contained in the vaunted lab blueprint.  I speak of frame relay.  The much-maligned WAN technology that no one seems to be able to explain the reason for existing anymore, yet the weary lab candidates find themselves pour over at the last minute.

Frame relay has been around forever.  And slightly shorter than forever, it’s been on the CCIE lab blueprint.  I’ve been actively studying for my exam since July 2008.  And as far back as I can remember, I’ve been studying frame relay.  And ever since that time people have been asking why it’s even on the exam.  I’ve heard people say it’s too old.  It’s past it’s prime.  No one offers it anymore.  Recently, with the change to the version 4 lab exam, MPLS was introduced as a configuration topic.  We all thought “Surely frame relay will be gone with its apparent successor on the lab now…”  And yet, frame relay is still there.  In fact, they added frame relay switching back onto the blueprint.  That was a huge change for me, never having dealt with that side of things before.  So, here is frame relay configuration of all kinds taunting us with is outdatedness.  Forcing us to keep up with our grandfather’s WAN technology.  And, in my opinion, that’s just the way Cisco likes it.

Just like RIP, frame relay wasn’t necessarily meant to be on the exam as a test of your ability to map IPs to DLCIs.  Frame relay introduces a whole level of configuration possibilities to the lab to test your wits without being obvious.  The whole point of keeping it on the lab isn’t to make you expend effort in getting 2 or 3 points.  It’s laying a foundation that could cost you 10 points or more.

I remember reading the Building Scalable Cisco Internetworks (BSCI) routing exam guide a few years back studying for my CCNP.  It was my first real exposure to the depths that routing protocols could go to.  In one the chapters over OSPF, there was a section that covered a ton of information on running OSPF over non-broadcast multi-access networks.  There were tables and charts and graphs galore.  All dedicated to one sub-topic.  Why was that?  Because the intricacies of running OSPF over frame relay are legion.  There are multiple network types that determine DR and BDR functionality and hello timers.  It’s like a mix-and-match sale at the department store.  So many fun combinations from so few parts.  That chart was just the tip of the iceberg, though.  Because once you’ve gotten a frame “cloud” in your lab, the real fun can start.

Suppose you have a really simple question in the lab like this:

Configure frame relay between R3 and R6.  Do not use static
mapping or inverse ARP. (2 points)

Straightforward, eh?  You could probably nail up this frame configuration in about 5 minutes.  Well, think about the rest of this example lab and find all the things that rely on Frame Relay to award points.  Let me give you a few examples:

Configure OSPF on all routers indicated in the diagram.  
Between R3 and R6, use a network type the provides for the
fastest recovery times without a DR/BDR election. (3 points)

Configure the link between R3 and R6 so that routing protocol
packets are never dropped.  If the queue has more than 20 packets,
reduce the bandwith to 32kbps. (3 points)

Configure the link between R3 and R6 to authenticate using CHAP.
(2 points)

Now, I’m not saying that all of those questions could end up in your lab.  But think about it like this:  That single 2-point FR question just became a 10-point chunk of your lab.  Now, if you don’t configure your FR section correctly, you are halfway to failing the lab from one misconfiguration!  Remember, no partial credit means that even if your whole OSPF section is right, failing to get FR working correctly means the script used to grade your exam won’t see all the routes on R3 and R6, and so you won’t get any points for OSPF.  It also means that your QoS won’t work, so there goes three more points.  And so on.  Soon, you could fail the lab simply because that outdated technology you didn’t really care about sucker punched you.

I’ve always held the belief that the CCIE lab is designed in a way to test every facet of being an network superhero.  Not just the obvious things like OSPF configuration or BGP troubleshooting, though.  As I’ve said before, the lab manual is carefully written by a team of highly competent people.  There are no extraneous words, and there are no unimportant tasks.  It’s like a game of Jenga.  The things you do at 10:00 a.m. have a direct impact on the last task you do at 4:00 p.m.  If your lab isn’t built carefully and solidly like a Jenga tower, it will topple down on the desk in front of you and knock your cup of colored pencils and markers onto the floor (metaphorically, of course).  Like it or not, frame relay is one of those blocks that forms the foundation of your tower.  Not because Cisco is protecting their investment in legacy technology.  Not because they are trying to wear lab candidates out with archaic configuration tasks.  Frame relay still exists on the blueprint because it teaches candidates to get their configurations right in that critical first hour of the lab because the Jenga tower that is your lab exam depends on each and every part.  And since frame relay can touch so many parts of the test, Cisco likes to keep it around to make sure you’re paying attention.

So the next time you find yourself staring at the blueprint and asking yourself “Why does Cisco still put frame relay on the lab?” you might try asking instead, “What does Cisco want me to learn from configuring frame relay?”  I think you’ll find the answer to the latter question a lot more enlightening.  With all of the things that depend on a simple frame relay link, one miscue could be the difference between a detailed score report and a simple 4-letter one.  Because from Cisco frame of reference, frame relay isn’t just a simple test question.  It could be one of the most important topics on your exam.

I’s and T’s and Crosses and Dots

Posted on by
Reply

My name is Tom, and I’m careless.

Yep, I admit it freely.  I’m the kind of person that rushes through things and gets the majority of the work done.  Often I leave a few things undone with the hope that I’ll go back later and fix them.  For me, the result is the key.  Sometimes it works out in my favor, sometimes it doesn’t.  More often than not, I find myself cursing out loud about this unfinished job or task months down the road and threatening to find the person responsible, only to later determine that I should be kicking my own butt for it.

One place where this particular habit of mine has caused me endless grief in inside the unforgiving walls of Cisco’s Building C lab in San Jose.  Yep, I can honestly say that at least one lab attempt was foiled due to my propensity to miss the little things.  I’ve previously written about some of the details of the lab, but I wanted to take some time in this post to talk about the details themselves.  As in, the details in the questions that will kill you if you give them the chance.

Let’s get it out there right now: there is NO partial credit in the CCIE lab.  None. Zilch.  If you fail to answer every portion of the question with completeness, you get zero points for that question.  Unlike the old days in elementary school, you don’t get points for trying.  This shouldn’t really come as a shock to anyone that’s taken a multiple choice test any time in their life.  On those tests, there is exactly one set of answer(s) for a particular question, and if you don’t select the proper repsonse(s), you don’t get the points.  The same thing goes for the questions you find in the CCIE lab exam.  Just because the questions may or may not have multiple parts doesn’t excuse your need to answer them fully.  Old Mr. Hollingsworth used to tell me regularly, “Son, close only counts in horseshoes and hand grenades.”  Since I don’t play horseshoes and my hand grenade supplier mysteriously dried up, I guess close just won’t cut it any more.

You might end up getting a question in the lab that says something along the lines of “Configure OSPF on R1, R3, and R6 according to the diagram.  Do not change router IDs.  Rename R1 to ‘SnugglesR1’.”  You could build the most perfect OSPF lab in history.  You could spend an hour optimizing things.  If you forget to rename Snuggles the Router, you will receive no credit for the question.  All that hard work will get flushed down the toilet.  You’ll get your score report at the end of the day and wonder why you didn’t get any points for all that time you spend making OSPF sing like a soprano.

In order to prevent this from happening to you, start training yourself now to read carefully and consider every facet of the questions you’ll see.  Remember that the questions in the lab are carefully constructed by a team that spends a ton of time evaluating every part.  There are no unnecessary words.  Candidates have pestered proctors over the meaning of single words on a question.  The questions are written as they are to make sure you take into account a number of factors.  They are also designed to slip in changes to tasks and additional configuration with a word or two.  And if you are careless, you’ll miss those phrases that signal changes and negations.

Surely, everyone has taken a test that has a question that says “Which of the following was NOT a <something> <something>”  Your job is to evaluate the choices and pick the one that is not something.  That single word changes the whole meaning of the question.  And for those that are careless or the kind the skim questions, the NOT might be missed and cause them to answer incorrectly.  Questions in the lab are the same way.  Skimming over them without reading critically can cause nuances to be missed and lead to incorrect solutions.  After 5 hours of staring at words on a monitor, things might start blurring a little, but attention must be paid to the last few questions, as those might be enough points to buoy over the passing mark.

I’ll be the first to admit that the pressure to get everything done in the allotted time may cause the candidate to want to rush, but you must resist that pressure.  Many CCIE lab prep courses and instructors will tell you to carefully read the questions before you ever start configuring.  I agree, with some additions.  I always take my scratch paper and write the task numbers down the side.  After I’ve accounted for Task 1.1, 1.2, 2.1, and so on, I then go back to the questions and make marks next to my list for any questions that may have multiple parts or tricky solutions.  That way, if I find myself rushing through after lunch the marks I made early in the day force me to pay attention to the question and ensure that I don’t miss something that might cause me to tank three or four points.  Those points add up over the course of the day, and more than a few careless mistakes can cost you a nice expensive soda can.

If you are serious about the CCIE lab, it’s worth your time to start working on ensuring that you pay close attention to each question and don’t make any careless mistakes due to reading too fast or missing important configuration requirements.  Your day is going to be stressful enough without the added pressure of fixing mistakes later in the lab as a result of forgetting to enable OSPF authentication or a typo on a VLAN interface.  You want to remember to dot every “i” and cross every “t” for each and every question.  That way, you can walk out of the lab and use that freshly-dotted “i” when you spell you new title as a CCIE.

2011 – Looking Forward

Posted on by
1

I almost wrote an end-of-year recap for this particular blog post.  I thought back to all of the things I’d accomplished over the past year.  It didn’t take me long to realize that I didn’t really keep track of them as well as I should.  The other thing that changed my mind was Greg’s great post about looking forward.  I’ve only been blogging for about 3 months.  I’ve really only had an online presence for about half the year.  So recapping what I’ve done wouldn’t really do much to help me take stock of what’s been going on.  But I’ve been trying to codify some things that I’m looking forward to in 2011 and I thought that putting them down in print would be a great way to make me own up to them.  So, without further ado, here’s what I’m looking forward to for the next 365 days.

1.  Passing the CCIE R&S lab. We are quickly getting to put-up or shut-up time when it comes to my CCIE lab.  I know that I’ve only failed when I decide to quit trying, but the trying is really starting to smart.  I’m in a unique position amongst some of my peers, in that my employer has been very gracious in allowing me to keep attempting the lab.  But I’m starting to feel like I’m imposing on their goodwill.  I’m starting to see a lot of RFPs being released that are requiring CCIE credentials to design what are essentially enhanced layer 2 networks.  I realize that these RFPs have been crafted in some degree to lock my employer out of consideration in the bidding process.  My pride tells me that I want to pass the lab for no other reason that to fly a big middle finger to them, as if to say “Ha! Guess what I did?!?”  In the end, I want to really succeed here because I’ve never let any test beat me, save one.  And I’m not about to let the CCIE become the second.

2.  Upgrade my VCP to version 4. The other thing that I do a lot of at my job that doesn’t revolve around networks concentrates on VMware.  I work with it more than I do with the actual OSes that get loaded to it, and I think it’s about time I made the move to getting certified on the current version.  There are some interesting possibilities that await should I manage to get there, including the idea of getting the VCAP4 – Design.  My job focus is quickly moving on toward building networks and systems on paper rather than physically, so some more designed-focused learning would do me some good.  But first things first.  I’ve got to get with the now.

3.  Start looking at the CCIE Voice. Heh, compared to #1, this one looks kind of silly.  Why start looking at another CCIE track when you aren’t even done with the one you started with?  If the truth be told, I’ve stuck with R&S as long as I have because of my stubborn streak.  I don’t work with BGP or MPLS in my every day job.  I doubt I ever will unless I switch roles and/or employers.  But I deal with voice every day.  It’s not what I started out to do, but I find it interesting.  And so I’m thinking that I might consider looking at some of the Voice courses and whether or not they appeal to me.  Who knows?  Maybe Voice will be an easier lab for me?

4.  Wikify my documentation. I’ve been putting this off for a lot longer than I should.  I need to take all of the information that I’ve gathered that resides on my laptop and put it into a form that other people can use and edit easily.  I want to have all of my knowledge in a place my peers can get to so that they might find the information they need quickly.  I want to clean up my haphazard note-taking style and make it readable.  I also want to be able to disappear for a few days at a time without getting ten phone calls and tons of e-mails.   I want to be able to pass the Bus Test.

5.  Start teaching more. Part of the reason that I started this blog was to collect all the random things that I come across and write them down in a place that I could easily find.  As an ancillary objective, I hope that other people might benefit from my research and study so that they could avoid the mistakes I’ve made.  I’ve considered bringing that into something a little more formal.  Some of my old college professors have talked to me about speaking to student groups.  My boss has discussed having me train user groups and train-the-trainer type scenarios.  I look at it as a two-fold opportunity.  I get to disseminate my knowledge, but I also gain the ability to tighten my presentation skills and put a little polish on my approach.  I don’t want to end up as a curmudgeon that sits behind a keyboard all day and loses all social ability.  I figure that forcing myself to get out and speak to people might just do that.

I figure five things should be a good list to work on.  Especially since  #1 is going to consume a lot of my time.  I hope to look back on this in 52 weeks and check off a few things.  I also hope that I can add a few more items to the list as I go.  Because surprises are always a good way to keep your edge sharp.

Twelve Days of Christmas Networking

Posted on by
1

In the spirit of Christmas, and because my wife has made me listen to the song about 400 times so far this year, I present the Twelve Days of Christmas, Networking Nerd style.  To save you all the trouble of singing the whole song, we’ll just skip to day tweleve.  On the Twelfth Day of Christmas, the Networking Nerd gave to me:

– Twelve character passwords

– Eleven 802.11n Access Points

– Ten Gigabit Ethernet

– Nine 9971 phones

– Eight-port switch blades

– Seven CCIE Tracks

– Six Hours in the CCIE Lab

– Five Magic Digits! (I hope…)

– Four-port FXOs

– Three Packet Pushers

– Two L2MP options

– And One Goal: To get my CCIE!

Special Thanks to JT (@WannabeCCIE) for giving me the idea for this.

Merry Christmas to all the folks out there.  May your holidays be filled with joy and caring.  May your families not drive you insane, and may your Christmas stocking be filled with all the goodies you asked Santa for.

 

Bluntly BPDU–The Redux

Posted on by
5

Firstly, you should read this blog entry.  Go ahead, I’ll be here when you get back.

All done?  Good.  Now, for the first part of my response.  Mea culpa. I fracked up.

Based on this document: http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.1E/native/configuration/guide/stp_enha.html#wp1033403 I assumed an incomplete and incorrect stance on the way BPDUFiltering works.  This guide says that when a BPDU is received on a Portfast port, the port loses its Portfast state, which disables the BPDUFiltering and it begins acting like a regular STP port again.    However, thanks to Santino Rizzo for gently pointing out that BPDUFilter when enabled on an interface completely strips the BPDUs from being received and transmitted.  When enabled globally, it performs as I originally assumed, with the ports being transitioned out of Portfast and into regular STP ports.  I think the differences in my understanding come from the fact that the linked guide above is only for CatOS, which only allows BPDUFilter to be enabled globally.  The reference documents for interface-level BPDUFilter (like this one: http://www.cisco.com/en/US/docs/switches/lan/catalyst4000/7.4/configuration/guide/stp_enha.pdf) state correctly that it strips BPDUs from being transmitted at the interface level.  Okay, with that out of the way, on to the meat of my post…

When I started my CCIE walkabout 3 years ago, I had the good fortune to attend one of Narbik Kocharian’s boot camps.  One thing he told me there sticks with me to this day.  When we got into a discussion about the behavior of a certain protocol or configuration, he would always fire up a live router or switch to test our theories.  He told us, “IOS never lies.”  And he’s right.  IOS always does what you tell it to.  And it does it pretty much every time.  The same configuration gives you the same output.  So when we need to test something, we can jump into IOS and get out answer.  Because it will always tell us the truth.  That was my failing last night.  While I had been thinking of the discussion on Saturday, I didn’t take the time to lab it up.  So, I took my afternoon to do just that.

Let me state the parameters of my lab experiment.  I used two 3560 switches, one 24-port and one 8-port.  Both were running 12.2(25)SEE IOS, the 24-port was running the enhanced image, and the 8-port was running the base image.  I wiped the configuration of the two switches before starting and only configured a hostname on the 24-port switch.  The 24-port switch served as my test bed, and was the root of the spanning tree for VLAN 1, the only VLAN configured on the switches.  The only configuration done on the 8-port switch was to shutdown the connecting port between tests, and then re-enable it when I wanted to observe the output.  Otherwise all testing was done on the 24-port switch.  And now, I give you the results:

1.  No Configuration

I started out with a control of no configuration under the port at all.  This would give me a good baseline to work from.  I plugged the switches in and enabled the ports.  I then used the show spanning-tree interface fa 0/1 detail command to see the BPDUs.  Here’s the output from that test:


 Port 3 (FastEthernet0/1) of VLAN0001 is forwarding
   Port path cost 19, Port priority 128, Port Identifier 128.3.
   Designated root has priority 32769, address 0019.2f4d.4580
   Designated bridge has priority 32769, address 0019.2f4d.4580
   Designated port id is 128.3, designated path cost 0
   Timers: message age 0, forward delay 0, hold 0
   Number of transitions to forwarding state: 1
   Link type is point-to-point by default
   Bpdu filter is enabled internally
   BPDU: sent 44, received 1

Okay, so that’s what we expected.  BPDUs are being sent from the root bridge.  The switch has received one BPDU from its partner.  This is pretty much what happens straight out of the box.  now, let’s turn on Portfast.

2.  Portfast

I enabled access mode on the port with switchport mode access so that I could enable Portfast on the interface with spanning-tree portfast.  Afterwards, I enabled the port on the other switch and checked the output:


 Port 3 (FastEthernet0/1) of VLAN0001 is forwarding
   Port path cost 19, Port priority 128, Port Identifier 128.3.
   Designated root has priority 32769, address 0019.2f4d.4580
   Designated bridge has priority 32769, address 0019.2f4d.4580
   Designated port id is 128.3, designated path cost 0
   Timers: message age 0, forward delay 0, hold 0
   Number of transitions to forwarding state: 1
   The port is in the portfast mode
   Link type is point-to-point by default
   Bpdu filter is enabled internally
   BPDU: sent 20, received 1

Again, as expected.  The port is still sending BPDUs and receiving them.  The difference here is that the switch is now bypassing the blocking, listening, and learning phases and begins forwarding traffic immediately.  Bad thing if there is a loop behind it.  Okay, so the behavior is as expected so far.  Now, let’s start turning on the BPDU protection stuff.

3. BPDUGuard enabled per interface

For this test, I wanted to see what BPDUGuard does when enabled on the interface:

 


00:18:48: %SPANTREE-2-BLOCK_BPDUGUARD: Received BPDU on port FastEthernet0/1 with BPDU Guard enabled. Disabling port.

00:18:48: %PM-4-ERR_DISABLE: bpduguard error detected on Fa0/1, putting Fa0/1 in err-disable state

TestSwitch#sh spann int fa 0/1 det

no spanning tree info available for FastEthernet0/1

TestSwitch#sh int fa 0/1

FastEthernet0/1 is down, line protocol is down (err-disabled)

As soon as the switch received the BPDU from its partner, it slammed the door shut and placed the interface in err-disable.  No STP info because the port is essentially shutdown.

4.  BPDUFilter enabled per interface

After clearing the err-disable state, I removed the BPDUGuard configuration and ensured the port was no longer err-disabled.  I then enabled BPDUFilter on the interface with spanning-tree bpdufilter enable.  Here’s what I saw when I ran the spanning tree interface command:


 Port 3 (FastEthernet0/1) of VLAN0001 is forwarding
   Port path cost 19, Port priority 128, Port Identifier 128.3.
   Designated root has priority 32769, address 0019.2f4d.4580
   Designated bridge has priority 32769, address 0019.2f4d.4580
   Designated port id is 128.3, designated path cost 0
   Timers: message age 0, forward delay 0, hold 0
   Number of transitions to forwarding state: 1
   The port is in the portfast mode
   Link type is point-to-point by default
   Bpdu filter is enabled internally
   BPDU: sent 0, received 0

We now have STP information about the port, but no BPDUs being received or transmitted.  This is consistent with Santino’s description of the way BPDUFilter performs on an interface.  If you plugged a switch in now, it would see itself as the root bridge for each of it’s VLANs, as it isn’t seeing any BPDUs coming from its ports.  This would be the strange behavior I had seen with BPDUFilter when I had configured it in production earlier.  Makes sense to me now.  Let’s see how bad we can screw things up when we start combining things.

5.  BPDUGuard enabled globally, BPDUFilter enabled per interface

Setting the interfaces back to default, I then enabled BPDUGuard globally with spanning-tree portfast bpduguard default. I then enabled BPDUFilter on the interface with the previous command.  Here’s what I saw:


 Port 3 (FastEthernet0/1) of VLAN0001 is forwarding
   Port path cost 19, Port priority 128, Port Identifier 128.3.
   Designated root has priority 32769, address 0019.2f4d.4580
   Designated bridge has priority 32769, address 0019.2f4d.4580
   Designated port id is 128.3, designated path cost 0
   Timers: message age 0, forward delay 0, hold 0
   Number of transitions to forwarding state: 1
   The port is in the portfast mode
   Link type is point-to-point by default
   Bpdu guard is enabled by default
   Bpdu filter is enabled internally
   BPDU: sent 0, received 0

In this case, the inteface specific configuration overrrode the global config.  And the interface never received a BPDU to trigger the BPDUGuard function, so it remains in the forwarding state, albeit with no STP information on that port.  Let’s reverse it.

6.  BPDUFilter enabled globally, BPDUGuard enabled per interface

Back to default interfaces and removal of all global BPDU protection.  Now, I enable BPDUFilter with spanning-tree portfast bpdufilter default and enable BPDUGuard on the interface.  What do you think will happen?


TestSwitch#
00:27:59: %SPANTREE-2-BLOCK_BPDUGUARD: Received BPDU on port FastEthernet0/1 wit
h BPDU Guard enabled. Disabling port.
00:27:59: %PM-4-ERR_DISABLE: bpduguard error detected on Fa0/1, putting Fa0/1 in err-disable state
TestSwitch#sh spann int fa 0/1 det
no spanning tree info available for FastEthernet0/1

TestSwitch#sh int fa 0/1
FastEthernet0/1 is down, line protocol is down (err-disabled)

Just like expected.  The BPDU was received and the port shutdown before BPDUFilter could remove it.  As Santino explained above, BPDUFilter enabled globally evaluates the BPDUs on the Portfast ports and if one is received, the port would be transitioned out of Portfast and into a regular STP port.  Except in this case, where BPDUGuard does its job and disables the port before that can happen.  So far, so good.  Now for the big one.

7.  BPDUGuard and BPDUFilter enabled per interface

Back to default one more time.  Now, I enable BPDUGuard and BPDUFilter on the interface.  Once the config is done, I re-enable the port.  Here’s the payoff:


 Port 3 (FastEthernet0/1) of VLAN0001 is forwarding
   Port path cost 19, Port priority 128, Port Identifier 128.3.
   Designated root has priority 32769, address 0019.2f4d.4580
   Designated bridge has priority 32769, address 0019.2f4d.4580
   Designated port id is 128.3, designated path cost 0
   Timers: message age 0, forward delay 0, hold 0
   Number of transitions to forwarding state: 1
   The port is in the portfast mode
   Link type is point-to-point by default
   Bpdu guard is enabled
   Bpdu filter is enabled internally
   BPDU: sent 0, received 0

TestSwitch#sh run | beg 0/1
interface FastEthernet0/1
 switchport mode access
 spanning-tree portfast
 spanning-tree bpdufilter enable
 spanning-tree bpduguard enable

Ah ha!  It looks like BPDUFilter is stopping BPDUs from being transmitted or received on the port, as my new understanding of BPDUFilter explains.  Since there are no BPDUs, there is nothing to trigger the BPDUGuard feature!  Success!  As I said in the last post, when both are enabled on the same interface, BPDUFilter wins.  Yes, I admit I got confused about the function of BPDUFilter on the interface, but the results are the same.  I just have a better understanding of it now.

In conclusion, I learned a lot in the last 24 hours.  Always trust IOS.  It may not be clear all the time, but it never lies.  Make sure you lab it up when you have a question about things.  Listen to the commenters on your blog, they know their stuff.  And don’t be afraid to be blunt.  If nothing else, it makes you sure of your position.