Avoiding Proctor Proctology

Posted on by
Reply

Other than perhaps professional sports referees, I can think of no more maligned group of individuals than the proctors in the CCIE labs.  The myths about them are legend.  They screw up your rack at lunch.  They do everything they can to make you fail.  They act obtuse when you ask questions because they don’t like passing new CCIEs.  For the most part, all the mystery and nastiness that surrounds the proctors is just a bunch of hokum.  They are people just like you and I.  They do their jobs just like you and I.  It just so happens that their job is closely identified with something that most people study for hundreds of hours to achieve, and when those people fail, they look to cast blame on others.

I’ve been to the lab once or twice.  I’ve met several proctors both there and at Cisco events.  People who only need to be known by first names.  People like Howard and Tom and Stefan.  And each time I encountered them outside the lab setting, they impressed me with their poise and charm and sense of humor.  Just like any position of authority, the CCIE lab proctor has a role to play once they step inside the walls of the lab.  They must project an  aura of authority and calm.  They must provide guidance where confusion reigns.  And above all, they must protect the integrity of the program.  When most people deal with them in this setting, they come off cold and uncaring.  Just like a judge or a police officer, it’s important to remember that the position of proctor is a role that must be played.  Keeping some things in mind before you step through the badged door will go a long way to making your proctor experience a smooth one.

1.  Be nice. Yes, it sounds silly, but it’s a very important thing to keep in mind.  People are amped up when the walk into the lab.  They are stressed and wired and strung out.  Some people retreat into a shell when stressed.  Some people lash out and are irritable.  What’s important to keep in mind is that when dealing with the proctor, you should keep a calm and even demeanor.  People tend to respond in kind with the emotions they are presented.  If you walk over to ask a question and are short and snippy, expect a short answer in return.  However, if you are nice and pleasant, it can go a long way toward getting a favorable answer quickly.  When you go to lunch, don’t be afraid to engage the proctor in some light conversation.  Talk about the weather or the food or a sports team.  Anything but the lab.  Engaging them outside the walls will give you a feel for how they react to things and can help you judge if they will be helpful when it comes time to utilize them.  Besides, it never hurts to be friendly.

2.  Ask ‘yes’ or ‘no’ questions. The most popular complaint I hear about proctors is that they never answer the question that you ask them.  Before you stand up to go ask for clarification, carefully word your question so that it can be answered with a simple ‘yes’ or ‘no’.  For instance, rather than asking “How should I configure this frame relay connection?”, instead ask “There are two ways two configure this frame relay connection, point-to-point and point-to-multipoint.  The question isn’t clear about which I should use.  I’m leaning toward configuring it as a point-to-point.  Would I be correct in this assumption?”  The proctor may still choose not to answer your question, but you’re more likely to get a binary yes/no response out of them.  By showing you understand the technology and you aren’t just trolling for an answer, you’ll appeal to their interpretive skills.  Remember that the proctor won’t give you an answer, no matter what.  They have too much riding on their job, reputation, and the CCIE program to ‘bend’ the rules for one candidate.

3.  Don’t assume a problem is something you can’t fix. Want to piss off a proctor really fast?  Walk up and say, “I can’t get OSPF to come up.  My rack must have a cabling problem.”  I promise that after the eye roll and gruff answer, you’ll fail the lab.  Candidates who are weak in troubleshooting skills tend to blame the physical layers first because it’s the one layer they can’t touch in the lab.  In the old 2-day lab, cabling could be an issue to resolve.  But in the 1-day lab, with cabling extracted from the candidate’s control, there is a 1-in-1,000 chance that the cabling is at fault.  Think about it like this: the lab equipment supports 3-5 candidates 5 days a week for 50 weeks a year.  Cabling issues will be caught and dealt with quickly.  Simple things like bad ports or bad cables will be caught when the lab racks are booted first thing in the morning.  You going to the proctor and claiming that OSPF is broken because your routers are wired backwards will only serve to irritate the proctor.  What will happen next is that the proctor will ask you to wait by his/her desk or make you wait in the RTP conference room.  They will go to your rack and start troubleshooting OSPF.  They’ll find out you misconfigured a network statement or forgot to enable authentication.  They’ll check to make absolutely sure it’s not a layer 1 problem, all while you are isolated away from your terminal.  Then, after 15-30 minutes, they’ll come get you and tell you, “It’s not a physical problem.”  That’s it.  No other information.  You won’t get to see what they did to find your problem.  You won’t get any hints about how to fix the issue.  And you won’t get those 15-30 minutes back.  Period.  On the other hand, if you go to the proctor with a long list of reasons why it has to be a layer 1 problem, like BERT or TDR output or a down/down physical interface that won’t come up at all, they’ll be more receptive due to your troubleshooting efforts.  And if they find a layer 1 fault during their troubleshooting, you’ll probably get the time back from their efforts and the time it takes to replace the faulty unit.

4.  Don’t cheat. Well, duh.  You’d think that should be a given.  But people still try to pull stupid things all the time.  And not just the braindumping.  Writing things down and then trying to sneak them out on the scratch paper, of which every scrap must be accounted for.  Trying to use a cell phone to discreetly look up answers.  Looking at other screens (for all the good it’ll do you).  Just don’t cheat, or even give the appearance you’re cheating.  That should keep the proctors from getting quite nasty with you.

In the end, just remember that these guys and gals are doing their jobs as defined by the CCIE program.  They work hard and do everything within their guidelines to help you pass.  They aren’t out to screw you, and if you keep your head about you and use some common sense and manners, they’ll be the best resource you’ll find in the lab.

Hooray for Bruno!

Posted on by
2

Twenty-four hours after Cisco ‘tweaked’ the CCIE lab troubleshooing section a little, we finally get a little bit of feedback about what they have wrought on the weary CCIE candidates.  This thread over on the Cisco Learning Network serves as the official announcement.  Further down the list is a response to some questions from concerned folks about what this could mean from Bruno van de Werve, who appears to be part of the CCIE program inside Cisco.  Bruno was also responsible for the excellent CCIE lab web interface video.  So, what did Bruno share with us?

1.  This is a virtual switch based on a router image, so the ports will be shutdown and not configured as switchports up front.  This about having a switch module in a router in Dynamips/GNS3 and I’m sure this is what the L2IOU is like.

2.  The image appears to be based more toward the 3550 rather than the 3560.  This comes from comments about the port being in dynamic desirable instead of the 3560 method.

3.  All interfaces are Ethernet, not FastEthernet or GigabitEthernet.  They are also arranged in groups of 4, e.g. e 0/0, e 0/1 and so on.  Due to the software limitations, you can only do interface range commands across the 4 ports of the module, so for instance setting up more than 4 trunks at once would require multiple interface range commands.  As an aside here, Bruno says that they are most likely not going to have more than 4 trunks per switch right now.

4.  These switches are based off of 12.2 mainline IOS with some L2 switching features added in.  They also lack any of the hardware ASICs necessary to do advanced features.  So, no Cat-QoS for instance (His words, not mine.  Cat-QoS, really???)

5.  There are only two switches at present in the TS section, and they are in the same IGP domain.  Bruno elaborates that they might add more switches later as features are added to the L2IOU image.

6.  There will still only be 10 questions in the TS section (his words).  So, at best you can probably expect 1-2 L2 questions for now.  He did say that there is a possibility that the questions could influence other tickets, so be on guard if you have a L2 question up front that it might affect an OSPF question later.

7.  There are no planned changes to the configuration section of the lab at this time.  No, they aren’t going to remove MPLS.  They aren’t screwing your config section totally.  Also, in a reply to one of the first commenters, Bruno said,

“At the moment, there won’t be any changes to the configuration section of the lab, so the two L2 troubleshooting questions will still be there.”

Ahem.  As my old econ professor in college used to say during test reviews…”Hint, hint, hint, oh hint.  You might see this on the exam.”  Guess what folks? Bruno just gave you a big hint.  You are still going to see L2 troubleshooting on the config section.  And if the historical trend is true, it’s going to be the first part of your lab.  “There are XXX faults in your initial lab configuration.  Correct these faults for 1 point each.  Be aware that these faults could impact configuration in other sections of the lab, and if they are not resolved and impact another working solution, you will not receive points for the non-working solution.”  So put your thinking caps on.

My Take

L2IOU was introduced to the troubleshooting section to crack down on the braindumpers.  Cisco knew they needed to add it to keep the cheaters off-balance, but the feature set of the IOU emulator wasn’t ready when the TS section went live.  I’m sure a team has been working on it night and day trying to get enough features built into it to make it a usable tool for the TS section.  It would be virtually impossible to have a physical setup for troubleshooting.  30+ routers would be a large rack, and now that you start adding physical switches in on top of it, it become unmanageable.  IOU made the most sense for the L3 section, but L2 is just as important for testing a candidate’s knowledge.  However, there wasn’t an emulator capable of perfectly virtualizing a switch.   And there still isn’t.  The fact that L2IOU got pushed out the door in the state that Bruno describes it in means that someone had to be done to stop the braindump hemmorage.  That’s the only reason I can think of to introduce a totally foreign software-emulated L2 device that by all accounts doesen’t resemble the feature set of the devices in the lab that you are already forced to troubleshoot a mere two hours after the beginning of the lab.  So why not put the additional troubleshooting questions in the config section where they can be done on real hardware?  Especially since there are only two L2IOU devices currently?

Because Cisco has never nailed down the exact nature of the TS section, it gives them more latitude in changing it quickly.  Adding ten new TS questions takes a lot less time than re-engineering a whole lab set.  Just like the OEQs before, the TS section is designed to weed out the weak candidates relying on someone else’s leaked materials rather than the tried-and-true studying methods that should be employed.

I still don’t like the change.  Not enough lead time.  A ham-handed attempt to fend off what appears to me to be a growing portion of lab candidates.  Seeing as how they appear to be phasing things in slowly and not increasing the impact of L2 until more features get baked in, I guess I can live with being a guinea pig for now.  But I sincerely hope that I pass my lab on this attempt so I don’t have to beta test new TS section features again.

Layer 2 – Electric Boogaloo

Posted on by
2

In alignment with CCIE level requirements, Cisco is adding L2 switching features to the CCIE R&S Troubleshooting exam through L2 IOS software on Unix (L2IOU) virtual environment. The new feature will be available starting January 17, 2011. The CCIE R&S exam consists of 2 sections b the troubleshooting (TS) section which runs two hours, and the Configuration (Config) section which is six hours. The Config lab utilizes actual physical devices in racks, whereas, the TS lab uses a virtual environment under IOU. IOU offers a very realistic simulation of router (L3+) features in the TS lab but until now had no L2 switch capability. With the addition of the new L2IOU, the TS lab will now include both L2 and L3 capabilities in the virtual environment.

Official Cisco Announcemnt (Thanks to Rob Routt for the text)

So, it appears that Cisco has finally added layer 2 (L2) troubleshooting to the first troubleshooting section of the lab.  Hmmm…

Ladies and gentlemen, a rant…

What?!?! Really?  Why did this have to happen four days before my exam!  Okay, it’s not fair to think that you’re singling anyone out with these changes, but when you added the open-ended questions (OEQ) in 2009, you did it the week before my second lab.  And that little change with the OEQs cost me one lab.  With luck like this, anyone going to Cisco Live Las Vegas 2011 should head down to the casino with me and bet the exact opposite of what I say. You’ll make millions.

I know the standard response.  Anything is fair game for troubleshooting on the TS section.  If I’m a prepared CCIE candidate, I should have no fear about this addition.  Everything will be fine, I’ll nail it.  All of these things are true, but it still doesn’t make me feel any better.  This is like going in to take a driver’s test only to find that the steering wheel is on the wrong side of the car!  Does it affect the test?  Not really, but it is a head scratcher.

If all I have to worry about is layer 3 faults, I don’t have to consider spanning tree or trunk ports or Q-in-Q tunneling or QoS mismatches or backup interfaces or Etherchannel misconfigurations or any one of a number of things.  No, adding L2 to the TS lab doesn’t make it necessarily harder, but it does broaden the range of things I have to think about when I start looking at a problem.  Irritating, this is.

Why the need to cram so much stuff into the beginning?  You know, you always did well with the first TS question of the config section – Hey, we broke three or four things in your lab that you need to fix before you get started.  You don’t get any points if they aren’t fixed, and since they probably are going to break other things, you won’t get points for other questions either.  Guess breaking things on five routers and four switches isn’t nearly as fun as doing it to 30 routers and who-knows-how-many switches.  I’m sure that the setup of the TS lab allows them to break lots and lots of interesting things in ways they can’t simulate in the config section.  But if that’s how things are going to be going forward, they need to remove any TS from the config section too, if it still exists.  After all, there are no sim questions on the CCIE written because they are covered in other sections.  Why should there be troubleshooting on the lab config section when you’ve so eloquently covered it in the TS section?  I figure that once the L2 TS stuff gets baked in, we might actually see this happen.

And another thing…one business day notice?  Really?  Come on now.  That’s just unfair.  Used to be, changes to the lab required six months of notification before going live to allow students the opportunity to study up on them and be prepared.  How do you think the folks taking the lab bright and early Monday morning are going to feel?  I guess that adding a whole new topic to the TS section doesn’t constitute a ‘major’ change anymore.  Or perhaps because you intentionally left the description of the TS section vague, it allows you the opportunity to ‘clarify’ it with little to no warning.

I’ve got a question, Cisco.  Are the braindumpers really hurting you that bad?  You can tell me all you want that the OEQs and the TS section were designed to better assess a candidate’s knowledge of every facet of networking.  With respect: bullshit.  The OEQs were a direct outgrowth of the interview process implemented at a specific testing site to catch people memorizing leaked lab questions and getting through with no difficulties.  The TS section replaced the OEQs once the process was refined to the point where they were no longer needed.  Have people started memorizing the TS section too?  Is it bad enough that we’re going to have a new exam every week in an effort to catch the unworthy candidates?  Perhaps we should just move to a totally interview-based exam with no access to CCO or documentation of any kind?  After all, the perfect CCIE candidate should be able to configure and troubleshoot a lab without access to a keyboard or a mouse or even a monitor!

Okay, I think that’s enough ranting for one morning.  I’m at T-minus 141 hours and counting.  That should be just enough time to brush up on my L2 TS skills in addition to all the other stuff I’m cramming for.  You know, I’m just about to the point where I’m not going to pass this lab because it’s a great career move or because it’s something I feel I need to do or even because it would show me as being at the pinnacle of my networking career.  I’m going to pass this lab just to spite Cisco and some of the dumb last-minute decisions their program mangers make.  Because I don’t know if I can handle CCIE Lab 3: CCIE With A Vengeance.

Frame (Relay) of Reference

Posted on by
6

A while back I wrote a CCIE lab-related post about RIP and why it’s still on the lab.  Most of the questions that I see lately revolve around another old technology and the curiosity of why it’s still contained in the vaunted lab blueprint.  I speak of frame relay.  The much-maligned WAN technology that no one seems to be able to explain the reason for existing anymore, yet the weary lab candidates find themselves pour over at the last minute.

Frame relay has been around forever.  And slightly shorter than forever, it’s been on the CCIE lab blueprint.  I’ve been actively studying for my exam since July 2008.  And as far back as I can remember, I’ve been studying frame relay.  And ever since that time people have been asking why it’s even on the exam.  I’ve heard people say it’s too old.  It’s past it’s prime.  No one offers it anymore.  Recently, with the change to the version 4 lab exam, MPLS was introduced as a configuration topic.  We all thought “Surely frame relay will be gone with its apparent successor on the lab now…”  And yet, frame relay is still there.  In fact, they added frame relay switching back onto the blueprint.  That was a huge change for me, never having dealt with that side of things before.  So, here is frame relay configuration of all kinds taunting us with is outdatedness.  Forcing us to keep up with our grandfather’s WAN technology.  And, in my opinion, that’s just the way Cisco likes it.

Just like RIP, frame relay wasn’t necessarily meant to be on the exam as a test of your ability to map IPs to DLCIs.  Frame relay introduces a whole level of configuration possibilities to the lab to test your wits without being obvious.  The whole point of keeping it on the lab isn’t to make you expend effort in getting 2 or 3 points.  It’s laying a foundation that could cost you 10 points or more.

I remember reading the Building Scalable Cisco Internetworks (BSCI) routing exam guide a few years back studying for my CCNP.  It was my first real exposure to the depths that routing protocols could go to.  In one the chapters over OSPF, there was a section that covered a ton of information on running OSPF over non-broadcast multi-access networks.  There were tables and charts and graphs galore.  All dedicated to one sub-topic.  Why was that?  Because the intricacies of running OSPF over frame relay are legion.  There are multiple network types that determine DR and BDR functionality and hello timers.  It’s like a mix-and-match sale at the department store.  So many fun combinations from so few parts.  That chart was just the tip of the iceberg, though.  Because once you’ve gotten a frame “cloud” in your lab, the real fun can start.

Suppose you have a really simple question in the lab like this:

Configure frame relay between R3 and R6.  Do not use static
mapping or inverse ARP. (2 points)

Straightforward, eh?  You could probably nail up this frame configuration in about 5 minutes.  Well, think about the rest of this example lab and find all the things that rely on Frame Relay to award points.  Let me give you a few examples:

Configure OSPF on all routers indicated in the diagram.  
Between R3 and R6, use a network type the provides for the
fastest recovery times without a DR/BDR election. (3 points)

Configure the link between R3 and R6 so that routing protocol
packets are never dropped.  If the queue has more than 20 packets,
reduce the bandwith to 32kbps. (3 points)

Configure the link between R3 and R6 to authenticate using CHAP.
(2 points)

Now, I’m not saying that all of those questions could end up in your lab.  But think about it like this:  That single 2-point FR question just became a 10-point chunk of your lab.  Now, if you don’t configure your FR section correctly, you are halfway to failing the lab from one misconfiguration!  Remember, no partial credit means that even if your whole OSPF section is right, failing to get FR working correctly means the script used to grade your exam won’t see all the routes on R3 and R6, and so you won’t get any points for OSPF.  It also means that your QoS won’t work, so there goes three more points.  And so on.  Soon, you could fail the lab simply because that outdated technology you didn’t really care about sucker punched you.

I’ve always held the belief that the CCIE lab is designed in a way to test every facet of being an network superhero.  Not just the obvious things like OSPF configuration or BGP troubleshooting, though.  As I’ve said before, the lab manual is carefully written by a team of highly competent people.  There are no extraneous words, and there are no unimportant tasks.  It’s like a game of Jenga.  The things you do at 10:00 a.m. have a direct impact on the last task you do at 4:00 p.m.  If your lab isn’t built carefully and solidly like a Jenga tower, it will topple down on the desk in front of you and knock your cup of colored pencils and markers onto the floor (metaphorically, of course).  Like it or not, frame relay is one of those blocks that forms the foundation of your tower.  Not because Cisco is protecting their investment in legacy technology.  Not because they are trying to wear lab candidates out with archaic configuration tasks.  Frame relay still exists on the blueprint because it teaches candidates to get their configurations right in that critical first hour of the lab because the Jenga tower that is your lab exam depends on each and every part.  And since frame relay can touch so many parts of the test, Cisco likes to keep it around to make sure you’re paying attention.

So the next time you find yourself staring at the blueprint and asking yourself “Why does Cisco still put frame relay on the lab?” you might try asking instead, “What does Cisco want me to learn from configuring frame relay?”  I think you’ll find the answer to the latter question a lot more enlightening.  With all of the things that depend on a simple frame relay link, one miscue could be the difference between a detailed score report and a simple 4-letter one.  Because from Cisco frame of reference, frame relay isn’t just a simple test question.  It could be one of the most important topics on your exam.

Party like it’s 1993!

Posted on by
7

I was once told by a consultant that he could figure out whether a client needed his services after about 30 seconds at a router console.  When I asked how he could be so sure after such a short amount of time, he consoled to a router in his lab and typed in “show clock”.  The lab router returned the now-familiar string of Mar 1, 1993 (he’d just booted it).  As soon as he showed me that one command, it all made sense.  Keeping accurate time in a computer environment is very important to systems.  Directory structures depend on an accurate clock to authorize logins and track audit events.  Novell and Windows both utilize systems to ensure that the clocks of all the servers in the network are synchronized.  And if those clocks drift out of sync, heaven help the server admins.

What about network equipment?  In days past, the routers and switches were typically neglected when it came to clock setting.  In fact, most older Cisco routers didn’t even include an on-board battery to keep the clock accurate.  And when the router would reboot, the software clock didn’t have an accurate hardware clock to refer to, so it used 00:00 1 Mar 1993 as the reference point.  But as systems have increased in complexity over the past several years, the need to have all the time on your equipment accurate has become paramount.  When debugging a call hand-off on a voice gateway, an accurate router clock ensures that you can match the time the call was placed with the debug message output.  If there is a security incursion into your network devices, you need to track the time the device was accessed in order to be able to accurately report the event to the proper authorities.

So how do we get our network clocks to report the right time?  Well, the process is fairly easy, provided you’ve done a little homework about a couple of things.  First, you need to know what timezone you are in and what your GMT offset is.  In today’s world, it is far easier to keep the clock of your device synced to GMT, then apply an offset to show you what the local time is.  That way, if you have devices spread all over the world, you never have to worry about a significant time difference because one clock was synced to local time and the other was synced to GMT with an offset applied.  For the purposes of the examples in this post, I’m going to assume the router is located in the central United States and is in the Central Time Zone.  The fact that I myself am in the Central Time Zone and therefore would not need to do any additional thinking to write my examples is purely coincidental.

In the case of my example, the central United States is in the Central Standard Time Zone (CST).  CST is six hours behind the GMT clock (GMT -6).

When you first connect to the router, you will either see the default time of 00:00 1 Mar 1993 (for older routers), or if you’re on an ISR or newer router, it may be synced fairly close to the actual time.  Starting with the ISR, Cisco started keeping the time synced more closely when the router was shipped from the factory, and the battery keeping the hardware clock time when powered off seemed to last a lot longer.

At this point, we need to set the router’s timezone with this command:

R1(config)# clock timezone <name> <GMT offset>
R1(config)# clock timezone CST -6

Once you’ve done that, the system should update with a message telling you that the current time zone has been updated.  Now, the router should know what time zone it’s in and adjust the clock offset appropriately.  You should also set the daylight savings time conditions as well.  For those not familiar, DST is a law that many countries have adopted that force sleepy engineers to reset the clock on the microwave at 2 a.m. on two days during the year.  Just when we had it down, they went and changed it a couple of years ago.  Hence the reason that Cisco doesn’t hard-code the DST settings on their routers.  They are more than happy to let you do it yourself.  In this example, we’re using the U.S. standard of the second Sunday in March and the first Sunday in November:

R1(config)# clock summer-time <name> recurring <week number start> 
        <day> <month> <time to start> <week number end> <day> <month> <time to end>
R1(config)# clock summer-time CDT recurring 2 Sun Mar 2:00 1 Sun Nov 2:00

And just like that, your router will perform just like every other smart device in your house and reset the clock for DST when necessary.  Now, if I could just get my microwave to do that.

Now that your router is in the right time zone, it’s probably a good idea to sync the clock to some kind of external time source.  That’s why Network Time Protocol (NTP) was created.  It allows distributed systems to sync their clocks with a time source directly connected to an atomic clock (the most precise kind).  NTP form as a hierarchy through use of strata. Stratum 0 NTP devices are directly connected to atomic clock sources, usually through RS-232 or other cabling.  The are not connected via a network.  Stratum 1 NTP devices are connected to Stratum 0 devices via a network connection.  These are the systems that are usually referred to as time servers.  These are usually the devices that you will sync one of your clocks to.  Why just one?  Well, while having all your devices synced to external time sources is a great idea, there are two issues that can arise.  First, having that much NTP traffic exiting your network isn’t exactly optimal.  There is no reason for 20 routers to each poll an external NTP server when one router could do the same thing, and the other 19 routers poll the synced router.  You can even designated your NTP-synced router as a Stratum 2 or 3 device if you’d like, then have it start serving time to your other devices.  The second issue with using all external NTP servers deals with security.  Some people are uncomfortable with the idea that all their devices are being told how to manage their clocks by an external device over which they have no control.  By configuring your devices to sync to one clock in your network that you control, there is more consistency and less reliance on systems outside your control that may be overloaded or unreliable.  For more info on what happens when someone starts abusing an NTP server, check out what NETGEAR did to the University of Wisconsin’s NTP server here.

For the sake of your sanity, it’s best to set the clock to something close to the actual time before you set the NTP server.  The reason for this is due to the propensity for NTP to look at your clock and declare it insane if you are too far drifted from the actual time.  I usually try to get my local clock somewhere in the neighborhood of one hour from the actual time, but the closer you are and the faster NTP will sync.  You do this with the following command (note the context)

R1#clock set <HH:MM:SS> <1-31> <Name of Month> <Year>
R1#clock set 01:00:00 10 Jan 2011

You must manually set the clock in enable mode, not global config mode.  Once you’ve gotten the clock close to your time, you can set the NTP server.  If you want to sync your router to an external time source, using the NIST Time Server list is always a good start.  If your router is capable of resolving DNS, a better idea is to use the NTP pool service.  This service is a cluster of NTP servers around the world that is served by round-robin DNS query, so no one server can get overloaded.  If you don’t feel comfortable syncing your clock to a server in France or Japan, you can always narrow down the focus of your query by using narrower DNS entries.  Check out their site for ways to do this.  Once you’ve figured out  how you are going to connect to NTP, and whether you are going to use external or internal servers, the command is pretty easy:

R1(config)# ntp server <name or ip>
R1(config)# ntp server 10.1.1.1
or
R1(config)# ntp server 0.pool.ntp.org

Yep, that’s it.  Simple once you’ve gotten all the planning down.  You can check your NTP sync status with the command show ntp status.  You can see which time servers your are polling with the command show ntp associations.  The commands have a lot of good info, but can be a little cryptic your first couple of trys.

Once you’ve gotten your clock in sync, there’s just two more commands you need to use to ensure that all your logs and debug outputs are using the right clock.  By default, logs and debugs use the system uptime for their output, so you could get a log message that says “1w3d” instead of the real time.  And if you have to start doing math on your debugs to figure out when a call was dropped, you’re going to be a cranky rock star.  From global config mode, go ahead and type in these two commands:

R1(config)# service timestamps log datetime msec
R1(config)# service timestamps debug datetime msec

The “MSEC” keyword on the end tracks the message down to the millisecond level, which I’ve always found very handy when trying to figure out sub-second error messages.  It also helps better match error logs which are all part of the same event, but spread out over multiple messages.

Now, your routers are all in sync and your logs and debugs are all outputting the correct time.  If a consultant tries to access your devices, you will appear to be one cool customer that doesn’t need any help on your networking devices.  You’ll also be able to troubleshoot faster and get all the fame and wealth appropriate to your station as the Network Miracle Worker.

For what it’s worth, I did a lot of searching about why the default time on a Cisco router without a battery backup is March 1, 1993.  No one seemed to have a definitive answer.  According to the Internet, the only really exciting thing that happened that day was George Steinbrenner being reinstated as Yankees owner.  I doubt anyone in San Jose is a Yankees fan, so I didn’t think that was it.  It also didn’t correspond to any neat numbers in the Unix Time Epoch.  1993 was the first year that Cisco acquired a company, but that happened in September of that year.  The only thing that happened early was the release of IOS 10.0.  I guess that Cisco decided that “10” was an important enough number that they wanted to start basing recent history sourced from this date.  The other possibility is that it was just an arbitrary date chosen by IOS engineers so that the router clock didn’t cycle all the way back to 1900.  MS-DOS had a similar functionality, wherein it would see the system BIOS clock set to 1900 and assume that the BIOS had to be wrong.  It would then set the clock to the earliest date it could (January 1, 1980).  Maybe Cisco just decided that 1993 was so early that if they noticed a router clock stuck on that date, it would just be assumed the the clock was not set.  Either than, or they were really big fans of this song…

I’s and T’s and Crosses and Dots

Posted on by
Reply

My name is Tom, and I’m careless.

Yep, I admit it freely.  I’m the kind of person that rushes through things and gets the majority of the work done.  Often I leave a few things undone with the hope that I’ll go back later and fix them.  For me, the result is the key.  Sometimes it works out in my favor, sometimes it doesn’t.  More often than not, I find myself cursing out loud about this unfinished job or task months down the road and threatening to find the person responsible, only to later determine that I should be kicking my own butt for it.

One place where this particular habit of mine has caused me endless grief in inside the unforgiving walls of Cisco’s Building C lab in San Jose.  Yep, I can honestly say that at least one lab attempt was foiled due to my propensity to miss the little things.  I’ve previously written about some of the details of the lab, but I wanted to take some time in this post to talk about the details themselves.  As in, the details in the questions that will kill you if you give them the chance.

Let’s get it out there right now: there is NO partial credit in the CCIE lab.  None. Zilch.  If you fail to answer every portion of the question with completeness, you get zero points for that question.  Unlike the old days in elementary school, you don’t get points for trying.  This shouldn’t really come as a shock to anyone that’s taken a multiple choice test any time in their life.  On those tests, there is exactly one set of answer(s) for a particular question, and if you don’t select the proper repsonse(s), you don’t get the points.  The same thing goes for the questions you find in the CCIE lab exam.  Just because the questions may or may not have multiple parts doesn’t excuse your need to answer them fully.  Old Mr. Hollingsworth used to tell me regularly, “Son, close only counts in horseshoes and hand grenades.”  Since I don’t play horseshoes and my hand grenade supplier mysteriously dried up, I guess close just won’t cut it any more.

You might end up getting a question in the lab that says something along the lines of “Configure OSPF on R1, R3, and R6 according to the diagram.  Do not change router IDs.  Rename R1 to ‘SnugglesR1’.”  You could build the most perfect OSPF lab in history.  You could spend an hour optimizing things.  If you forget to rename Snuggles the Router, you will receive no credit for the question.  All that hard work will get flushed down the toilet.  You’ll get your score report at the end of the day and wonder why you didn’t get any points for all that time you spend making OSPF sing like a soprano.

In order to prevent this from happening to you, start training yourself now to read carefully and consider every facet of the questions you’ll see.  Remember that the questions in the lab are carefully constructed by a team that spends a ton of time evaluating every part.  There are no unnecessary words.  Candidates have pestered proctors over the meaning of single words on a question.  The questions are written as they are to make sure you take into account a number of factors.  They are also designed to slip in changes to tasks and additional configuration with a word or two.  And if you are careless, you’ll miss those phrases that signal changes and negations.

Surely, everyone has taken a test that has a question that says “Which of the following was NOT a <something> <something>”  Your job is to evaluate the choices and pick the one that is not something.  That single word changes the whole meaning of the question.  And for those that are careless or the kind the skim questions, the NOT might be missed and cause them to answer incorrectly.  Questions in the lab are the same way.  Skimming over them without reading critically can cause nuances to be missed and lead to incorrect solutions.  After 5 hours of staring at words on a monitor, things might start blurring a little, but attention must be paid to the last few questions, as those might be enough points to buoy over the passing mark.

I’ll be the first to admit that the pressure to get everything done in the allotted time may cause the candidate to want to rush, but you must resist that pressure.  Many CCIE lab prep courses and instructors will tell you to carefully read the questions before you ever start configuring.  I agree, with some additions.  I always take my scratch paper and write the task numbers down the side.  After I’ve accounted for Task 1.1, 1.2, 2.1, and so on, I then go back to the questions and make marks next to my list for any questions that may have multiple parts or tricky solutions.  That way, if I find myself rushing through after lunch the marks I made early in the day force me to pay attention to the question and ensure that I don’t miss something that might cause me to tank three or four points.  Those points add up over the course of the day, and more than a few careless mistakes can cost you a nice expensive soda can.

If you are serious about the CCIE lab, it’s worth your time to start working on ensuring that you pay close attention to each question and don’t make any careless mistakes due to reading too fast or missing important configuration requirements.  Your day is going to be stressful enough without the added pressure of fixing mistakes later in the lab as a result of forgetting to enable OSPF authentication or a typo on a VLAN interface.  You want to remember to dot every “i” and cross every “t” for each and every question.  That way, you can walk out of the lab and use that freshly-dotted “i” when you spell you new title as a CCIE.

The Nerd’s Going to Have a Field Day!

Posted on by
1

Guess who gets to go to San Jose for something OTHER than the lab for once?  That’s right, THIS GUY <—–!

Thanks to the wonderful folks over at Gestalt IT, I have been invited to be a delegate at Tech Field Day 5!  This is a tremendous opportunity for me to get involved with the sponsoring technology companies and hear about their products and strategies for the coming months.  Tech Field Day offers engineers and technical people the chance to hear about these great things in an environment conducive to learning (i.e. no sales pitches).  I’ve read the coverage of the previous Tech Field Day events from some of my other blogging brethren (and sistren) and found the information that they’ve given the community to be quite valuable.  I’m looking forward to the opportunity to spend some time listening to the best and brightest that will be in front of the delegates, and of course I can’t wait to pick their brains about technology!

Disclosure

What would an event be without some kind of disclosure?  In this case, the TFD sponsors are paying for my travel costs and lodging costs during my stay in San Jose.  That being said, they are NOT putting any limitations on what I say about the information I receive or the feelings that I have about the conference, other than to respect any information that might not be public knowledge or embargoed (which I would have done anyway).  I can assure you that any and all opinions expressed about the content of TFD are mine and mine alone.  I want to give you the unvarnished truth behind what I see and hear.

How Do I Get Involved?

Gestalt IT is always keeping an eye out for TFD delegates.  I’ll admit that while I have wanted to be a delegate for a while now, I never thought I’d be one until I was asked.  There are some things to keep in mind:

1.  Read the TFD FAQ and the Becoming a Field Day Delegate pages for more information on each.  Those pages are the best source of information about the process and criteria for TFD and its delegates.

2.  Understand there isn’t some kind of strange conspiracy or secret machine driving this.  These people look for independent critical thinkers that aren’t afraid to voice their opinions about subjects.  Don’t be afraid to show your independence and be sure to speak about technical subjects.

3.  Ask questions.  Anywhere and everywhere.  You never learn if you don’t ask questions.  I question things all the time.  And if you have any questions that you’d like me to ask at TFD, please let me know in the comments.  I’ll be making more posts as the list of presenters becomes final so that you know who I’ll be interacting with.

Be sure to follow @TechFieldDay on Twitter for more information about TFD 5 as the date approaches in February.  You can also follow the #TechFieldDay hash tag for updates live as the delegates tweet about them.  For those of you that might not want to see all the TFD-related posts, you can also use the #TechFieldDay tag to filter posts in most major Twitter clients.

For more information about sponsors and delegates, head on over to the TFD5 page on GestaltIT.

The Nerd Presents: Tips for Presenting

Posted on by
5

Everyone in the world has at least one good presentation in them.  It doesn’t take much to put something down in a few slides and talk about it.  For most people, the hardest part is getting up in front of a group and actually speaking.  Once you get over that, the rest is easy.  However, in my job I get to listen to a lot of presentations.  I’ve had a lot of time to look beyond the content to things that tarnish your image when in front of customers or learners.  I won’t profess to be an expert when it comes to the art of presentation, but I think most would agree with me that looking at these tips will help out in the polishing department.

– Close down Outlook and turn off your mail notifications. As professionals, we are all married to Outlook/Thunderbird/Entourage.  No matter what it seems impossible to escape it today with the ability to load it on our desktops, laptops, and mobile devices.  However, when you stand in front of me to start pitching your software or tell me about a new technology, please turn off your mail client and notification system.  Think about it like this: you don’t leave your cell phone ringer on when your presenting because of the distraction.  Why would you leave the new mail popup in the corner?  At best, it causes me to shift my focus from the content of the slide deck to whatever new message you just received.  At worst, I may be privy to inside information from your company, sales targets and customers, or in rare and somewhat ironic cases, end of life notices for the very product you are trying to sell me.  Ask yourself this question: If you were listening to me tell you about how great my Project Foobar is and I receive an email from my lead product specialist with the subject “Inability of Project Foobar to Address Basic Business Needs” would you still be interested in hearing my pitch?

– Shut down instant messengers. For that portion of the crowd that thinks email is so yesterday, there is the instant messenger (IM).  People use a variety of clients, from the tried-and-true AOL instant messenger to newer things like Trillain or Pidgin or even Cisco Unified Personal Communicator.  Guess what?   Shut it down before you start talking to me.  All of the reasons above still apply to IM conversations.  In the case of IM though, people are a lot more informal.  So conversations may not start out with simple hellos.  You may get something more pointed or perhaps a greeting too salty for the taste of the group your are presenting to.  Imagine a co-worker sending you profanity laced tirade during a speaking engagement with a Catholic school.  Or something leaning toward the more delicate and personal from your spouse when you are speaking to a prospective customer.  The ability to embed pictures in IMs makes this prospect even scarier.  And before you say “I can just set myself to away” think about all the times that an “emergency” has come up and you’ve been pinged on IM even when your away.  Chalk that particular one up to most people assuming that “away” means “I’m sitting right here and I just don’t want to talk to you right now”.  Better to just shut yourself off from the IM cloud for a while and not take any chances.

– Change your desktop wallpaper to something bland. I’m guilty of this one, so allow me to start casting stones.  I like wallpapers.  Generally something abstract or landscape oriented.  I do have the occasional cool picture of something fire and ice related.  But for the most part, I tend to avoid pictures of people or animals or quotes.  Especially if they could be construed as the least bit offensive.  But even my conservative taste in wallpaper can be distracting when presenting.  You say, “But no one is going to see my desktop if Powerpoint is up the whole time.”  True enough, but how many times are you only using Powerpoint?  What happens if you have to switch slide shows?  Or look at a document on your desktop?  Or switch to a web browser to load a live video?  There are a variety of reasons to jump out of Powerpoint, and if you don’t think ahead of time, you might just find yourself showing a picture of your last trip to Cancun to all of the members of your church group.  Even in the case that it’s a picture of your newborn daughter, your presentation focus will be lost as people start cooing about how cute she is, how old she is, whether or not she’s sleeping yet, whether or not you’re sleeping, ad infintium.  In my book, it’s best just to change your wallpaper to basic black and move on.

– Collect all your documents related to the presentation in one folder on your desktop. Most of my presentations are loaded with technical content.  Many of them, however, don’t have the density of the documents I used to put them together.  Making my slides into eye charts won’t help my audience understand my topic any better.  But if I mention that there is a document that includes more technical depth to this particular subject, invariably someone is going to ask to see that document.  Or ask about a fact or figure from it.  That leads to me needed to go spelunking through my file system to find it.  Call me somewhat old-fashioned, but I don’t really like people staring at my file structure and folder contents.  Especially if those folders contain competitive information.  What might happen if my customer sees a document named “Juniper ASA Comparison and Debunking.pdf”?  Sure, if I’m presenting one of those products it shouldn’t really matter, right?  But what if the other product is one that the customer has never heard of?  Yeah, if you’re researching firewalls and you’ve gotten to the point of hearing a presentation about one, hopefully you know about the other.  But in my mind, just the presence of that document could derail your presentation with questions that might not be pertinent to the discussion at hand.  Better to copy all of the relevant documents that you have sourced from your presentation into a folder labeled “Presentation Documents” and put it on your desktop so you don’t spend precious minutes searching for it.  And while you’re at it, consider changing your browser’s homepage if you shell out to the Internet during presentations.  Google is a good safe bet.  Your sports book?  Not so much…

– Don’t read the slides back to me. Pet. Peeve. Number. One.  Don’t read your slides back to me.  I’ve walked out of presentations that I’ve paid for the honor of attending for this gaffe.  If you are reading the slides back to me word-for-word, it tells me you’ve done no research on the topic and you have no depth on knowledge on the subject.  Marketing people are the worst when it comes to this.  They just assume that what has been printed on the slide is the definitive answer to everyone’s problems and just start reading it to me like gospel.  Guess what?  I can read too!  As you’re outlining the contents of that slide, I’ve already glanced over it and picked out the most relevant pieces of information that interest me.  If you then start at the top and read the bullet points to me, I going to guess this is all new to you too.  I treat my slide deck like I would treat a stack of 3”x5” index cards that I use for notes.  I expand on each of the bullet points in my slide deck with additional discussion topics.  That’s also one of the reasons I print my slide deck ahead of time and make it available to the people that I speak to.  That way, they can jot down the notes I speak about and reference them against the printed slides.  The way I see it, you came to see me speak, not look at my fancy multiple-build-slide transition heavy corporate approved 100-slide deck.  If you want me to read the slides back to you, it’s going to feel way too much like circle time in my son’s kindergarten class.

Many, many moons ago I was an intern at IBM in Rochester, MN.  My first-line manager decided that the other intern and I needed to get some practice giving presentations to clients/customers.  She therefore decided to make us present some Windows 2000 tips to a group of users that had recently received new Thinkpad T20s (how’s that for dating myself?).  After I had put together my slides, my mentor told me that I needed to go grab a brand new laptop from the laptop pool and use it instead of my personal machine.  What I questioned her reasoning, she told me that by using a fresh laptop out of the box, the usual cruft that come along with my personal machine would be absent.  I wouldn’t need to worry about some of the things I’ve listed above, like Outlook (in this case Lotus Notes) or my desktop wallpaper.  I could concentrate on my presentation.  And while I won’t say that her advice made my presentation into something that changed the fabric of the IT culture at IBM, it was successful because I didn’t have any technologically-enhanced blunders.

If you don’t have the opportunity to give yourself a new laptop every time you need to present, you could always have a clean virtual machine that consists of a basic OS with a PDF reader and presentation software.  That way, you don’t have to worry about getting any unnecessary things popping up inside that VM.  Just make sure to keep it updated from time to time to ensure your machine won’t pop up with a Windows Update restart prompt every 15 minutes during your slides.

Just some things to keep in mind when it’s time to jump up in front of a hostile crowd and start talking about how this information will change society or how your product is the greatest thing since sliced bread.  If you don’t have to worry about some of the more mundane things in the background of your presentation, you’ll knock their socks off with the content in your slides.  Just be sure not the mention Gartner.  That tends to get the natives restless.

2011 – Looking Forward

Posted on by
1

I almost wrote an end-of-year recap for this particular blog post.  I thought back to all of the things I’d accomplished over the past year.  It didn’t take me long to realize that I didn’t really keep track of them as well as I should.  The other thing that changed my mind was Greg’s great post about looking forward.  I’ve only been blogging for about 3 months.  I’ve really only had an online presence for about half the year.  So recapping what I’ve done wouldn’t really do much to help me take stock of what’s been going on.  But I’ve been trying to codify some things that I’m looking forward to in 2011 and I thought that putting them down in print would be a great way to make me own up to them.  So, without further ado, here’s what I’m looking forward to for the next 365 days.

1.  Passing the CCIE R&S lab. We are quickly getting to put-up or shut-up time when it comes to my CCIE lab.  I know that I’ve only failed when I decide to quit trying, but the trying is really starting to smart.  I’m in a unique position amongst some of my peers, in that my employer has been very gracious in allowing me to keep attempting the lab.  But I’m starting to feel like I’m imposing on their goodwill.  I’m starting to see a lot of RFPs being released that are requiring CCIE credentials to design what are essentially enhanced layer 2 networks.  I realize that these RFPs have been crafted in some degree to lock my employer out of consideration in the bidding process.  My pride tells me that I want to pass the lab for no other reason that to fly a big middle finger to them, as if to say “Ha! Guess what I did?!?”  In the end, I want to really succeed here because I’ve never let any test beat me, save one.  And I’m not about to let the CCIE become the second.

2.  Upgrade my VCP to version 4. The other thing that I do a lot of at my job that doesn’t revolve around networks concentrates on VMware.  I work with it more than I do with the actual OSes that get loaded to it, and I think it’s about time I made the move to getting certified on the current version.  There are some interesting possibilities that await should I manage to get there, including the idea of getting the VCAP4 – Design.  My job focus is quickly moving on toward building networks and systems on paper rather than physically, so some more designed-focused learning would do me some good.  But first things first.  I’ve got to get with the now.

3.  Start looking at the CCIE Voice. Heh, compared to #1, this one looks kind of silly.  Why start looking at another CCIE track when you aren’t even done with the one you started with?  If the truth be told, I’ve stuck with R&S as long as I have because of my stubborn streak.  I don’t work with BGP or MPLS in my every day job.  I doubt I ever will unless I switch roles and/or employers.  But I deal with voice every day.  It’s not what I started out to do, but I find it interesting.  And so I’m thinking that I might consider looking at some of the Voice courses and whether or not they appeal to me.  Who knows?  Maybe Voice will be an easier lab for me?

4.  Wikify my documentation. I’ve been putting this off for a lot longer than I should.  I need to take all of the information that I’ve gathered that resides on my laptop and put it into a form that other people can use and edit easily.  I want to have all of my knowledge in a place my peers can get to so that they might find the information they need quickly.  I want to clean up my haphazard note-taking style and make it readable.  I also want to be able to disappear for a few days at a time without getting ten phone calls and tons of e-mails.   I want to be able to pass the Bus Test.

5.  Start teaching more. Part of the reason that I started this blog was to collect all the random things that I come across and write them down in a place that I could easily find.  As an ancillary objective, I hope that other people might benefit from my research and study so that they could avoid the mistakes I’ve made.  I’ve considered bringing that into something a little more formal.  Some of my old college professors have talked to me about speaking to student groups.  My boss has discussed having me train user groups and train-the-trainer type scenarios.  I look at it as a two-fold opportunity.  I get to disseminate my knowledge, but I also gain the ability to tighten my presentation skills and put a little polish on my approach.  I don’t want to end up as a curmudgeon that sits behind a keyboard all day and loses all social ability.  I figure that forcing myself to get out and speak to people might just do that.

I figure five things should be a good list to work on.  Especially since  #1 is going to consume a lot of my time.  I hope to look back on this in 52 weeks and check off a few things.  I also hope that I can add a few more items to the list as I go.  Because surprises are always a good way to keep your edge sharp.

Changing CallManager’s IP Address

Posted on by
9

Network renumbering happens from time to time.  You outgrow a segment or buy a company and need to readdress things.  Or you start doing that new IPv6-thingy and need to renumber IPv4 to make more sense.  In any case, you go through everything that is attached to the network and make your changes.  The routers act just fine.  The switches couldn’t care less.  Even the toaster is still churning out perfect slices.  And then you get to CallManager.  As soon as you type in your password to the web administration page, ominous organ music starts playing in the back ground after a thunderclap.  You start to get the feeling that maybe this isn’t such a good idea.

There’s a good reason for that.  CallManager is very dependent on using IP addresses to communicate with the rest of the CUCM cluster (you did cluster your call processors, didn’t you???).  In fact, Cisco’s best practice is to use IP for communication with the cluster as opposed to relying on DNS.  You have to make the choice during the platform installation, and the only way to change is to completely reload the system.  For the purposes of this post, I’m going to assume you followed best practice and are using IP addresses for your communications.  When the time comes to change the IP addresses of your cluster members, you shouldn’t fret about the complexities.  That is, provided you have some patience and some familiarity with the CUCM command line interface.

1.  Make sure your cluster is healthy. This can’t be stressed enough.  If you’ve got database replication issues or network instability, this whole process is going to suck like a graviton-powered Hoover.  SSH to the publisher using your favorite SSH client and log in using the platform administrator login.  Once there, run this command:

show perf query class "Number of Replicates Created and State of Replication"

It should come back with the number of replicates created and what the replicate state is.  The state should say “2”.  This means your cluster is healthy and replicating like it should.  You could also see a “0” here if you only have one publisher and no subscribers.  This would be the case if you’re running a Business Edition (CUCMBE) all-in-one system.  Next, you should check the network connectivity with this command:

utils diagnose module validate_network

This command should return a status of “passed”.  If those command both succeed with no errors, your cluster is healthy and ready for the next step.

2.  Change the Subscriber Server IP in CUCM Administration. This is where my first IP change attempt failed spectacularly.  You need to change the IP address of the server in the CUCM Admin page before you do anything at the OS level.  This is due to the fact that the database services rely on the IP for a great number of things, and changing the IP at the OS level without changing the database first will cause the DB services to fail to start when you reboot. You need to change the subscribers before the publisher in order to maintain consistency.  First, login into the administration page, then go to System –> Server and select the node name for the subscriber you want to start with.  Change the IP address on this page to reflect the subscriber’s new IP address.  At this point, stop what you are doing and go get some coffee.  Just walk away from the PC for a few minutes.  While you’re up, I like my coffee with sugar and cream.

Back already?  And, I see you didn’t bring me any coffee.  Oh well.  We need to verify that the database has replicated the changes across the cluster before we start monkeying around with the OS configuration.  SSH into the publisher and run this command:

run sql select name,nodeid from ProcessNode

You should see the new IP address in the output table.  Now we need to actually change the subscriber’s IP.

3.  Change the Subscriber IP address in the OS. Fire up that SSH client again and connect to the subscriber you just changed in the publisher admin page.  Don’t worry, the address is still the same at the OS level, so you’ll be able to connect.  Once on the command line, you need to change the IP.  I recommend doing it this way to be sure the changes stick and make it a little easier to reboot the system afterwards.  If you’re moving the server to a different subnet, make sure to change the default gateway first like this:

set default gateway x.x.x.x

After that, or if you didn’t need to change the gateway, you need to change the IP address for the server.

set network ip eth0 x.x.x.x y.y.y.y

When you complete this command, you’re going to get a warning popup:

***   W A R N I N G   ***

If there are IP addresses (not hostnames)
configured in CallManager Administration
under System -> Servers
then you must change the IP address there BEFORE
changing it here or call processing will fail.
This will cause the system to restart
=======================================================
Note: To recognize the new IP address all nodes within
the cluster will have to be manually rebooted.
=======================================================

You can safely ignore this warning because you’re following these steps and you’ve already done this.  Type “Yes” at the prompt to force the subscriber to reboot.  Now would be a great time to enjoy that coffee.  It’s going to take a little longer than usual for the subscriber to come back up due to the changes to need to be made to the Tomcat server and the DB server.  At this point, if this is the only change you are making, Cisco recommends you reboot all the other cluster servers to change the name resolution and hosts file entries.  If you’ve still got more servers to do, go ahead and keep changing the subscribers as above.  Make sure the database replicates with the new IP before changing the OS IP address.  If your cluster is configured correctly, the phones should fail over to the active subscribers as you take down the other subscribers for the IP changes.  Once you’ve completed the subscriber IP changes, you’re almost done.

4.  Change the Publisher Server IP Addresses. The above steps are still correct for the the publisher server, except for one added step.  After you’ve returned with your fifth cup of coffee following the DB replication issues, you need to change the publisher IP on all the subscribers.  Go to the OS admin page of each one and go to Settings –> IP –> Publisher.  Once there, change the IP address of the Publisher server to the new IP you are going to set on the publisher.  If you happen to be a CLI junkie, you can do this from the comfort of your SSH program provided you’re on CUCM 6.1(2) or later with this command:

set network cluster publisher ip x.x.x.x

Be warned that you’ll need to reboot immediately after typing in that command.  Once that’s taken care of on all the subscribers, you can proceed to change the IP address in the OS admin of the publisher and then reboot.  After the publisher comes back up, communication should be restored and all the ominous organ music and thunderclapping in the background should stop.

Should you find yourself unlucky enough to use DNS for cluster identification, you’re going to want to start putting some Irish Crème in that coffee.  You’ll also want to refer to this page to get some information about the additional steps required to sort out the DNS mess on your system as you start changing IPs.

And there you have it.  With a little patience, a few cups of coffee, and a little CLI wizardry, even the mean, nasty CallManager can get a new IP address.  Just be sure to check the database replication after changing the IP in the system administration page.  Because if the DB changes haven’t replicated before you start changing OS settings, you’re going to have a bunch of fun getting things back to a good running state before you try again.  Good luck, and may the voice be with you.