I’s and T’s and Crosses and Dots

Posted on by
Reply

My name is Tom, and I’m careless.

Yep, I admit it freely.  I’m the kind of person that rushes through things and gets the majority of the work done.  Often I leave a few things undone with the hope that I’ll go back later and fix them.  For me, the result is the key.  Sometimes it works out in my favor, sometimes it doesn’t.  More often than not, I find myself cursing out loud about this unfinished job or task months down the road and threatening to find the person responsible, only to later determine that I should be kicking my own butt for it.

One place where this particular habit of mine has caused me endless grief in inside the unforgiving walls of Cisco’s Building C lab in San Jose.  Yep, I can honestly say that at least one lab attempt was foiled due to my propensity to miss the little things.  I’ve previously written about some of the details of the lab, but I wanted to take some time in this post to talk about the details themselves.  As in, the details in the questions that will kill you if you give them the chance.

Let’s get it out there right now: there is NO partial credit in the CCIE lab.  None. Zilch.  If you fail to answer every portion of the question with completeness, you get zero points for that question.  Unlike the old days in elementary school, you don’t get points for trying.  This shouldn’t really come as a shock to anyone that’s taken a multiple choice test any time in their life.  On those tests, there is exactly one set of answer(s) for a particular question, and if you don’t select the proper repsonse(s), you don’t get the points.  The same thing goes for the questions you find in the CCIE lab exam.  Just because the questions may or may not have multiple parts doesn’t excuse your need to answer them fully.  Old Mr. Hollingsworth used to tell me regularly, “Son, close only counts in horseshoes and hand grenades.”  Since I don’t play horseshoes and my hand grenade supplier mysteriously dried up, I guess close just won’t cut it any more.

You might end up getting a question in the lab that says something along the lines of “Configure OSPF on R1, R3, and R6 according to the diagram.  Do not change router IDs.  Rename R1 to ‘SnugglesR1’.”  You could build the most perfect OSPF lab in history.  You could spend an hour optimizing things.  If you forget to rename Snuggles the Router, you will receive no credit for the question.  All that hard work will get flushed down the toilet.  You’ll get your score report at the end of the day and wonder why you didn’t get any points for all that time you spend making OSPF sing like a soprano.

In order to prevent this from happening to you, start training yourself now to read carefully and consider every facet of the questions you’ll see.  Remember that the questions in the lab are carefully constructed by a team that spends a ton of time evaluating every part.  There are no unnecessary words.  Candidates have pestered proctors over the meaning of single words on a question.  The questions are written as they are to make sure you take into account a number of factors.  They are also designed to slip in changes to tasks and additional configuration with a word or two.  And if you are careless, you’ll miss those phrases that signal changes and negations.

Surely, everyone has taken a test that has a question that says “Which of the following was NOT a <something> <something>”  Your job is to evaluate the choices and pick the one that is not something.  That single word changes the whole meaning of the question.  And for those that are careless or the kind the skim questions, the NOT might be missed and cause them to answer incorrectly.  Questions in the lab are the same way.  Skimming over them without reading critically can cause nuances to be missed and lead to incorrect solutions.  After 5 hours of staring at words on a monitor, things might start blurring a little, but attention must be paid to the last few questions, as those might be enough points to buoy over the passing mark.

I’ll be the first to admit that the pressure to get everything done in the allotted time may cause the candidate to want to rush, but you must resist that pressure.  Many CCIE lab prep courses and instructors will tell you to carefully read the questions before you ever start configuring.  I agree, with some additions.  I always take my scratch paper and write the task numbers down the side.  After I’ve accounted for Task 1.1, 1.2, 2.1, and so on, I then go back to the questions and make marks next to my list for any questions that may have multiple parts or tricky solutions.  That way, if I find myself rushing through after lunch the marks I made early in the day force me to pay attention to the question and ensure that I don’t miss something that might cause me to tank three or four points.  Those points add up over the course of the day, and more than a few careless mistakes can cost you a nice expensive soda can.

If you are serious about the CCIE lab, it’s worth your time to start working on ensuring that you pay close attention to each question and don’t make any careless mistakes due to reading too fast or missing important configuration requirements.  Your day is going to be stressful enough without the added pressure of fixing mistakes later in the lab as a result of forgetting to enable OSPF authentication or a typo on a VLAN interface.  You want to remember to dot every “i” and cross every “t” for each and every question.  That way, you can walk out of the lab and use that freshly-dotted “i” when you spell you new title as a CCIE.

The Nerd’s Going to Have a Field Day!

Posted on by
1

Guess who gets to go to San Jose for something OTHER than the lab for once?  That’s right, THIS GUY <—–!

Thanks to the wonderful folks over at Gestalt IT, I have been invited to be a delegate at Tech Field Day 5!  This is a tremendous opportunity for me to get involved with the sponsoring technology companies and hear about their products and strategies for the coming months.  Tech Field Day offers engineers and technical people the chance to hear about these great things in an environment conducive to learning (i.e. no sales pitches).  I’ve read the coverage of the previous Tech Field Day events from some of my other blogging brethren (and sistren) and found the information that they’ve given the community to be quite valuable.  I’m looking forward to the opportunity to spend some time listening to the best and brightest that will be in front of the delegates, and of course I can’t wait to pick their brains about technology!

Disclosure

What would an event be without some kind of disclosure?  In this case, the TFD sponsors are paying for my travel costs and lodging costs during my stay in San Jose.  That being said, they are NOT putting any limitations on what I say about the information I receive or the feelings that I have about the conference, other than to respect any information that might not be public knowledge or embargoed (which I would have done anyway).  I can assure you that any and all opinions expressed about the content of TFD are mine and mine alone.  I want to give you the unvarnished truth behind what I see and hear.

How Do I Get Involved?

Gestalt IT is always keeping an eye out for TFD delegates.  I’ll admit that while I have wanted to be a delegate for a while now, I never thought I’d be one until I was asked.  There are some things to keep in mind:

1.  Read the TFD FAQ and the Becoming a Field Day Delegate pages for more information on each.  Those pages are the best source of information about the process and criteria for TFD and its delegates.

2.  Understand there isn’t some kind of strange conspiracy or secret machine driving this.  These people look for independent critical thinkers that aren’t afraid to voice their opinions about subjects.  Don’t be afraid to show your independence and be sure to speak about technical subjects.

3.  Ask questions.  Anywhere and everywhere.  You never learn if you don’t ask questions.  I question things all the time.  And if you have any questions that you’d like me to ask at TFD, please let me know in the comments.  I’ll be making more posts as the list of presenters becomes final so that you know who I’ll be interacting with.

Be sure to follow @TechFieldDay on Twitter for more information about TFD 5 as the date approaches in February.  You can also follow the #TechFieldDay hash tag for updates live as the delegates tweet about them.  For those of you that might not want to see all the TFD-related posts, you can also use the #TechFieldDay tag to filter posts in most major Twitter clients.

For more information about sponsors and delegates, head on over to the TFD5 page on GestaltIT.

The Nerd Presents: Tips for Presenting

Posted on by
5

Everyone in the world has at least one good presentation in them.  It doesn’t take much to put something down in a few slides and talk about it.  For most people, the hardest part is getting up in front of a group and actually speaking.  Once you get over that, the rest is easy.  However, in my job I get to listen to a lot of presentations.  I’ve had a lot of time to look beyond the content to things that tarnish your image when in front of customers or learners.  I won’t profess to be an expert when it comes to the art of presentation, but I think most would agree with me that looking at these tips will help out in the polishing department.

– Close down Outlook and turn off your mail notifications. As professionals, we are all married to Outlook/Thunderbird/Entourage.  No matter what it seems impossible to escape it today with the ability to load it on our desktops, laptops, and mobile devices.  However, when you stand in front of me to start pitching your software or tell me about a new technology, please turn off your mail client and notification system.  Think about it like this: you don’t leave your cell phone ringer on when your presenting because of the distraction.  Why would you leave the new mail popup in the corner?  At best, it causes me to shift my focus from the content of the slide deck to whatever new message you just received.  At worst, I may be privy to inside information from your company, sales targets and customers, or in rare and somewhat ironic cases, end of life notices for the very product you are trying to sell me.  Ask yourself this question: If you were listening to me tell you about how great my Project Foobar is and I receive an email from my lead product specialist with the subject “Inability of Project Foobar to Address Basic Business Needs” would you still be interested in hearing my pitch?

– Shut down instant messengers. For that portion of the crowd that thinks email is so yesterday, there is the instant messenger (IM).  People use a variety of clients, from the tried-and-true AOL instant messenger to newer things like Trillain or Pidgin or even Cisco Unified Personal Communicator.  Guess what?   Shut it down before you start talking to me.  All of the reasons above still apply to IM conversations.  In the case of IM though, people are a lot more informal.  So conversations may not start out with simple hellos.  You may get something more pointed or perhaps a greeting too salty for the taste of the group your are presenting to.  Imagine a co-worker sending you profanity laced tirade during a speaking engagement with a Catholic school.  Or something leaning toward the more delicate and personal from your spouse when you are speaking to a prospective customer.  The ability to embed pictures in IMs makes this prospect even scarier.  And before you say “I can just set myself to away” think about all the times that an “emergency” has come up and you’ve been pinged on IM even when your away.  Chalk that particular one up to most people assuming that “away” means “I’m sitting right here and I just don’t want to talk to you right now”.  Better to just shut yourself off from the IM cloud for a while and not take any chances.

– Change your desktop wallpaper to something bland. I’m guilty of this one, so allow me to start casting stones.  I like wallpapers.  Generally something abstract or landscape oriented.  I do have the occasional cool picture of something fire and ice related.  But for the most part, I tend to avoid pictures of people or animals or quotes.  Especially if they could be construed as the least bit offensive.  But even my conservative taste in wallpaper can be distracting when presenting.  You say, “But no one is going to see my desktop if Powerpoint is up the whole time.”  True enough, but how many times are you only using Powerpoint?  What happens if you have to switch slide shows?  Or look at a document on your desktop?  Or switch to a web browser to load a live video?  There are a variety of reasons to jump out of Powerpoint, and if you don’t think ahead of time, you might just find yourself showing a picture of your last trip to Cancun to all of the members of your church group.  Even in the case that it’s a picture of your newborn daughter, your presentation focus will be lost as people start cooing about how cute she is, how old she is, whether or not she’s sleeping yet, whether or not you’re sleeping, ad infintium.  In my book, it’s best just to change your wallpaper to basic black and move on.

– Collect all your documents related to the presentation in one folder on your desktop. Most of my presentations are loaded with technical content.  Many of them, however, don’t have the density of the documents I used to put them together.  Making my slides into eye charts won’t help my audience understand my topic any better.  But if I mention that there is a document that includes more technical depth to this particular subject, invariably someone is going to ask to see that document.  Or ask about a fact or figure from it.  That leads to me needed to go spelunking through my file system to find it.  Call me somewhat old-fashioned, but I don’t really like people staring at my file structure and folder contents.  Especially if those folders contain competitive information.  What might happen if my customer sees a document named “Juniper ASA Comparison and Debunking.pdf”?  Sure, if I’m presenting one of those products it shouldn’t really matter, right?  But what if the other product is one that the customer has never heard of?  Yeah, if you’re researching firewalls and you’ve gotten to the point of hearing a presentation about one, hopefully you know about the other.  But in my mind, just the presence of that document could derail your presentation with questions that might not be pertinent to the discussion at hand.  Better to copy all of the relevant documents that you have sourced from your presentation into a folder labeled “Presentation Documents” and put it on your desktop so you don’t spend precious minutes searching for it.  And while you’re at it, consider changing your browser’s homepage if you shell out to the Internet during presentations.  Google is a good safe bet.  Your sports book?  Not so much…

– Don’t read the slides back to me. Pet. Peeve. Number. One.  Don’t read your slides back to me.  I’ve walked out of presentations that I’ve paid for the honor of attending for this gaffe.  If you are reading the slides back to me word-for-word, it tells me you’ve done no research on the topic and you have no depth on knowledge on the subject.  Marketing people are the worst when it comes to this.  They just assume that what has been printed on the slide is the definitive answer to everyone’s problems and just start reading it to me like gospel.  Guess what?  I can read too!  As you’re outlining the contents of that slide, I’ve already glanced over it and picked out the most relevant pieces of information that interest me.  If you then start at the top and read the bullet points to me, I going to guess this is all new to you too.  I treat my slide deck like I would treat a stack of 3”x5” index cards that I use for notes.  I expand on each of the bullet points in my slide deck with additional discussion topics.  That’s also one of the reasons I print my slide deck ahead of time and make it available to the people that I speak to.  That way, they can jot down the notes I speak about and reference them against the printed slides.  The way I see it, you came to see me speak, not look at my fancy multiple-build-slide transition heavy corporate approved 100-slide deck.  If you want me to read the slides back to you, it’s going to feel way too much like circle time in my son’s kindergarten class.

Many, many moons ago I was an intern at IBM in Rochester, MN.  My first-line manager decided that the other intern and I needed to get some practice giving presentations to clients/customers.  She therefore decided to make us present some Windows 2000 tips to a group of users that had recently received new Thinkpad T20s (how’s that for dating myself?).  After I had put together my slides, my mentor told me that I needed to go grab a brand new laptop from the laptop pool and use it instead of my personal machine.  What I questioned her reasoning, she told me that by using a fresh laptop out of the box, the usual cruft that come along with my personal machine would be absent.  I wouldn’t need to worry about some of the things I’ve listed above, like Outlook (in this case Lotus Notes) or my desktop wallpaper.  I could concentrate on my presentation.  And while I won’t say that her advice made my presentation into something that changed the fabric of the IT culture at IBM, it was successful because I didn’t have any technologically-enhanced blunders.

If you don’t have the opportunity to give yourself a new laptop every time you need to present, you could always have a clean virtual machine that consists of a basic OS with a PDF reader and presentation software.  That way, you don’t have to worry about getting any unnecessary things popping up inside that VM.  Just make sure to keep it updated from time to time to ensure your machine won’t pop up with a Windows Update restart prompt every 15 minutes during your slides.

Just some things to keep in mind when it’s time to jump up in front of a hostile crowd and start talking about how this information will change society or how your product is the greatest thing since sliced bread.  If you don’t have to worry about some of the more mundane things in the background of your presentation, you’ll knock their socks off with the content in your slides.  Just be sure not the mention Gartner.  That tends to get the natives restless.

2011 – Looking Forward

Posted on by
1

I almost wrote an end-of-year recap for this particular blog post.  I thought back to all of the things I’d accomplished over the past year.  It didn’t take me long to realize that I didn’t really keep track of them as well as I should.  The other thing that changed my mind was Greg’s great post about looking forward.  I’ve only been blogging for about 3 months.  I’ve really only had an online presence for about half the year.  So recapping what I’ve done wouldn’t really do much to help me take stock of what’s been going on.  But I’ve been trying to codify some things that I’m looking forward to in 2011 and I thought that putting them down in print would be a great way to make me own up to them.  So, without further ado, here’s what I’m looking forward to for the next 365 days.

1.  Passing the CCIE R&S lab. We are quickly getting to put-up or shut-up time when it comes to my CCIE lab.  I know that I’ve only failed when I decide to quit trying, but the trying is really starting to smart.  I’m in a unique position amongst some of my peers, in that my employer has been very gracious in allowing me to keep attempting the lab.  But I’m starting to feel like I’m imposing on their goodwill.  I’m starting to see a lot of RFPs being released that are requiring CCIE credentials to design what are essentially enhanced layer 2 networks.  I realize that these RFPs have been crafted in some degree to lock my employer out of consideration in the bidding process.  My pride tells me that I want to pass the lab for no other reason that to fly a big middle finger to them, as if to say “Ha! Guess what I did?!?”  In the end, I want to really succeed here because I’ve never let any test beat me, save one.  And I’m not about to let the CCIE become the second.

2.  Upgrade my VCP to version 4. The other thing that I do a lot of at my job that doesn’t revolve around networks concentrates on VMware.  I work with it more than I do with the actual OSes that get loaded to it, and I think it’s about time I made the move to getting certified on the current version.  There are some interesting possibilities that await should I manage to get there, including the idea of getting the VCAP4 – Design.  My job focus is quickly moving on toward building networks and systems on paper rather than physically, so some more designed-focused learning would do me some good.  But first things first.  I’ve got to get with the now.

3.  Start looking at the CCIE Voice. Heh, compared to #1, this one looks kind of silly.  Why start looking at another CCIE track when you aren’t even done with the one you started with?  If the truth be told, I’ve stuck with R&S as long as I have because of my stubborn streak.  I don’t work with BGP or MPLS in my every day job.  I doubt I ever will unless I switch roles and/or employers.  But I deal with voice every day.  It’s not what I started out to do, but I find it interesting.  And so I’m thinking that I might consider looking at some of the Voice courses and whether or not they appeal to me.  Who knows?  Maybe Voice will be an easier lab for me?

4.  Wikify my documentation. I’ve been putting this off for a lot longer than I should.  I need to take all of the information that I’ve gathered that resides on my laptop and put it into a form that other people can use and edit easily.  I want to have all of my knowledge in a place my peers can get to so that they might find the information they need quickly.  I want to clean up my haphazard note-taking style and make it readable.  I also want to be able to disappear for a few days at a time without getting ten phone calls and tons of e-mails.   I want to be able to pass the Bus Test.

5.  Start teaching more. Part of the reason that I started this blog was to collect all the random things that I come across and write them down in a place that I could easily find.  As an ancillary objective, I hope that other people might benefit from my research and study so that they could avoid the mistakes I’ve made.  I’ve considered bringing that into something a little more formal.  Some of my old college professors have talked to me about speaking to student groups.  My boss has discussed having me train user groups and train-the-trainer type scenarios.  I look at it as a two-fold opportunity.  I get to disseminate my knowledge, but I also gain the ability to tighten my presentation skills and put a little polish on my approach.  I don’t want to end up as a curmudgeon that sits behind a keyboard all day and loses all social ability.  I figure that forcing myself to get out and speak to people might just do that.

I figure five things should be a good list to work on.  Especially since  #1 is going to consume a lot of my time.  I hope to look back on this in 52 weeks and check off a few things.  I also hope that I can add a few more items to the list as I go.  Because surprises are always a good way to keep your edge sharp.

Changing CallManager’s IP Address

Posted on by
9

Network renumbering happens from time to time.  You outgrow a segment or buy a company and need to readdress things.  Or you start doing that new IPv6-thingy and need to renumber IPv4 to make more sense.  In any case, you go through everything that is attached to the network and make your changes.  The routers act just fine.  The switches couldn’t care less.  Even the toaster is still churning out perfect slices.  And then you get to CallManager.  As soon as you type in your password to the web administration page, ominous organ music starts playing in the back ground after a thunderclap.  You start to get the feeling that maybe this isn’t such a good idea.

There’s a good reason for that.  CallManager is very dependent on using IP addresses to communicate with the rest of the CUCM cluster (you did cluster your call processors, didn’t you???).  In fact, Cisco’s best practice is to use IP for communication with the cluster as opposed to relying on DNS.  You have to make the choice during the platform installation, and the only way to change is to completely reload the system.  For the purposes of this post, I’m going to assume you followed best practice and are using IP addresses for your communications.  When the time comes to change the IP addresses of your cluster members, you shouldn’t fret about the complexities.  That is, provided you have some patience and some familiarity with the CUCM command line interface.

1.  Make sure your cluster is healthy. This can’t be stressed enough.  If you’ve got database replication issues or network instability, this whole process is going to suck like a graviton-powered Hoover.  SSH to the publisher using your favorite SSH client and log in using the platform administrator login.  Once there, run this command:

show perf query class "Number of Replicates Created and State of Replication"

It should come back with the number of replicates created and what the replicate state is.  The state should say “2”.  This means your cluster is healthy and replicating like it should.  You could also see a “0” here if you only have one publisher and no subscribers.  This would be the case if you’re running a Business Edition (CUCMBE) all-in-one system.  Next, you should check the network connectivity with this command:

utils diagnose module validate_network

This command should return a status of “passed”.  If those command both succeed with no errors, your cluster is healthy and ready for the next step.

2.  Change the Subscriber Server IP in CUCM Administration. This is where my first IP change attempt failed spectacularly.  You need to change the IP address of the server in the CUCM Admin page before you do anything at the OS level.  This is due to the fact that the database services rely on the IP for a great number of things, and changing the IP at the OS level without changing the database first will cause the DB services to fail to start when you reboot. You need to change the subscribers before the publisher in order to maintain consistency.  First, login into the administration page, then go to System –> Server and select the node name for the subscriber you want to start with.  Change the IP address on this page to reflect the subscriber’s new IP address.  At this point, stop what you are doing and go get some coffee.  Just walk away from the PC for a few minutes.  While you’re up, I like my coffee with sugar and cream.

Back already?  And, I see you didn’t bring me any coffee.  Oh well.  We need to verify that the database has replicated the changes across the cluster before we start monkeying around with the OS configuration.  SSH into the publisher and run this command:

run sql select name,nodeid from ProcessNode

You should see the new IP address in the output table.  Now we need to actually change the subscriber’s IP.

3.  Change the Subscriber IP address in the OS. Fire up that SSH client again and connect to the subscriber you just changed in the publisher admin page.  Don’t worry, the address is still the same at the OS level, so you’ll be able to connect.  Once on the command line, you need to change the IP.  I recommend doing it this way to be sure the changes stick and make it a little easier to reboot the system afterwards.  If you’re moving the server to a different subnet, make sure to change the default gateway first like this:

set default gateway x.x.x.x

After that, or if you didn’t need to change the gateway, you need to change the IP address for the server.

set network ip eth0 x.x.x.x y.y.y.y

When you complete this command, you’re going to get a warning popup:

***   W A R N I N G   ***

If there are IP addresses (not hostnames)
configured in CallManager Administration
under System -> Servers
then you must change the IP address there BEFORE
changing it here or call processing will fail.
This will cause the system to restart
=======================================================
Note: To recognize the new IP address all nodes within
the cluster will have to be manually rebooted.
=======================================================

You can safely ignore this warning because you’re following these steps and you’ve already done this.  Type “Yes” at the prompt to force the subscriber to reboot.  Now would be a great time to enjoy that coffee.  It’s going to take a little longer than usual for the subscriber to come back up due to the changes to need to be made to the Tomcat server and the DB server.  At this point, if this is the only change you are making, Cisco recommends you reboot all the other cluster servers to change the name resolution and hosts file entries.  If you’ve still got more servers to do, go ahead and keep changing the subscribers as above.  Make sure the database replicates with the new IP before changing the OS IP address.  If your cluster is configured correctly, the phones should fail over to the active subscribers as you take down the other subscribers for the IP changes.  Once you’ve completed the subscriber IP changes, you’re almost done.

4.  Change the Publisher Server IP Addresses. The above steps are still correct for the the publisher server, except for one added step.  After you’ve returned with your fifth cup of coffee following the DB replication issues, you need to change the publisher IP on all the subscribers.  Go to the OS admin page of each one and go to Settings –> IP –> Publisher.  Once there, change the IP address of the Publisher server to the new IP you are going to set on the publisher.  If you happen to be a CLI junkie, you can do this from the comfort of your SSH program provided you’re on CUCM 6.1(2) or later with this command:

set network cluster publisher ip x.x.x.x

Be warned that you’ll need to reboot immediately after typing in that command.  Once that’s taken care of on all the subscribers, you can proceed to change the IP address in the OS admin of the publisher and then reboot.  After the publisher comes back up, communication should be restored and all the ominous organ music and thunderclapping in the background should stop.

Should you find yourself unlucky enough to use DNS for cluster identification, you’re going to want to start putting some Irish Crème in that coffee.  You’ll also want to refer to this page to get some information about the additional steps required to sort out the DNS mess on your system as you start changing IPs.

And there you have it.  With a little patience, a few cups of coffee, and a little CLI wizardry, even the mean, nasty CallManager can get a new IP address.  Just be sure to check the database replication after changing the IP in the system administration page.  Because if the DB changes haven’t replicated before you start changing OS settings, you’re going to have a bunch of fun getting things back to a good running state before you try again.  Good luck, and may the voice be with you.

The Recertification Treadmill

Posted on by
2

I like tests.  Probably a lot more than I should.  Oh, it wasn’t always like this.  I dreaded test days in college.  Cramming chapters worth of information into my brain so that it could just be regurgitated later and forgotten shortly after than.  In fact, I can distinctly remember studying the OSI model for one of my IT infrastructure classes and thinking, “I only need to remember this for the exam.  After that, I’ll never see it again.”  Of course, that same OSI model is now permanently tattooed on the insides of my eyelids.

Then I entered the Real World.  I found out about certification tests and all they entail.  You mean I can take one test proving my mastery of a subject and you guys send me a certificate and a little wallet card?  Sign me up!  It also helped that my employer is a partner with multiple vendors and needed me to take as many tests as I could to keep their partner status up-to-date.  So I set off on my odyssey of test taking.  I’ve got certifications from Novell, Microsoft, CompTIA, Cisco, HP, (ISC)2, and many more.  I’ve taken enough tests that the test administrator at my local testing center recognizes me in the street.  I know more about the ins and outs of testing procedure than most people should.  And, I’ve been handsomely rewarded for my test taking prowess.  And, for the most part, I’ve enjoyed every second of my learning.  Except for recert day.

Yes, every once in a while one of the vendors sends me a note that says I’m due for renewal.  My professional title is now in jeopardy if I don’t study some new information and go see my local Pearson/Prometric guru.  So I start pouring over material in an effort to not need new business cards.  I cram all that new information in my stuffed head and run out to take the test again.  And I pass.  And for a while, I’m a golden boy again.  Until recert day comes up again.

Some vendors tell  you that you can keep your certification for ever and ever.  Like my MCSE.  Of course, I’m not technically “current” with that one, especially now that the new title is MCITP (or something like that).  So, while I’m a whiz when it comes to Windows 2000, I’m not really authorized on the new hotness of Server 2008.  Oh well.  Other vendors, like Cisco, keep the same certification title, but they change the tests around from time to time.  Like my CCVP.  I originally certified on CUCM 4.1.  Back when there was a separate test for those gatekeeper thingies.  And then Cisco went and released a new CCVP track about CUCM 6.x.  I didn’t have to recertify because my CCVP was still good.  But now, they have eliminated the CCVP and changed the voice certification track to the CCNP: Voice.  You can still take the CCVP tests and get grandfathered in before the change to the new CUCM 8.x material if you want.  And that’s what I found myself doing about 2 months ago.  I figured since I worked with voice everyday it shouldn’t be too rough to just jump in and take the tests.  My reasoning was that the partner requirements for Advanced Unified Communications would change after the CCVP –> CCNP: Voice move, so I wanted to get out in front of this change before I was forced to.  I managed to stumble through the troubleshooting test and both CallManager tests in fairly short order.  As I brushed up on my CVOICE basics, I remembered that a previous visit to the Certification Tracker showed that I hadn’t taken the QoS exam, even though I distinctly remembered the pain and agony of that one.  I wrote in to Cisco Cert Support, hoping that I didn’t have to go through it all over again.  While I kept studying for my CVOICE test, I got the response.  It seems that those tests expire after 3 years, and I would need to retake it again for it to be valid.  However, according to Cisco, I was already a CCNP: Voice, so I wouldn’t need to retake it.  Huh?  When did that happen?

Cisco’s recertification policy for professional level exams says that taking any professional test with a ‘642’ prefix will recertify your CCxP.  Little did I know at the time that my first test, Troubleshooting Unified Communications, had recertified my CCVP and triggered the upgrade to a CCNP:Voice.  So, the CUCM tests were for naught.  The CVOICE test did give me a CCNA: Voice tag, so I’ve got that going for me now.  The Cisco recert cycle is nothing new to me.  I’ve been taking the CCIE written exam every year because it’s the only way to keep my specialist designations current.  In order to keep my employer in the good partner graces, I have to keep remembering OSPF and MPLS trivia and take the CCIE written at least every two years.  It’s the only way for me to keep my certifications current without devoting all my time to studying and taking tests instead of doing the job I was hired for.  I was confused in this particular instance with the CCNP: Voice because the certification website never said anything about there being an upgrade path from my 4.2 CCVP to the 8.x CCNP: Voice.  I’m happy nonetheless, but I started thinking about the whole recertification process and why it bothers me somewhat.

I can take any 642 level Cisco exam and recertify all my CCxA and CCxP titles.  I can take the CCIE written and do the same, including my specialist tags.  VMware makes me take a new test and sit through 5 days of training to get a VCP4.  Microsoft wants me to take a whole new set of tests to become a new MCSE/MCITP.  Novell just keeps certifying me on Linux stuff even though I haven’t taken Novell test in years.  And we won’t talk about HP.  Ethan has a great post about recerting his CCIE that hits on a lot of good points.  Normally, we have to either shut down our productivity for a few weeks to get into the recertification groove, or try and find time outside of work to study.  Either way, it seems like a colossal waste of time. It’s almost like being elected to the House of Representatives.  You need to start campaigning for re-election right after you’ve been elected.  It’s just annoying that I have to take time out of my schedule to relearn things I’m already doing.  Is there any way to fix this?

Find a lawyer.  Any lawyer.  If you’re having trouble, check behind the nearest ambulance.  Now, ask them how many times they’ve retaken the bar exam.  Odds are good they’ll stare at you and tell you that you’ve lost your mind.  Lawyers don’t have to resit the bar exam every time they need to renew their fancy degree.  They are allowed to use Continuing Professional Education credits.  All they have to do is take a class or attend a conference and they can count that learning toward recertifying their degree and certification requirements.  IT people are the same.  We spend a lot of our time watching webcasts and going to trade shows.  I go to Cisco Live Networkers almost every year.  When I’m there, I take the opportunity to learn about technologies I don’t encounter in my every day job, like TRiLL or FabricPath.  I’m doing an awful lot to keep current with trends and technology in the industry, and it feels like it’s all for my own edification.  It doesn’t really count toward anything.  Except in one case – my CISSP.  Because (ISC)2 uses a CPEs too.

The vendor-neutral certification bodies have it right, in my opinion.  (ISC)2, BICSI, and CWNP all have a CPE policy.  They say that you can go to conferences or read books and count that learning toward your certification.  They want you to prove that you’re staying current, and in return they’ll make sure you are current when it comes to certifications.  Sure, in the case of the CISSP, most of the learning needs to be focused on security, but that’s how it should be.  I can count some amount of general education credits toward my CISSP, but the bulk of the education needs to be focused on the subject matter of the certification.  I think something like this would be a great addition to Cisco’s arsenal.  Give your certified professionals a chance to apply the learning they do every day toward recertification.  You’d sell more Cisco Press books if I knew I could read one and count 5 points toward my CCSP.  There’d be even more attendees at Networkers if it counted for 40 CPEs every year.  But, there also need to be some restrictions.

Some vendors don’t like the idea that one test can recertify all your titles.  Juniper doesn’t.  So make sure that the education credits only count toward a specific area of knowledge.  The Migrating CUCM class from Brandon Ta that I go to every year could count toward my CCVP, but not my CCSP.  My TRiLL webcasts could count for points to recertify my CCIE R&S or SP, but not the CCIE Wireless.  If you marry the education to a specific certification, you’ll see much higher attendance for those kinds of things.  For people like us that spend time writing about things on the Interwebs, authoring articles for places like Network World or Information Week could count as well, since you are disseminating the knowledge you’ve obtained to the masses.  Even teaching could count toward recertifying.

This idea is not without issue, though.  The first argument is that allowing certified individuals to use CPEs might cause problems with the cottage industry that has sprung up around teaching these subjects to people.  Ask yourself, How many people would go to VMware classroom learning if it wasn’t required to obtain the VCP?  I’m sure the answer would be “A whole lot less.”  It’s no secret that Cisco and HP and Microsoft make a lot of money offering classes to people in order to get the certified on technology.  Companies can specialize in just teaching certification coursework and turn a tidy profit.  And these same companies might not be too keen on the idea of a revenue stream drying up because Cisco or Novell decided to be noble and not require everyone to take a new test every 2 years.

Another consequence, though one for the better, would be the contraction of the “braindump” market.  A lot of people talk about the braindump market catering to those who want a fast track to the CCNA or other entry-level cert.  I’m of the opinion that a larger portion of the dumping population consists of already-certified individuals that have neither the time nor the energy to study for a recertification exam.  These people are facing a deadline of needing to stay current with whatever alphabet soup comes after their name, except now that they have a steady job they don’t have the time to devote to studying all night to pass.  Faced with the option of letting their certification expire, or paying money to someone for the answers to the test, they swallow their pride and take the easy way out.  In their mind, no harm is done because they were already a CCxA in the first place.  They know the material, they just don’t have time to remember what the “vendor answer” is on the test.  Now, give these same people the opportunity to apply a webcast or vendor presentation that they’d sit through anyway to that CCxA.  I bet that more than half the dumping sites would go away within a year.  When the market starts drying up, it’s time to move on.

I really hope that the vendors out there take the time in 2011 to reassess their recertification strategies.  Giving certified professionals more options when it comes to proving they know their material can only build goodwill in the future.  Because the current method feels way too much like a treadmill right now.  I keep running in place as fast as I can just to stay where I’m at.  I think things need to change in order to make the education and learning that I do have a tangible impact on my certification progress.  Because sooner or later I’m not going to be able to keep up with the recertification treadmill.  And we all know what the result is when that happens…

COBRAS!

Posted on by
Reply

If you are a voice networking professional, and you are tasked with working on any Cisco voice mail product, do yourself a favor and go download the Cisco (Unified) Backup and Restore Application Suite (COBRAS).  You’ll thank me later.

If you’ve ever tried to upgrade Unity, you know the pain that comes from the good old Disaster Recovery Tool (DiRT).  Cisco’s best practice for upgrading Unity involves using DiRT to backup your existing database, installing a new server, installing the old version of Unity and performing a restore, then attempting to upgrade the new server to the new version of Unity.  Any one of those steps if fraught with danger and terror.  DiRT was never really designed to do upgrades.  It was only ever meant to restore your Unity configuration and data in the event of a meteor strike or alien invasion.  Other than those two corner cases, it pretty much sucks.  In case you couldn’t tell, I’m not a fan of DiRT.  It’s like magnetic tape.  It serves one purpose that it’s good for, but if you start getting creative you are asking for trouble.

When Cisco started pushing Unity Connection as a viable alternative to Unity, the need arose to find a way to get the data out of Unity and import it into Unity Connection.  This is not a job for DiRT.  The worst-case-scenario is that you need to pop up two web browsers and input the information from one system into the other.  Not a fun job if you have even 50 mailboxes.  A nightmare if you have a few thousand.  And, you can’t move user passwords and PINs.  A better solution needed to be found.  And, thanks to some enterprising TAC engineers, we have COBRAS.

COBRAS started its life as a very unsupported tool on the Cisco Unity Tools website.  Anyone that has worked on Unity for more than five minutes has been to this website.  At various points in its life, it has been referred to as AnswerMonkey.net and LindborgLabs.com.  My best guess is that it started as a repository run by some TAC engineers for the purpose of giving the long-suffering Unity support people a place to download tools and scripts to help get Unity working as properly as you can for a program that requires you to speak in tongues to fix it.  About three years ago, I had the good fortune to take a class at Cisco Live Networkers that dealt with the problem of migrating CallManager and Unity to newer versions.  I specifically took the class because I was about to face an unpleasant customer upgrade from CUCM 4.1 to 6.1.  At the same time, the customer wanted to move from Unity 4 to 5.  All of the official documentation I read said that the DiRT migration was the best way to move to new hardware.  Luckily, the class from Brandon Ta at Cisco Networkers pointed me in the direction of COBRAS.  Of course, when I went to download it from the Unity Tools website, the warning messages and dire predictions told me why I hadn’t seen it before – it wasn’t quite supported.  As in, it wasn’t supported at all.   Still, faced with the choice of something that I knew I wouldn’t like or the idea of trying something new that had little support, I bit the bullet and went with the new idea.  And boy did I like it.  COBRAS pulled all the Unity information from the old system in short order.  When I brought the new system online, a quick import into Unity 5 got the voicemails flowing again in no time.  Rather than spend hours waiting for the inevitable issues with DiRT restores, I could instead concentrate of cursing the Data Migration Assistant.  But that’s another story entirely.

Flash forward to this past month.  We’d been running Unity at our office for several years, and my users had become very dependent on unified messaging.  When the Windows admins decided to upgrade to Exchange 2007, they forgot to warn me about when they had planned on doing this.  So, when I walked the next morning, the voicemail integration was offline.  It took a couple of hours before I was able to install the correct engineering special and repair all the Unity permissions with the scripts from the Grand Unity Grimoire.  I’ve known that an upgrade to Exchange 2010 is in the works at some point.  I’ve also grown tired of the difficulties that Unity presents.  I can only administer it from Internet Explorer.  The need to keep Active Directory healthy just for the sake of Unity was annoying.  The need to know intimate details about Exchange 2003 made me cringe.  Even my conversations with product managers from Cisco didn’t leave me all that excited about Unity.  But, I couldn’t move to Unity Connection just yet.  Because of the unified messaging issue.  My coworkers couldn’t fathom the idea of NOT getting voicemails in one inbox.  IMAP wasn’t going to cut it.  So I bided my time and plotted the demise of Unity.  When Cisco formally announced the feature set for Unity Connection 8.5, I jumped for joy.  Unity Connection 8.5 contains a unified messaging agent that allows you to synchronize your Unity Connection mailstore with Exchange 2003, 2007, and 2010.  My users would receive the same benefits as they had now with Unity, and I would get a unified management platform on the back end that was no longer soulbound with Exchange.

Once I got Unity Connection installed on a spare server, I needed to export 50 mailboxes worth of data from Unity.  I checked Cisco Unity Tools and found they had updated COBRAS to support the latest versions of Unity Connection, and in fact COBRAS was now supported by TAC!  It felt like this angel finally got its wings!  I downloaded the Unity Export and Connection Import programs and installed them.  The Unity Export program needs to be installed on Unity.  In fact, it’s only support on Windows Server platforms.  The Connection Import program can be installed on any Windows system, but you also need to install the IBM Informix Database drivers to allow communication with the Unity Connection database directly.  I resorted to installing the program and database drivers in Windows XP virtual machine, as my 64-bit Windows 7 installation has already show an intolerance for drivers in general, which my USB-to-serial adapter will attest to.  Once I installed the programs, I exported all the configuration data from Unity in one shot.  It took all of 20 minutes.  I was shocked.  I had fully expected this whole export to eat up my entire afternoon.  When I went to export the voicemail messages from the database, I found that I needed to be logged in as the “UnityMsgStoreSvc” account to have access to that particular database.  Hopefully, you’ve got that account’s password jotted down somewhere in the deep, dark recesses of the documentation black hole.  The message export process took a little longer, mainly because there are some users on my system that have never deleted a voicemail.  In all, I exported 168 MB of WAV files into a backup folder, along with a database of the account configuration.

Now, to import into Unity Connection.  When you first fire up the Import program, you’ll be asked to pick the backup you wish to restore from.  You then have to navigate though a 68-step wizard.  Don’t worry, it’s not as bad as it sounds.  Many of the steps are verification of the Unity Connection configuration.  And there are many steps that get skipped depending on what kind of data you are importing.  It took me a couple of steps to get the messages imported due to some configuration issues (click here for those steps).  Once that was accomplished, everything went great.  I was able to mirror the Unity database onto the Unity Connection server.  I setup a separate voicemail profile in CUCM and awaited my cutover.  Just like expected, the actual cutover took about 10 minutes.  Once the call handlers and voicemail greetings were verified everything was done.

I’m now ready to shutdown the old Unity server and remove all the old voice mail profile information.  Once that’s done, Unity and I have a date in the parking lot.  I’m hoping to recreate something like this scene.  Fast forward to 1:09 for the good stuff, and be warned the audio has NSFW words.

In the end, I don’t think any of this would have been possible without the help of COBRAS.  It may not be a ruthless terrorist organization determined to rule the world, but I’m sure it would help them migrate their voicemail server.  And now you know, and as always knowing is half the battle.

Twelve Days of Christmas Networking

Posted on by
1

In the spirit of Christmas, and because my wife has made me listen to the song about 400 times so far this year, I present the Twelve Days of Christmas, Networking Nerd style.  To save you all the trouble of singing the whole song, we’ll just skip to day tweleve.  On the Twelfth Day of Christmas, the Networking Nerd gave to me:

– Twelve character passwords

– Eleven 802.11n Access Points

– Ten Gigabit Ethernet

– Nine 9971 phones

– Eight-port switch blades

– Seven CCIE Tracks

– Six Hours in the CCIE Lab

– Five Magic Digits! (I hope…)

– Four-port FXOs

– Three Packet Pushers

– Two L2MP options

– And One Goal: To get my CCIE!

Special Thanks to JT (@WannabeCCIE) for giving me the idea for this.

Merry Christmas to all the folks out there.  May your holidays be filled with joy and caring.  May your families not drive you insane, and may your Christmas stocking be filled with all the goodies you asked Santa for.

 

What’s in a Title?

Posted on by
3

“A title by any other name would stink as bad.” –Okay, it’s not Shakespeare, but it’s close.

After my little engineering diatribe, I’ve been thinking of new titles to that I can use besides engineer or rock star.  Because rock star makes you sound pretentious.  And I got really tired of waking up at 5 in the morning to feather my hair with a case of Aquanet.  I shy away from terms like “architect” and “champion” because they may sound cool, but they convey absolutely nothing about what I do.  So, I started making a list:

  • Director of Bailing People’s Asses Out of the Fire
  • Chief Google Search Officer (CGSO)
  • Vice President of Explaining Things to People that Don’t Understand Me
  • Executive Chairman of Just Buy What I Tell You and Don’t Ask Questions
  • President of Throwing Salesmen Under the Bus
  • Head of Deciphering TLAs
  • High Priest of Unity/Exchange Voodoo
  • Sergeant-at-Arms of Explaining Why Your Hair-Brained Idea Won’t Work
  • Chief Caffeine Consumer
  • Vice Regent of Solving Executive Problems RIGHT NOW!
  • Owner/Operator of I Told You So, INC

I think I’m going to need to get bigger business cards..

Feel free to leave some of your favorite titles in the comments.  Just make sure they are descriptive about what your job title is.  And for the love of all that’s holy, DON’T put “engineer”.

They Hackin Everybody Out Here

Posted on by
1

I’ve learned a couple of important lessons in my time as an Internet citizen.  First, don’t taunt the Internet Hate Machine known more colloquially known as “Anonymous”.  Secondly, keep your passwords secure and complex and don’t use them for every website.  Should you do #1 and neglect #2, be certain that #1 will bite you in the ass.  As the people at Gawker Media learned this past week.

A group known as Gnosis posted a 500MB torrent containing various data pulled from a variety of Gawker Media websites.  They claimed the hack was due to Gawker’s hubris and their mocking of previous hacks.  There is also evidence to support the idea that some in Gawker may have taken a stance against the actions of Anonymous in their crusade against those that were involved in the Wikileaks debacle in early December.  While the file contains things like chat logs and FTP servers for various sites that probably don’t want them published, there was a singular gem amongst the chaff.  The most critical piece of this file is the dump of the Gawker MySQL database.  Gnosis was able to access the database and pull the table containing the list of user IDs and passwords.  According to the README.TXT contained in the torrent (and reposted across several websites), they decided to stop dumping the database after about 1.3 million users.  Gnosis then turned to using John the Ripper to decrypt the passwords, which were stored in the table in DES-encrypted format.  The good news is that Gawker decided to store the passwords in a non-plaintext format.  The bad news?  DES is limited to using 8-character keys for encryption (Check this out for more information).  That means that only the first eight characters of the passwords were encrypted and stored.  So, if you were diligent and created a super hard password like “passwordc4n7b3|-|4ck3d”, it would only store “password” in encrypted format.  So, armed with a password database, a sophisticated cracking tool, and a weak encryption algorithm, Gnosis set out to see what they could see.

What did they find?  Well, for one, people violated my second rule by making some pretty easy-to-guess passwords.  Like “password”.  No kidding.  It was the second most popular password out of the bunch, with about 2,100 people out of the 300,000 released hashes using it.  What was more popular than that one?  How about “123456”?  More than 3,000 people used that one.  And the third most popular one was “12345678”.  For a full list of the most popular passwords, check out the Wall Street Journal Blog.

Guess what?  Those passwords SUCK!  Yes, they are easy to remember.  Yes, it’s slightly more secure that not having a password.  Guess what?  They’re also quite easy to guess.  Thanks to rainbow tables, it’s not hard to find the DES hash for password.  In fact, just so you know, it’s “uDGdyZA2EBdWk”.  Just search for that string in the database and you’ll know tons of accounts with unsecured passwords.  Because I know that everyone reading this knows how to make a secure password, I won’t patronize you with password policy.  But, just in case my mom ever decides to read this, a proper password includes ALL of these things:

  • At least EIGHT characters (the more, the better)
  • A number
  • A capital letter
  • A symbol
  • Non-obvious (see above for a list of some obvious stuff)

If you password doesn’t meet those guidelines, it’s probably not that secure.  The longer and more complex the password, the more likely it is to stand up to a dictionary attack or brute force attempt.  However, even if you have a nice, complicated password, reuse of it all over the place can still get you in trouble, as the Gawker people found out on Monday.

Once the Gnosis people got finished having their way with the the Gawker MySQL database, they took their hack to the next level.  They thought to themselves, “I wonder if these people use the same password everywhere?”  So, armed with a list of e-mail addresses and usernames and passwords, they started checking around.  Getting into GMail and Yahoo mail accounts.  Logging into Twitter and Facebook.  Causing general chaos.  Like Twitter accounts randomly tweeting about acai berry products.  The first thought was a new URL-exploiting worm.  Then they realization that a lot of people that were singing the praises of the lowly acai berry were victims of a hijack attack from people that had downloaded the torrent from the Gnosis hack.  Because these users had utilized the same password across multiple accounts, a security breech in one had exposed all of them.

In my opinion, Gawker’s response to the hack wasn’t quite as effective as it could have been.  They posted banners on all their websites advising users to change their passwords.  Except they had taken down the database for some time to patch the holes in it.  Which left their password reset mechanism offline.  What should have happened was an immediate, blanket password reset of EVERY account in the Gawker database.  Gawker already had their e-mail addresses, which would be used to mail the password after a manual reset.  It should be a simple matter to reset the password automatically and send off the new temporary password to the account in the database.  Instead, the users were forced to take the steps themselves or risk further exposure.  A little forethought and perhaps some heavy-handed security admin 101 might have gone a long way to restoring user faith in Gawker.

What we have here is a case of the perfect storm of an older system riddled with insecure passwords that was compromised by a determined foe and then exploited far beyond what anyone except the most pessimistic security expert could have imagined.  Hacks of this magnitude are becoming more and more common, and as we spend more and more time online the information exposure becomes worse each time.  It is quickly reaching the point where it will be necessary to start compartmentalizing our lives in order to keep ourselves secure.  Many people I know have instituted something like this already.  Sites like Facebook and LinkedIn get one type of password.  E-mail and banking sites get a totally different password that is more secure.  For IT professionals, keeping track of multiple passwords isn’t that difficult, especially with password management tools such as 1Password to help us keep our lives straight.  But, to be fair, IT professionals aren’t the true targets of these kinds of hacks.

IT professionals and technology-savvy people are hard targets.  We rotate passwords.  We make secure logins.  We’re always conscious of what information is being stored and shared.  We make lousy hack targets.  But, people like my mom that use the Internet for Facebook and e-mail and shopping are prime targets.  They make accounts on websites like the ones run by Gawker to make a comment on a story.  They use the same password that they use for their Yahoo Mail account and Facebook.  And when something like this comes along and upsets everyone’s apple cart, those people are the ones that suffer.  They aren’t walled off and sure of what information may have leaked.  And they aren’t sure of what passwords to change or when to do it.  And so they might find themselves on the news talking about getting hacked and all the doom and dismay that it has caused.  And who knows?  Maybe someone will autotune my mom into an Internet meme.  Let’s hope not.  Because if there’s anything worse in this world than password database leaks or FBI backdoors into IPSec, it’s listening to my mom sing, autotuned or not.