The Recertification Treadmill

Posted on by
2

I like tests.  Probably a lot more than I should.  Oh, it wasn’t always like this.  I dreaded test days in college.  Cramming chapters worth of information into my brain so that it could just be regurgitated later and forgotten shortly after than.  In fact, I can distinctly remember studying the OSI model for one of my IT infrastructure classes and thinking, “I only need to remember this for the exam.  After that, I’ll never see it again.”  Of course, that same OSI model is now permanently tattooed on the insides of my eyelids.

Then I entered the Real World.  I found out about certification tests and all they entail.  You mean I can take one test proving my mastery of a subject and you guys send me a certificate and a little wallet card?  Sign me up!  It also helped that my employer is a partner with multiple vendors and needed me to take as many tests as I could to keep their partner status up-to-date.  So I set off on my odyssey of test taking.  I’ve got certifications from Novell, Microsoft, CompTIA, Cisco, HP, (ISC)2, and many more.  I’ve taken enough tests that the test administrator at my local testing center recognizes me in the street.  I know more about the ins and outs of testing procedure than most people should.  And, I’ve been handsomely rewarded for my test taking prowess.  And, for the most part, I’ve enjoyed every second of my learning.  Except for recert day.

Yes, every once in a while one of the vendors sends me a note that says I’m due for renewal.  My professional title is now in jeopardy if I don’t study some new information and go see my local Pearson/Prometric guru.  So I start pouring over material in an effort to not need new business cards.  I cram all that new information in my stuffed head and run out to take the test again.  And I pass.  And for a while, I’m a golden boy again.  Until recert day comes up again.

Some vendors tell  you that you can keep your certification for ever and ever.  Like my MCSE.  Of course, I’m not technically “current” with that one, especially now that the new title is MCITP (or something like that).  So, while I’m a whiz when it comes to Windows 2000, I’m not really authorized on the new hotness of Server 2008.  Oh well.  Other vendors, like Cisco, keep the same certification title, but they change the tests around from time to time.  Like my CCVP.  I originally certified on CUCM 4.1.  Back when there was a separate test for those gatekeeper thingies.  And then Cisco went and released a new CCVP track about CUCM 6.x.  I didn’t have to recertify because my CCVP was still good.  But now, they have eliminated the CCVP and changed the voice certification track to the CCNP: Voice.  You can still take the CCVP tests and get grandfathered in before the change to the new CUCM 8.x material if you want.  And that’s what I found myself doing about 2 months ago.  I figured since I worked with voice everyday it shouldn’t be too rough to just jump in and take the tests.  My reasoning was that the partner requirements for Advanced Unified Communications would change after the CCVP –> CCNP: Voice move, so I wanted to get out in front of this change before I was forced to.  I managed to stumble through the troubleshooting test and both CallManager tests in fairly short order.  As I brushed up on my CVOICE basics, I remembered that a previous visit to the Certification Tracker showed that I hadn’t taken the QoS exam, even though I distinctly remembered the pain and agony of that one.  I wrote in to Cisco Cert Support, hoping that I didn’t have to go through it all over again.  While I kept studying for my CVOICE test, I got the response.  It seems that those tests expire after 3 years, and I would need to retake it again for it to be valid.  However, according to Cisco, I was already a CCNP: Voice, so I wouldn’t need to retake it.  Huh?  When did that happen?

Cisco’s recertification policy for professional level exams says that taking any professional test with a ‘642’ prefix will recertify your CCxP.  Little did I know at the time that my first test, Troubleshooting Unified Communications, had recertified my CCVP and triggered the upgrade to a CCNP:Voice.  So, the CUCM tests were for naught.  The CVOICE test did give me a CCNA: Voice tag, so I’ve got that going for me now.  The Cisco recert cycle is nothing new to me.  I’ve been taking the CCIE written exam every year because it’s the only way to keep my specialist designations current.  In order to keep my employer in the good partner graces, I have to keep remembering OSPF and MPLS trivia and take the CCIE written at least every two years.  It’s the only way for me to keep my certifications current without devoting all my time to studying and taking tests instead of doing the job I was hired for.  I was confused in this particular instance with the CCNP: Voice because the certification website never said anything about there being an upgrade path from my 4.2 CCVP to the 8.x CCNP: Voice.  I’m happy nonetheless, but I started thinking about the whole recertification process and why it bothers me somewhat.

I can take any 642 level Cisco exam and recertify all my CCxA and CCxP titles.  I can take the CCIE written and do the same, including my specialist tags.  VMware makes me take a new test and sit through 5 days of training to get a VCP4.  Microsoft wants me to take a whole new set of tests to become a new MCSE/MCITP.  Novell just keeps certifying me on Linux stuff even though I haven’t taken Novell test in years.  And we won’t talk about HP.  Ethan has a great post about recerting his CCIE that hits on a lot of good points.  Normally, we have to either shut down our productivity for a few weeks to get into the recertification groove, or try and find time outside of work to study.  Either way, it seems like a colossal waste of time. It’s almost like being elected to the House of Representatives.  You need to start campaigning for re-election right after you’ve been elected.  It’s just annoying that I have to take time out of my schedule to relearn things I’m already doing.  Is there any way to fix this?

Find a lawyer.  Any lawyer.  If you’re having trouble, check behind the nearest ambulance.  Now, ask them how many times they’ve retaken the bar exam.  Odds are good they’ll stare at you and tell you that you’ve lost your mind.  Lawyers don’t have to resit the bar exam every time they need to renew their fancy degree.  They are allowed to use Continuing Professional Education credits.  All they have to do is take a class or attend a conference and they can count that learning toward recertifying their degree and certification requirements.  IT people are the same.  We spend a lot of our time watching webcasts and going to trade shows.  I go to Cisco Live Networkers almost every year.  When I’m there, I take the opportunity to learn about technologies I don’t encounter in my every day job, like TRiLL or FabricPath.  I’m doing an awful lot to keep current with trends and technology in the industry, and it feels like it’s all for my own edification.  It doesn’t really count toward anything.  Except in one case – my CISSP.  Because (ISC)2 uses a CPEs too.

The vendor-neutral certification bodies have it right, in my opinion.  (ISC)2, BICSI, and CWNP all have a CPE policy.  They say that you can go to conferences or read books and count that learning toward your certification.  They want you to prove that you’re staying current, and in return they’ll make sure you are current when it comes to certifications.  Sure, in the case of the CISSP, most of the learning needs to be focused on security, but that’s how it should be.  I can count some amount of general education credits toward my CISSP, but the bulk of the education needs to be focused on the subject matter of the certification.  I think something like this would be a great addition to Cisco’s arsenal.  Give your certified professionals a chance to apply the learning they do every day toward recertification.  You’d sell more Cisco Press books if I knew I could read one and count 5 points toward my CCSP.  There’d be even more attendees at Networkers if it counted for 40 CPEs every year.  But, there also need to be some restrictions.

Some vendors don’t like the idea that one test can recertify all your titles.  Juniper doesn’t.  So make sure that the education credits only count toward a specific area of knowledge.  The Migrating CUCM class from Brandon Ta that I go to every year could count toward my CCVP, but not my CCSP.  My TRiLL webcasts could count for points to recertify my CCIE R&S or SP, but not the CCIE Wireless.  If you marry the education to a specific certification, you’ll see much higher attendance for those kinds of things.  For people like us that spend time writing about things on the Interwebs, authoring articles for places like Network World or Information Week could count as well, since you are disseminating the knowledge you’ve obtained to the masses.  Even teaching could count toward recertifying.

This idea is not without issue, though.  The first argument is that allowing certified individuals to use CPEs might cause problems with the cottage industry that has sprung up around teaching these subjects to people.  Ask yourself, How many people would go to VMware classroom learning if it wasn’t required to obtain the VCP?  I’m sure the answer would be “A whole lot less.”  It’s no secret that Cisco and HP and Microsoft make a lot of money offering classes to people in order to get the certified on technology.  Companies can specialize in just teaching certification coursework and turn a tidy profit.  And these same companies might not be too keen on the idea of a revenue stream drying up because Cisco or Novell decided to be noble and not require everyone to take a new test every 2 years.

Another consequence, though one for the better, would be the contraction of the “braindump” market.  A lot of people talk about the braindump market catering to those who want a fast track to the CCNA or other entry-level cert.  I’m of the opinion that a larger portion of the dumping population consists of already-certified individuals that have neither the time nor the energy to study for a recertification exam.  These people are facing a deadline of needing to stay current with whatever alphabet soup comes after their name, except now that they have a steady job they don’t have the time to devote to studying all night to pass.  Faced with the option of letting their certification expire, or paying money to someone for the answers to the test, they swallow their pride and take the easy way out.  In their mind, no harm is done because they were already a CCxA in the first place.  They know the material, they just don’t have time to remember what the “vendor answer” is on the test.  Now, give these same people the opportunity to apply a webcast or vendor presentation that they’d sit through anyway to that CCxA.  I bet that more than half the dumping sites would go away within a year.  When the market starts drying up, it’s time to move on.

I really hope that the vendors out there take the time in 2011 to reassess their recertification strategies.  Giving certified professionals more options when it comes to proving they know their material can only build goodwill in the future.  Because the current method feels way too much like a treadmill right now.  I keep running in place as fast as I can just to stay where I’m at.  I think things need to change in order to make the education and learning that I do have a tangible impact on my certification progress.  Because sooner or later I’m not going to be able to keep up with the recertification treadmill.  And we all know what the result is when that happens…

COBRAS!

Posted on by
Reply

If you are a voice networking professional, and you are tasked with working on any Cisco voice mail product, do yourself a favor and go download the Cisco (Unified) Backup and Restore Application Suite (COBRAS).  You’ll thank me later.

If you’ve ever tried to upgrade Unity, you know the pain that comes from the good old Disaster Recovery Tool (DiRT).  Cisco’s best practice for upgrading Unity involves using DiRT to backup your existing database, installing a new server, installing the old version of Unity and performing a restore, then attempting to upgrade the new server to the new version of Unity.  Any one of those steps if fraught with danger and terror.  DiRT was never really designed to do upgrades.  It was only ever meant to restore your Unity configuration and data in the event of a meteor strike or alien invasion.  Other than those two corner cases, it pretty much sucks.  In case you couldn’t tell, I’m not a fan of DiRT.  It’s like magnetic tape.  It serves one purpose that it’s good for, but if you start getting creative you are asking for trouble.

When Cisco started pushing Unity Connection as a viable alternative to Unity, the need arose to find a way to get the data out of Unity and import it into Unity Connection.  This is not a job for DiRT.  The worst-case-scenario is that you need to pop up two web browsers and input the information from one system into the other.  Not a fun job if you have even 50 mailboxes.  A nightmare if you have a few thousand.  And, you can’t move user passwords and PINs.  A better solution needed to be found.  And, thanks to some enterprising TAC engineers, we have COBRAS.

COBRAS started its life as a very unsupported tool on the Cisco Unity Tools website.  Anyone that has worked on Unity for more than five minutes has been to this website.  At various points in its life, it has been referred to as AnswerMonkey.net and LindborgLabs.com.  My best guess is that it started as a repository run by some TAC engineers for the purpose of giving the long-suffering Unity support people a place to download tools and scripts to help get Unity working as properly as you can for a program that requires you to speak in tongues to fix it.  About three years ago, I had the good fortune to take a class at Cisco Live Networkers that dealt with the problem of migrating CallManager and Unity to newer versions.  I specifically took the class because I was about to face an unpleasant customer upgrade from CUCM 4.1 to 6.1.  At the same time, the customer wanted to move from Unity 4 to 5.  All of the official documentation I read said that the DiRT migration was the best way to move to new hardware.  Luckily, the class from Brandon Ta at Cisco Networkers pointed me in the direction of COBRAS.  Of course, when I went to download it from the Unity Tools website, the warning messages and dire predictions told me why I hadn’t seen it before – it wasn’t quite supported.  As in, it wasn’t supported at all.   Still, faced with the choice of something that I knew I wouldn’t like or the idea of trying something new that had little support, I bit the bullet and went with the new idea.  And boy did I like it.  COBRAS pulled all the Unity information from the old system in short order.  When I brought the new system online, a quick import into Unity 5 got the voicemails flowing again in no time.  Rather than spend hours waiting for the inevitable issues with DiRT restores, I could instead concentrate of cursing the Data Migration Assistant.  But that’s another story entirely.

Flash forward to this past month.  We’d been running Unity at our office for several years, and my users had become very dependent on unified messaging.  When the Windows admins decided to upgrade to Exchange 2007, they forgot to warn me about when they had planned on doing this.  So, when I walked the next morning, the voicemail integration was offline.  It took a couple of hours before I was able to install the correct engineering special and repair all the Unity permissions with the scripts from the Grand Unity Grimoire.  I’ve known that an upgrade to Exchange 2010 is in the works at some point.  I’ve also grown tired of the difficulties that Unity presents.  I can only administer it from Internet Explorer.  The need to keep Active Directory healthy just for the sake of Unity was annoying.  The need to know intimate details about Exchange 2003 made me cringe.  Even my conversations with product managers from Cisco didn’t leave me all that excited about Unity.  But, I couldn’t move to Unity Connection just yet.  Because of the unified messaging issue.  My coworkers couldn’t fathom the idea of NOT getting voicemails in one inbox.  IMAP wasn’t going to cut it.  So I bided my time and plotted the demise of Unity.  When Cisco formally announced the feature set for Unity Connection 8.5, I jumped for joy.  Unity Connection 8.5 contains a unified messaging agent that allows you to synchronize your Unity Connection mailstore with Exchange 2003, 2007, and 2010.  My users would receive the same benefits as they had now with Unity, and I would get a unified management platform on the back end that was no longer soulbound with Exchange.

Once I got Unity Connection installed on a spare server, I needed to export 50 mailboxes worth of data from Unity.  I checked Cisco Unity Tools and found they had updated COBRAS to support the latest versions of Unity Connection, and in fact COBRAS was now supported by TAC!  It felt like this angel finally got its wings!  I downloaded the Unity Export and Connection Import programs and installed them.  The Unity Export program needs to be installed on Unity.  In fact, it’s only support on Windows Server platforms.  The Connection Import program can be installed on any Windows system, but you also need to install the IBM Informix Database drivers to allow communication with the Unity Connection database directly.  I resorted to installing the program and database drivers in Windows XP virtual machine, as my 64-bit Windows 7 installation has already show an intolerance for drivers in general, which my USB-to-serial adapter will attest to.  Once I installed the programs, I exported all the configuration data from Unity in one shot.  It took all of 20 minutes.  I was shocked.  I had fully expected this whole export to eat up my entire afternoon.  When I went to export the voicemail messages from the database, I found that I needed to be logged in as the “UnityMsgStoreSvc” account to have access to that particular database.  Hopefully, you’ve got that account’s password jotted down somewhere in the deep, dark recesses of the documentation black hole.  The message export process took a little longer, mainly because there are some users on my system that have never deleted a voicemail.  In all, I exported 168 MB of WAV files into a backup folder, along with a database of the account configuration.

Now, to import into Unity Connection.  When you first fire up the Import program, you’ll be asked to pick the backup you wish to restore from.  You then have to navigate though a 68-step wizard.  Don’t worry, it’s not as bad as it sounds.  Many of the steps are verification of the Unity Connection configuration.  And there are many steps that get skipped depending on what kind of data you are importing.  It took me a couple of steps to get the messages imported due to some configuration issues (click here for those steps).  Once that was accomplished, everything went great.  I was able to mirror the Unity database onto the Unity Connection server.  I setup a separate voicemail profile in CUCM and awaited my cutover.  Just like expected, the actual cutover took about 10 minutes.  Once the call handlers and voicemail greetings were verified everything was done.

I’m now ready to shutdown the old Unity server and remove all the old voice mail profile information.  Once that’s done, Unity and I have a date in the parking lot.  I’m hoping to recreate something like this scene.  Fast forward to 1:09 for the good stuff, and be warned the audio has NSFW words.

In the end, I don’t think any of this would have been possible without the help of COBRAS.  It may not be a ruthless terrorist organization determined to rule the world, but I’m sure it would help them migrate their voicemail server.  And now you know, and as always knowing is half the battle.

Twelve Days of Christmas Networking

Posted on by
1

In the spirit of Christmas, and because my wife has made me listen to the song about 400 times so far this year, I present the Twelve Days of Christmas, Networking Nerd style.  To save you all the trouble of singing the whole song, we’ll just skip to day tweleve.  On the Twelfth Day of Christmas, the Networking Nerd gave to me:

– Twelve character passwords

– Eleven 802.11n Access Points

– Ten Gigabit Ethernet

– Nine 9971 phones

– Eight-port switch blades

– Seven CCIE Tracks

– Six Hours in the CCIE Lab

– Five Magic Digits! (I hope…)

– Four-port FXOs

– Three Packet Pushers

– Two L2MP options

– And One Goal: To get my CCIE!

Special Thanks to JT (@WannabeCCIE) for giving me the idea for this.

Merry Christmas to all the folks out there.  May your holidays be filled with joy and caring.  May your families not drive you insane, and may your Christmas stocking be filled with all the goodies you asked Santa for.

 

What’s in a Title?

Posted on by
3

“A title by any other name would stink as bad.” –Okay, it’s not Shakespeare, but it’s close.

After my little engineering diatribe, I’ve been thinking of new titles to that I can use besides engineer or rock star.  Because rock star makes you sound pretentious.  And I got really tired of waking up at 5 in the morning to feather my hair with a case of Aquanet.  I shy away from terms like “architect” and “champion” because they may sound cool, but they convey absolutely nothing about what I do.  So, I started making a list:

  • Director of Bailing People’s Asses Out of the Fire
  • Chief Google Search Officer (CGSO)
  • Vice President of Explaining Things to People that Don’t Understand Me
  • Executive Chairman of Just Buy What I Tell You and Don’t Ask Questions
  • President of Throwing Salesmen Under the Bus
  • Head of Deciphering TLAs
  • High Priest of Unity/Exchange Voodoo
  • Sergeant-at-Arms of Explaining Why Your Hair-Brained Idea Won’t Work
  • Chief Caffeine Consumer
  • Vice Regent of Solving Executive Problems RIGHT NOW!
  • Owner/Operator of I Told You So, INC

I think I’m going to need to get bigger business cards..

Feel free to leave some of your favorite titles in the comments.  Just make sure they are descriptive about what your job title is.  And for the love of all that’s holy, DON’T put “engineer”.

They Hackin Everybody Out Here

Posted on by
1

I’ve learned a couple of important lessons in my time as an Internet citizen.  First, don’t taunt the Internet Hate Machine known more colloquially known as “Anonymous”.  Secondly, keep your passwords secure and complex and don’t use them for every website.  Should you do #1 and neglect #2, be certain that #1 will bite you in the ass.  As the people at Gawker Media learned this past week.

A group known as Gnosis posted a 500MB torrent containing various data pulled from a variety of Gawker Media websites.  They claimed the hack was due to Gawker’s hubris and their mocking of previous hacks.  There is also evidence to support the idea that some in Gawker may have taken a stance against the actions of Anonymous in their crusade against those that were involved in the Wikileaks debacle in early December.  While the file contains things like chat logs and FTP servers for various sites that probably don’t want them published, there was a singular gem amongst the chaff.  The most critical piece of this file is the dump of the Gawker MySQL database.  Gnosis was able to access the database and pull the table containing the list of user IDs and passwords.  According to the README.TXT contained in the torrent (and reposted across several websites), they decided to stop dumping the database after about 1.3 million users.  Gnosis then turned to using John the Ripper to decrypt the passwords, which were stored in the table in DES-encrypted format.  The good news is that Gawker decided to store the passwords in a non-plaintext format.  The bad news?  DES is limited to using 8-character keys for encryption (Check this out for more information).  That means that only the first eight characters of the passwords were encrypted and stored.  So, if you were diligent and created a super hard password like “passwordc4n7b3|-|4ck3d”, it would only store “password” in encrypted format.  So, armed with a password database, a sophisticated cracking tool, and a weak encryption algorithm, Gnosis set out to see what they could see.

What did they find?  Well, for one, people violated my second rule by making some pretty easy-to-guess passwords.  Like “password”.  No kidding.  It was the second most popular password out of the bunch, with about 2,100 people out of the 300,000 released hashes using it.  What was more popular than that one?  How about “123456”?  More than 3,000 people used that one.  And the third most popular one was “12345678”.  For a full list of the most popular passwords, check out the Wall Street Journal Blog.

Guess what?  Those passwords SUCK!  Yes, they are easy to remember.  Yes, it’s slightly more secure that not having a password.  Guess what?  They’re also quite easy to guess.  Thanks to rainbow tables, it’s not hard to find the DES hash for password.  In fact, just so you know, it’s “uDGdyZA2EBdWk”.  Just search for that string in the database and you’ll know tons of accounts with unsecured passwords.  Because I know that everyone reading this knows how to make a secure password, I won’t patronize you with password policy.  But, just in case my mom ever decides to read this, a proper password includes ALL of these things:

  • At least EIGHT characters (the more, the better)
  • A number
  • A capital letter
  • A symbol
  • Non-obvious (see above for a list of some obvious stuff)

If you password doesn’t meet those guidelines, it’s probably not that secure.  The longer and more complex the password, the more likely it is to stand up to a dictionary attack or brute force attempt.  However, even if you have a nice, complicated password, reuse of it all over the place can still get you in trouble, as the Gawker people found out on Monday.

Once the Gnosis people got finished having their way with the the Gawker MySQL database, they took their hack to the next level.  They thought to themselves, “I wonder if these people use the same password everywhere?”  So, armed with a list of e-mail addresses and usernames and passwords, they started checking around.  Getting into GMail and Yahoo mail accounts.  Logging into Twitter and Facebook.  Causing general chaos.  Like Twitter accounts randomly tweeting about acai berry products.  The first thought was a new URL-exploiting worm.  Then they realization that a lot of people that were singing the praises of the lowly acai berry were victims of a hijack attack from people that had downloaded the torrent from the Gnosis hack.  Because these users had utilized the same password across multiple accounts, a security breech in one had exposed all of them.

In my opinion, Gawker’s response to the hack wasn’t quite as effective as it could have been.  They posted banners on all their websites advising users to change their passwords.  Except they had taken down the database for some time to patch the holes in it.  Which left their password reset mechanism offline.  What should have happened was an immediate, blanket password reset of EVERY account in the Gawker database.  Gawker already had their e-mail addresses, which would be used to mail the password after a manual reset.  It should be a simple matter to reset the password automatically and send off the new temporary password to the account in the database.  Instead, the users were forced to take the steps themselves or risk further exposure.  A little forethought and perhaps some heavy-handed security admin 101 might have gone a long way to restoring user faith in Gawker.

What we have here is a case of the perfect storm of an older system riddled with insecure passwords that was compromised by a determined foe and then exploited far beyond what anyone except the most pessimistic security expert could have imagined.  Hacks of this magnitude are becoming more and more common, and as we spend more and more time online the information exposure becomes worse each time.  It is quickly reaching the point where it will be necessary to start compartmentalizing our lives in order to keep ourselves secure.  Many people I know have instituted something like this already.  Sites like Facebook and LinkedIn get one type of password.  E-mail and banking sites get a totally different password that is more secure.  For IT professionals, keeping track of multiple passwords isn’t that difficult, especially with password management tools such as 1Password to help us keep our lives straight.  But, to be fair, IT professionals aren’t the true targets of these kinds of hacks.

IT professionals and technology-savvy people are hard targets.  We rotate passwords.  We make secure logins.  We’re always conscious of what information is being stored and shared.  We make lousy hack targets.  But, people like my mom that use the Internet for Facebook and e-mail and shopping are prime targets.  They make accounts on websites like the ones run by Gawker to make a comment on a story.  They use the same password that they use for their Yahoo Mail account and Facebook.  And when something like this comes along and upsets everyone’s apple cart, those people are the ones that suffer.  They aren’t walled off and sure of what information may have leaked.  And they aren’t sure of what passwords to change or when to do it.  And so they might find themselves on the news talking about getting hacked and all the doom and dismay that it has caused.  And who knows?  Maybe someone will autotune my mom into an Internet meme.  Let’s hope not.  Because if there’s anything worse in this world than password database leaks or FBI backdoors into IPSec, it’s listening to my mom sing, autotuned or not.

Stuxnet: Be Afraid

Posted on by
6

“Doesn’t that bother any of you? Because it scares the living piss outta me!” – Lloyd Bridge as Admiral Tug Benson

That pretty much sums up my feelings about the Stuxnet worm the more and more I read about it.  It seems like every week brings more and more dastardly information about this worm and its consequences for cyber warfare in general for the foreseeable future.  First, a refresher course for those that might not be totally familiar with this little gem.

Anatomy of a Scary Virus

A Belarusian security firm got it’s hands on a sample of a new worm in mid-June of 2010.  It was a Windows-based attack that seemed to be quite virulent from the very beginning.  More disturbing, however, was the complexity that lay just beneath the surface upon further examination.  Stuxnet targetted 4 separate zero-day exploits in Windows.  In the security arena, this is the equivalent of showing your hand too early in a poker game.  Zero-day exploits have great value on the black market for virus writers, so they tend to be hoarded and exploited only when a significant advantage can be had.  For a virus to use four of them at once meant that it was serious about infecting things.  Secondly, it installed a rootkit on the target system.  While this isn’t necessarily new in and of itself, the way it succeeded was brilliant.  The writers of the virus hijacked to signed security certificates from trusted manufacturers JMicron and Realtek.  This meant that the kernel mode drivers necessary for rootkit operation could be installed without so much as a blip of a warning.  Also disturbing was the method in which the virus was constructed, a mish-mash of C and C++ code.  This is quite odd for a trade that typically uses simple coding techniques.

After digging into the payload and operation of the virus, the malicious intent cranked up two or three more notches.  The virus used a data cable connect between the PC and a Siemens Programmable Logic Controller (PLC) to hop into the PLC where it really started its nefarious work.  Firstly, a rootkit was installed to hide the infection.  Then, using the PLC it started messing with variable frequency drives that were slaved to the unit.  Specifically, it was looking for drives that spin between frequencies of 807 Hz and 1210 Hz.  Why so specific, you ask?  Because drives that run at those frequencies just happen to be of the same kind that are used in centrifuges, which are critical to process needed to enrich uranium in nuclear power plants.  Once it found the target, it didn’t make itself obvious by disabling the drive.  Instead, it varied the rotational speed of the unit, ramping it up to 1400 Hz then back down to 2 Hz then back up again.  To the outside observer, it would just look like the device was going haywire or having mechanical difficulties.  At worst, you might think to pull the drive out and replace it with another unit.  Of course, as soon as that unit was connected to the PLC, it would be infected by the Stuxnet worm and the whole process would begin all over again.

A New Chapter in Warfare

Once the security firm started tracing the command and control centers for the virus, the trail started going cold as servers were shutdown and erased from the face of the Internet.  Usually, those kinds of disappearing acts are perpetrated by the kind of three-letter agencies that don’t like to make the headlines.  And so it was that a large number of security researchers started speculating about the nature and purpose of Stuxnet.  Symantec believes that a well-coordinated team of 5 to 10 individuals spent several months writing the virus.  As well, the largest number of infected systems appears to be located in Iran.  Based on the specific target of the virus (industrial equipment known to be purchased by Iran), it seems quite plausible to assume that someone or something wanted to make sure that the equipment didn’t function correctly.  But, rather than take it out completely, the idea behind Stuxnet was to mask the damage done and make it look like mechanical failure.  Indeed, since it was looking for such specific target criteria, it might have laid dormant for months before unmasking itself.  The speculation currently is that the worm was designed to do one thing with brutal efficiency – cripple the Iranian nuclear program.  Not by airstrikes or conventional means, but with cyber warfare.

When you think back on many of the malware programs that have sprung up and been quite irritating over the last few years, realize that the authors wanted to make a statement with them.  Whether it was the theft of personal information or the hijacking of your PC for less-than-honorable purposes, each author left a stamp or calling card.  These are the kinds of people that do things for fame and fortune.  They want the exposure.  If someone finds out who wrote Code Red or Nimda, all the better for them.  Exposure gives credibility and prestige in that community.  Even something like the SQL Slammer worm was an attempt to exploit a known vulnerability, perhaps for use by someone at a later date.  Only the ham-handedness of the coding caused it to race out of control and be fought back so quickly.  And so security professionals see these viruses and malware infections and combat them as best we can.  But we only catch them because we can see the tell-tale signs.

Stuxnet appears to have been coded by a person or persons who don’t ever intend to be known.  Their job succeeds when no one knows they did anything.  These kind of people don’t leave marks or traces of any kind when they are done.  They are professional.  The pick a target and pursue it relentlessly until it is neutralized.  And when all is said and done, no one would think twice about the cause of the misfortune to be man-made or inflicted.

Imagine if this had happened in America?  Infected USB drives are scattered around a parking lot at a facility that services nuclear power plants.  Or mailed to key individuals that have access to sensitive areas.  Imagine the chaos that could ensue if the payload hadn’t been designed to subtly cripple, but instead was crafted to cause mayhem and disorder?  Imagine what might happen if it were to occur on the scale of something that we can’t live without, like the GPS constellation?  The idea that agencies and organizations that have made careers out of the kind of malicious and nasty tricks that mark intelligence and spying are now beginning to focus on cyber warfare is frightning.  Think about what could happen if the most prolific and successful malware creators were hired for a job that would pay a fortune, provided the attack was successful and left zero trace.  Would it be worth several million dollars if a country could cripple the military command and control functions of their enemy with a moment’s notice?  What would happen if an invading army had no fear about its ability to render any and all resistance moot with the press of a button from some previous malware infection that went totally undetected until it was too late?

Granted, this all pie-in-the-sky rambling, but the directions that these types of programs can be taken in boggles even the most die-hard security researchers.  Think about how many information system breaches we’ve seen.  Now think about what would happen if it was targeted to, say the Department of Defense.  Or the Social Security Administration? And no amount of money or threat of prosecution could deter the people doing it.  State-sponsored terrorism is bad enough today.  What happens when state-sponsored cyber terrorism becomes more prevalent?  And before you answer that question too quickly, look at what happened with GMail just a few months ago.  And realize that many in the security realm are starting to believe that those attacks were state-sponsored.

For those of you science fiction fans out there, my thought exercises may sound eerily similar to the reimagined Battlestar Galactica mini-series, where the Cylons were able to cripple the entire military effectiveness of the Colonials with a few well-placed programs.  We all laughed at it and said that it made for great story telling, but it was still just fiction.  Well, with the rise of Stuxnet and inevitably more programs like it, we can only hope that the escalation of cyber warfare doesn’t lead us to some kind of horrible conclusion.  Because it’s something like that which makes me truly afraid.

Flip MinoPRO – My Review

Posted on by
4

A while back, I talked about using a Cisco Flip video camera to document my walkthroughs and act as a visual note taking tool.  Little did I know that just after that post, I was going to get a chance to look at the new hotness to my old-and-busted Flip.  I present…the Flip MinoPRO:

Pretty, ain’t it?

The Cisco Flip MinoPRO is the “prosumer” offering that Cisco is targeting for the business/enterprise customer.  If you’ve been living under a rock for the last year or so, John Chambers is driving video like a herd of cattle in City Slickers.  From the Flip acquisition to Tandberg to the new Cisco Cius, Chambers believes that video is the future.  By putting a video camera in the hands of your employees, you can get them to capture things and share them with many other people.  Want to save the New Employee Orientation for posterity?  How about a lesson on BPDUGuard?  Chalk talk with the virtualization architect about Vmotion?  Chambers wants you to record all of this and let everyone see what you’ve done.

So how does the Flip MinoPRO fall into this?  One of the things that I noticed about the Flip Mino that I’ve been carrying is that it felt a little…well, flimsy.  It was made out of plastic and weighed next to nothing.  While this a great thing for the consumer space, it always seemed to be a little disposable to me.  In fact, my Mino has already started to show a few cracks around the edges.  I’m sure that most of it is due to me being a little rougher on it that the average family of 4, but if this is a device that is going to be used day in and day out by your employees, it should probably be a little more rugged than your average CD jewel case.  With the Flip MinoPRO, it is definitely sturdier.  When I first pulled it out of the box, I noticed the housing was made out of metal, not plastic.  It has a great heft while staying light.  The PRO really feels nice.  I’m sure if I threw it at someone, I could cause a bruise.

The unit itself is fully capable of recording in 720p HD (1280 x 720).  This is wonderful for getting details on things in racks and noticing all the little nuances you might miss on a cursory glance.  In addition, the audio seems to be somewhat better, although if you’re expecting movie camera-quality you’ll be disappointed.  The unit can pick up lots of ambient noise, but it’s still a condenser microphone, so loud background noise will overwhelm it.  Another nice upgrade from my Mino is the storage size.  The old camera had 1GB of storage, which at it’s standard definition was good for about one hour of recording time.  The new camera has a staggering 16GB of storage space.  Even with the increased HD recording, that’s 4 hours of recording space available to me now.  Whereas before I had to be judicious about removing video after I was finished with it, I now can leave it on the camera until I’m ready to use it.  I don’t have to be concerned if I can’t capture a whole event or lecture, as I’m now able to get as much as I can.  In addition, the PRO can also function as a USB memory stick.  While it’s a bit more unwieldy than the USB drive I carry in my pocket, it’s quite handy if I need to copy something back and forth and it’s already plugged in.  In addition to the HD recording, the output of the camera got a matching upgrade.  Gone are the three-pronged composite video/audio outputs of yore.  In their place is a shiny new HDMI output!  This means that when you’ve finished recording your video in it’s pixel-orgy quality, you can plug your camera into any HDTV with a simple cable and play away.  It also makes it much easier to find a cable instead of always forgetting the proprietary one that came with the old camera.

The software for the camera stays the same as the consumer version (FlipShare), but as businesses have different requirements for their video, Cisco too has the answer.  Cisco FocalPoint is a video portal run by Cisco that allows you to upload video directly from either the FocalPoint software or from FlipShare.  Acting essentially as a corporate YouTube, you gain the additional options of keeping video secure as it is uploaded to the portal, as well as wiping the camera after the upload.  The hallmark of FocalPoint is the ability to easily search the video that you’ve fed into it, while at the same time securing the content so that only authorized users have access.  That way, your manager’s retreat videos don’t get downloaded and sent to Wikileaks.  FocalPoint is a licensed product, so you’ll need to have a license to create a workspace and attach users.

Now, just like every silver lining has a cloud, so too must this device have a drawback.  It’s not necessarily unique to the Flip MinoPRO, but to HD video in general.  I’ve noticed that my ‘notes’ are starting to take up much more space now that I’ve switched to the new-fangled HiDef stuff.  I’m clocking in at about 1 Meg/second.  That means that any of my longer videos will no longer be able to be disseminated via e-mail.  I’m sure that this is what Cisco had in mind when they created FocalPoint, as the only other method available to me currently is burning the movies to a CD/DVD, or uploading them to a shared internal storage location.  Oh well, if nothing else you can sell a lot more storage space with each new Flip MinoPRO.

If you’d like to get a look at what kind of video the Flip MinoPRO is capable of, take a look at this video of me on a Segway.  Be sure to select 720p to see it in all it’s…um, glory.

ISR G2: ISR Harder

Posted on by
5

There have been a lot of questions recently about the new Integrated Services Router (ISR) G2 line of routers from Cisco.  These new routers are designated by a ‘9’ in the hundreds digit of the model number, e.g. 2901 or 3945.  They are the replacement models for the original line of ISRs, the x800 series.  A little history:

Old and Not-Quite-Busted

The original line of ISRs from Cisco was designed to start incorporating more and more of the integrated network services as defined by Cisco’s “Network as a Platform” idea.  They incorporated things like Packet Voice DSP Modules (PVDM) for voice transcoding and AIM slots for things like Unity Express voice mail modules.  Also, security related modules could be purchased and loaded, and VPN encryption was accelerated on the main board itself through the use of a specialized ASIC.  They also included more RAM and compact flash memory then the previous models to allow for all kinds of fun things like CallManager Express and larger and larger routing tables.  As well, newer services such as MPLS and GET VPN were designed around the idea of using these newer routers with all the additional horsepower to achieve better performance.  And since there introduction, they are quickly becoming some of the most popular routers out in production.  So imagine my surprise when Cisco not only releases a second generation model, but announces the end of sale of the previous generation.

New, um, Warmness

The new line of ISRs, the G2 as Cisco refers to them, are evolutionary upgrades to the product line.  From the specs listed on Cisco’s website, they appear to incorporate newer technology and refreshes of hardware, but nothing spectacularly groundbreaking.  For example:

  • The G2 comes standard with gigabit ethernet ports for the 2900 and 3900.  The 2911 and up includes at least 3 ports, with one being an SFP module for things like fiber, which are becoming increasing more common as a handoff.
  • The amount of RAM has increased to 512MB across the board as the default.  The G2 can also increase to a maximum of 2GB of RAM.
  • There are two compact flash slots on the G2.  One is preloaded with a 256MB flash module, and both can be upgraded to a max of 4GB of flash each.
  • The x900 series includes the new USB console connection, which allows for the use of a simple USB A-to-mini-B connector instead of the ubiquitous blue rollover cable.  Good news for those of us that are getting tired of hauling around USB-to-serial adaptors that don’t work half the time with OSX or Windows 7.
  • Also included are two USB 2.0 ports (an upgrade from the 1.1 ports on the x800s) for things like security token use and file transfers
  • Support for newer HWICs, VWICs, and PVDM3 modules that provide DSPs for voice and video.

As you can see, this is at best an evolutionary upgrade.  Much like refreshing the Dell Vostro or the Lenovo Thinkpad, Cisco has given us better hardware in the box.  It’s now more modern and better suited for increasing connectivity speeds.  But was all this really necessary for a new product line launch?  Or the sunsetting of the old?  Before you answer that question, let’s look at the OTHER piece of the puzzle.  The software.

IOS with a capital “I”

Despite what a Cisco router looks like on the outside, what’s always been important to us is what’s under the hood.  The real guts of the router for networking professionals is the operating system.  A router could look like a pile of circuit boards and plastic packing peanuts as long as it shovels packets and gives us a way to program it.  And so, Cisco’s IOS has moved up to version 15.  Yes, they went from 12.4 to 15.  I can see skipping 13 out of superstitious reasons, but while they vaulted past 14 is beyond me.  Maybe 15 was a nice round number.  At any rate, along with the launch of IOS 15 was a change in the licensing model.  That’s where the real meat of this upgrade lives.

Go to Cisco’s website and check out how many software images are available for download for the 12.4 code train on the 2811.  Go ahead, I’ll stay here and burn a Lady Gaga CD…

Back already?  What, you don’t have a service contract with a 2800?  You can’t see the images to download them?  Oh, alright.  I’ll check for you.

Twenty four.  Yes, that’s right.  While some of them are bundle upgrades, there are still a lot of images out there.  With names like IP Base, SP Services, Advanced IP Services, and Advance Enterprise Services.  What?  What do those mean?  Unless you use the IOS feature navigator, who on earth knows?  Each one of those images contains some functionality that you need.  Need a CME upgrade?  You need SP Services or better.  MPLS?  Your looking for Advanced IP services.  Each feature you need from a service set requires you to download a specific file.  And in some cases, you get more functionality than you know what to do with.  If you download Advanced Enterprise services, you get the whole kitchen sink to deal with.  And that is the crux of Cisco’s problem.

It is entirely possible to purchase a 2811 router with an IP Base IOS image and then upgrade it to an Advanced Enterprise Services image (DISCLAIMER: I do not advise that you do this.  It’s illegal.  And it’ll get you in tons of trouble with Cisco should they find out.  And, you have  your conscience to live with afterward.)  The only limiting factor on the platform is the amount of RAM and flash storage needed.  As well, when you download the proper image looking for CCME, you inadvertently get the MPLS code and a whole host of other things as well.  It gets in the way and doesn’t allow things to run smoothly.

Now, go check out how many images are available for the 2911.  Oh, yeah.  Service contract thing again…

One.  Exactly one IOS image available for the 2911.  No IP Base or SP Services.  It’s labeled a “universal” IOS image.  What exactly does that mean?

The Cisco Code

For those of you that play video games, you might remember one called Quake.  Made by id Software all the way back in good old 1996, it was a first-person shooter.  It was also the subject of an interesting sales tactic by the publisher, GT Interactive.  They used a method called “Test Drive”, which allowed you to purchase the shareware version of the game at a nice low price.  You could play the first few levels of the game and decide if you liked it.  If you did, you called up GT and told them you wanted to buy the whole game.  They e-mailed you a key and told you to type it in.  As soon as you did, you unlocked the whole game on the CD you purchased.  That way, there was no second trip to the store and no additional install time.  GT saved a ton on packaging by only selling one version of the game.  The “full” version you bought for the regular $50 price tag just included the unlock code in the box.

Now, it appears that Cisco is doing something very similar with IOS to prevent OS piracy and lock down feature sets.  When you order an ISR G2 router, you get the basic image with all the basic routing functions.  If you want to do CCME, you need to buy a Unified Communications license.  You want to do VPN?  You have to buy a security license.  MPLS? Advanced data license.  This way, Cisco can give you all the functionality on the router when you buy it, but you only have to unlock what you need.  No silly MPLS commands to get in the way of the clean, sleek dial peers and CUBE settings.

Of course, this does allow for other nefarious things as well.  VWICs now require you to have a Unified Communications license or they won’t work.  They bark and complain when you try to activate them if the license isn’t correct.  The commands don’t show up if the license isn’t right, just like the old days when you had the wrong IOS on the router.  The difference is the commands are still in IOS 15, they’re just hidden until you have the right license key.  Once you type in the right key (or upload the license file into flash), reboot and the thing magically starts working again!  This is also a way to make sure you keep up with your maintenance, as the ability to mark a key with an expiration date is now a distinct possibility.  Don’t pay your SmartNet?  No SSL VPN for you!  As well, if you purchase the router on E-Bay or through a 3rd party seller, it is very easy to disable the license for that router and force it to be repurchased.

My Thoughts

I’m all for upgrades.  New hardware makes me drool and faster software with fewer bugs is something everyone can enjoy.  But the licensing thing drives me bonkers.  I understand the reasons why you have to have it.  The pirates and dishonest people out there have seen to it that every slip up or advantage they can use to screw Cisco out of a few dollars are well worth it, no matter what price they might pay.  But it also makes the lives of an honest network engin…um, rock star miserable.  I’m all for finding a way to make the software easy to use and install without needing to spend half my install tracking down the one little slip of paper that came in the box with the right key.  Or worse, the key is in an envelope that got shipped to the Houston office when the router is is Columbus.  Things like that make enemies of your valuable resources.  And with everyone out there gunning for you now, the fewer enemies you have, the better.

I’m going to go one installing ISR G2s for my customers.  I don’t have a whole lot of choice in the matter.  Moreover, I actually like what they’ve done with the platform as far as USB console cables and such.  But when it comes down to uploading that silly license, I’m still going to grumble.  Not much I can do about that.

Bluntly BPDU–The Redux

Posted on by
5

Firstly, you should read this blog entry.  Go ahead, I’ll be here when you get back.

All done?  Good.  Now, for the first part of my response.  Mea culpa. I fracked up.

Based on this document: http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.1E/native/configuration/guide/stp_enha.html#wp1033403 I assumed an incomplete and incorrect stance on the way BPDUFiltering works.  This guide says that when a BPDU is received on a Portfast port, the port loses its Portfast state, which disables the BPDUFiltering and it begins acting like a regular STP port again.    However, thanks to Santino Rizzo for gently pointing out that BPDUFilter when enabled on an interface completely strips the BPDUs from being received and transmitted.  When enabled globally, it performs as I originally assumed, with the ports being transitioned out of Portfast and into regular STP ports.  I think the differences in my understanding come from the fact that the linked guide above is only for CatOS, which only allows BPDUFilter to be enabled globally.  The reference documents for interface-level BPDUFilter (like this one: http://www.cisco.com/en/US/docs/switches/lan/catalyst4000/7.4/configuration/guide/stp_enha.pdf) state correctly that it strips BPDUs from being transmitted at the interface level.  Okay, with that out of the way, on to the meat of my post…

When I started my CCIE walkabout 3 years ago, I had the good fortune to attend one of Narbik Kocharian’s boot camps.  One thing he told me there sticks with me to this day.  When we got into a discussion about the behavior of a certain protocol or configuration, he would always fire up a live router or switch to test our theories.  He told us, “IOS never lies.”  And he’s right.  IOS always does what you tell it to.  And it does it pretty much every time.  The same configuration gives you the same output.  So when we need to test something, we can jump into IOS and get out answer.  Because it will always tell us the truth.  That was my failing last night.  While I had been thinking of the discussion on Saturday, I didn’t take the time to lab it up.  So, I took my afternoon to do just that.

Let me state the parameters of my lab experiment.  I used two 3560 switches, one 24-port and one 8-port.  Both were running 12.2(25)SEE IOS, the 24-port was running the enhanced image, and the 8-port was running the base image.  I wiped the configuration of the two switches before starting and only configured a hostname on the 24-port switch.  The 24-port switch served as my test bed, and was the root of the spanning tree for VLAN 1, the only VLAN configured on the switches.  The only configuration done on the 8-port switch was to shutdown the connecting port between tests, and then re-enable it when I wanted to observe the output.  Otherwise all testing was done on the 24-port switch.  And now, I give you the results:

1.  No Configuration

I started out with a control of no configuration under the port at all.  This would give me a good baseline to work from.  I plugged the switches in and enabled the ports.  I then used the show spanning-tree interface fa 0/1 detail command to see the BPDUs.  Here’s the output from that test:


 Port 3 (FastEthernet0/1) of VLAN0001 is forwarding
   Port path cost 19, Port priority 128, Port Identifier 128.3.
   Designated root has priority 32769, address 0019.2f4d.4580
   Designated bridge has priority 32769, address 0019.2f4d.4580
   Designated port id is 128.3, designated path cost 0
   Timers: message age 0, forward delay 0, hold 0
   Number of transitions to forwarding state: 1
   Link type is point-to-point by default
   Bpdu filter is enabled internally
   BPDU: sent 44, received 1

Okay, so that’s what we expected.  BPDUs are being sent from the root bridge.  The switch has received one BPDU from its partner.  This is pretty much what happens straight out of the box.  now, let’s turn on Portfast.

2.  Portfast

I enabled access mode on the port with switchport mode access so that I could enable Portfast on the interface with spanning-tree portfast.  Afterwards, I enabled the port on the other switch and checked the output:


 Port 3 (FastEthernet0/1) of VLAN0001 is forwarding
   Port path cost 19, Port priority 128, Port Identifier 128.3.
   Designated root has priority 32769, address 0019.2f4d.4580
   Designated bridge has priority 32769, address 0019.2f4d.4580
   Designated port id is 128.3, designated path cost 0
   Timers: message age 0, forward delay 0, hold 0
   Number of transitions to forwarding state: 1
   The port is in the portfast mode
   Link type is point-to-point by default
   Bpdu filter is enabled internally
   BPDU: sent 20, received 1

Again, as expected.  The port is still sending BPDUs and receiving them.  The difference here is that the switch is now bypassing the blocking, listening, and learning phases and begins forwarding traffic immediately.  Bad thing if there is a loop behind it.  Okay, so the behavior is as expected so far.  Now, let’s start turning on the BPDU protection stuff.

3. BPDUGuard enabled per interface

For this test, I wanted to see what BPDUGuard does when enabled on the interface:

 


00:18:48: %SPANTREE-2-BLOCK_BPDUGUARD: Received BPDU on port FastEthernet0/1 with BPDU Guard enabled. Disabling port.

00:18:48: %PM-4-ERR_DISABLE: bpduguard error detected on Fa0/1, putting Fa0/1 in err-disable state

TestSwitch#sh spann int fa 0/1 det

no spanning tree info available for FastEthernet0/1

TestSwitch#sh int fa 0/1

FastEthernet0/1 is down, line protocol is down (err-disabled)

As soon as the switch received the BPDU from its partner, it slammed the door shut and placed the interface in err-disable.  No STP info because the port is essentially shutdown.

4.  BPDUFilter enabled per interface

After clearing the err-disable state, I removed the BPDUGuard configuration and ensured the port was no longer err-disabled.  I then enabled BPDUFilter on the interface with spanning-tree bpdufilter enable.  Here’s what I saw when I ran the spanning tree interface command:


 Port 3 (FastEthernet0/1) of VLAN0001 is forwarding
   Port path cost 19, Port priority 128, Port Identifier 128.3.
   Designated root has priority 32769, address 0019.2f4d.4580
   Designated bridge has priority 32769, address 0019.2f4d.4580
   Designated port id is 128.3, designated path cost 0
   Timers: message age 0, forward delay 0, hold 0
   Number of transitions to forwarding state: 1
   The port is in the portfast mode
   Link type is point-to-point by default
   Bpdu filter is enabled internally
   BPDU: sent 0, received 0

We now have STP information about the port, but no BPDUs being received or transmitted.  This is consistent with Santino’s description of the way BPDUFilter performs on an interface.  If you plugged a switch in now, it would see itself as the root bridge for each of it’s VLANs, as it isn’t seeing any BPDUs coming from its ports.  This would be the strange behavior I had seen with BPDUFilter when I had configured it in production earlier.  Makes sense to me now.  Let’s see how bad we can screw things up when we start combining things.

5.  BPDUGuard enabled globally, BPDUFilter enabled per interface

Setting the interfaces back to default, I then enabled BPDUGuard globally with spanning-tree portfast bpduguard default. I then enabled BPDUFilter on the interface with the previous command.  Here’s what I saw:


 Port 3 (FastEthernet0/1) of VLAN0001 is forwarding
   Port path cost 19, Port priority 128, Port Identifier 128.3.
   Designated root has priority 32769, address 0019.2f4d.4580
   Designated bridge has priority 32769, address 0019.2f4d.4580
   Designated port id is 128.3, designated path cost 0
   Timers: message age 0, forward delay 0, hold 0
   Number of transitions to forwarding state: 1
   The port is in the portfast mode
   Link type is point-to-point by default
   Bpdu guard is enabled by default
   Bpdu filter is enabled internally
   BPDU: sent 0, received 0

In this case, the inteface specific configuration overrrode the global config.  And the interface never received a BPDU to trigger the BPDUGuard function, so it remains in the forwarding state, albeit with no STP information on that port.  Let’s reverse it.

6.  BPDUFilter enabled globally, BPDUGuard enabled per interface

Back to default interfaces and removal of all global BPDU protection.  Now, I enable BPDUFilter with spanning-tree portfast bpdufilter default and enable BPDUGuard on the interface.  What do you think will happen?


TestSwitch#
00:27:59: %SPANTREE-2-BLOCK_BPDUGUARD: Received BPDU on port FastEthernet0/1 wit
h BPDU Guard enabled. Disabling port.
00:27:59: %PM-4-ERR_DISABLE: bpduguard error detected on Fa0/1, putting Fa0/1 in err-disable state
TestSwitch#sh spann int fa 0/1 det
no spanning tree info available for FastEthernet0/1

TestSwitch#sh int fa 0/1
FastEthernet0/1 is down, line protocol is down (err-disabled)

Just like expected.  The BPDU was received and the port shutdown before BPDUFilter could remove it.  As Santino explained above, BPDUFilter enabled globally evaluates the BPDUs on the Portfast ports and if one is received, the port would be transitioned out of Portfast and into a regular STP port.  Except in this case, where BPDUGuard does its job and disables the port before that can happen.  So far, so good.  Now for the big one.

7.  BPDUGuard and BPDUFilter enabled per interface

Back to default one more time.  Now, I enable BPDUGuard and BPDUFilter on the interface.  Once the config is done, I re-enable the port.  Here’s the payoff:


 Port 3 (FastEthernet0/1) of VLAN0001 is forwarding
   Port path cost 19, Port priority 128, Port Identifier 128.3.
   Designated root has priority 32769, address 0019.2f4d.4580
   Designated bridge has priority 32769, address 0019.2f4d.4580
   Designated port id is 128.3, designated path cost 0
   Timers: message age 0, forward delay 0, hold 0
   Number of transitions to forwarding state: 1
   The port is in the portfast mode
   Link type is point-to-point by default
   Bpdu guard is enabled
   Bpdu filter is enabled internally
   BPDU: sent 0, received 0

TestSwitch#sh run | beg 0/1
interface FastEthernet0/1
 switchport mode access
 spanning-tree portfast
 spanning-tree bpdufilter enable
 spanning-tree bpduguard enable

Ah ha!  It looks like BPDUFilter is stopping BPDUs from being transmitted or received on the port, as my new understanding of BPDUFilter explains.  Since there are no BPDUs, there is nothing to trigger the BPDUGuard feature!  Success!  As I said in the last post, when both are enabled on the same interface, BPDUFilter wins.  Yes, I admit I got confused about the function of BPDUFilter on the interface, but the results are the same.  I just have a better understanding of it now.

In conclusion, I learned a lot in the last 24 hours.  Always trust IOS.  It may not be clear all the time, but it never lies.  Make sure you lab it up when you have a question about things.  Listen to the commenters on your blog, they know their stuff.  And don’t be afraid to be blunt.  If nothing else, it makes you sure of your position.

Calm Before the Storm: BPDUGuard & BPDUFilter

Posted on by
13

A couple of days ago, a discussion erupted on Twitter  regarding the explanation and use cases for two of Cisco’s layer 2 edge protection technologies: BPDUGuard and BPDUFilter.  There were some interesting explanations and scenarios offered up, and I thought I’d give my take on it here as it will take a few more than 140 characters to lay out.

For those of you not familiar with BPDUs and why we need to guard and filter them, here’s the dime store tour of bridging 101.  The bridge is the most basic layer 2 device you can imagine.  It is designed to connect one network to another network.  The original bridge was designed by Radia Perlman while working at Digital Equipment Corp.  It was originally put in place to connect one of their customer’s LANs to another.  The story of the first bridge is outlined here: http://youtu.be/N-25NoCOnP4 and is highly recommended viewing for those not familiar with the origins of switching technology.  Radia was tasked with designing a method for bridges to detect loops in the network and avoid them at all costs, as a bridging loop would be catastrophic for data transmission.  She succeeded by creating a protocol that essentially form the network into a tree of nodes that prune links leading back to the root node.  In order to form this tree, each bridge sends out a special data packet called a Bridge Protocol Data Unit (BPDU).  This packet contains the information necessary for each bridge to build a path back to the root node/bridge and form a loop free path through the network.  If you’re interested in the exhausting detail of how BPDUs and spanning tree protocol (STP) work at a fundamental layer, check out the Wikipedia link.  You might say, “That’s great, but what does that have to do with my switches?” Well, if a bridge connects two networks together and segments their collision domains, think of a switch as the ultimate extension of that paradigm.  A switch is a device that bridges networks on every port, segmenting each port into its own collision domain.  Only, in today’s networks we don’t use hubs behind the bridge to connect end user devices, we use the switch as the user/system connection device.

So, now we know why switches send out BPDUs.  And now we hopefully know that BPDUs are very critical to the proper operation of a network.  So why on earth would we want to block or filter them?  Well, firstly any time that a new BPDU is seen on the network from a switch, it must send BPDUs toward the current root with the Topology Change Notification (TCN) flag enabled.  When the root bridge receives these BPDUs, it sets the TCN flag on its own BPDUs and forces the remaining nodes in the spanning tree to age out their topology tables.  The switches must then recalculate the spanning tree topology to ensure that the new switch has a path to the current root bridge or that the new switch IS the new root bridge.  This calculation can cause traffic to stop on your network for the duration of the calculation, which is 50 seconds by default.  So, when might the new switch become the new root bridge and cause chaos and despair in your network?  By default, bridges use a value known as the Bridge Priority to determine which one is the root.  A bridge with a lower priority is elected as the root bridge for that particular spanning tree instance.  Out of the box, the Bridge Priority value for most switches running regular 802.1d spanning tree is 32768.  So, assuming that all the bridge priorities in the network are the same, how do we break the tie?  The tie breaker is the MAC address of the bridge.  In most cases, this means the the device with the lowest MAC address is elected the root bridge.  And, in almost every case, the device with the lowest MAC address is the oldest bridge in your network.  So, if you pull an old switch out of the storage closet and plug it into the network, you’re going to cause a spanning tree election, and if you haven’t modified the Bridge Priority on your switches that old switch just might be elected the root bridge.  Which would cause your network to stop forwarding traffic for 50 critical seconds.  Those 50 seconds feel like an eternity to your users.  A word to the wise: ALWAYS set the bridge priority on the switch you want to be the root bridge.  Trust me, it’ll save you hours of pain in the future.

Users dislike a non-responsive network.  Immensely.  And under the default circumstances, when a user plugs a device into the switch, the switch does its job to determine if this device is sending BPDUs.  Which means the port has to do through the 50-second spanning tree process.  In most cases, this is not only unnecessary but annoying for the end user.  They don’t really care why what the switch is doing is so critical.  They just want to check their email.  How do we resolve this without breaking spanning tree?  Cisco decided to fix this with Portfast.  Portfast is a spanning tree enhancement that allows a network admin to say “This port is only going to ever have a PC plugged into it.  Please ignore the normal spanning tree process.”  What happens is that the port is immediately placed into the forwarding state, bypassing the learning and listening phases.  Spanning tree is not disabled on the port, but we also don’t take the time to listen for BPDUs or learn the information they contain.  This works great for end user nodes.  They get to check e-mail right away and you don’t get calls about the “slow network”.  And this works 90% of the time.  The other 10% is the stuff nightmares are made of.

Gertrude has one network port in her office.  She has a computer.  She bought a network printer and a new laptop.  She wants all three of these devices plugged into the network at the same time.  She buys a switch from a big box store so she can plug all these things in at the same time, not wanting to bother the IT department since they’ll either say ‘no’ or take a month to run a new cable to her office.  In her haste to get everything plugged in, she accidentally plugs one end of the network cable into the switch, and the other end into another port on the switch.  Then, she plugs her switch into the port on the wall.  And, if this port is Portfast-enabled, you’ve got yourself a Category 5 broadcast storm.  If you’re lucky enough to have never lived through one of these, count yourself fortunate.  Watch a spanning tree loop propagate through a network is like watching a firestorm.  Switch CPU’s spike to 100% load trying to process all the BPDUs flooding the network.  Users find themselves unable to get to the network, or in VoIP networks find themselves unable to use their phones.  Servers start going haywire and seeing themselves fighting for static IP address with…themselves.  And the only way for the IT department to fix the problem in most cases is to start unplugging switches until the culprit is found.  And heaven help Gertrude when they find her switch…

How could something like this happen?  Because Portfast assumes that the designated port is never going to have a switch connected to it, so it never bothers to listen for the BPDUs that would be a tell-tale sign of a loop.  It would never block the port initially while waiting for more information.  The Portfast switch gleefully starts forwarding packets and counting toward meltdown.  Portfast assumes that nothing bad could come from that port.  Anyone that works in IT knows that assumption is the mother of all frack-ups.  So, Cisco gave us two protocols to combat frack-ups, BPDUGuard and BPDUFilter.

BPDUGuard is a Portfast enhancement that functions as a fail-closed gatekeeper for the port.  As soon as a BPDU is detected on the port, BPDUGuard slams it shut and places the port into ‘err-disable’ mode.  Unless a recovery mode is configure (it isn’t by default), that port stay shutdown until the admins recover it.  In the above example, Gertrude plugs her switch in, and the switch detects a BPDU on a BPDUGuard-enabled port.  It gets disabled, and Gertrude can’t get on the network.  She calls into the IT helpdesk with her problem. The IT staff notice the port is err-disabled and investigate.  The IT staff go out to her office and find the switch before they re-enable the port.  After a stern talking-to, the network is saved and Gertrude gets her additional cable sometime next month.  BPDUGuard is the most-configured protection mechanism for this kind of issue.  Most IT admins want the port to shut off before the damage is done.  The problem with BPDUGuard is that if you aren’t the network admin, or if you aren’t in a position to turn the port back on quickly the user will experience an outage until the port is recovered.  If you’re a network admin that uses portfast, you should turn on BPDUGuard.  Don’t ask, just turn it on and save yourself even more hours of pain in the future.

BPDUFilter is a Portfast enhancement that functions as the fail-open moderator for the port.  Firstly, it prevents a switch from transmitting BPUDs on a Portfast enabled port (the switch still transmits BPDUs on Portfast ports).  If a BPDU is detected on the Portfast-enabled port, the Portfast state is removed and the port is forced to transition through the normal states of blocking, listening, and learning before it can begin forwarding.  In the above example, when Gertrude plugs her switch in, the uplink switch will detect the BPDU and force the device to transition through the regular spanning tree process.  It should also detect the loop and disable the highest-numbered port on the switch to disable the loop.  Gertrude will have to wait an additional minute before her port is up completely, but it will start forwarding.  The IT admins may never know what happened unless they notice Gertrude’s port is no longer in Portfast mode, or that a new switch is transmitting BPDUs from her switch port.  So why in the world would you use BPDUFilter?  In my experience, it is used when you are not the network admin for a given network and have no easy way to re-enable those ports that would be disabled by BPDUGuard.  Or, if the network policy for the particular network states that ports should begin forwarding immediately but that users should be able to connect devices without the port becoming disabled.  For the record, if you ever find a network policy that looks like this send it to me.  I’d really like to know who came up with it.  BPDUFilter is rarely used in my experience as a Portfast protection mechanism.

So, as these things usually happen, the question was asked during our discussion “What happens if you enable both BPDUGuard and BPDUFilter at the same time?” Well, I found a great blog post on the subject here: http://bit.ly/cKpBTd Essentially, if you enable BPDUFilter globally and enable BPDUGuard on a particular interface, the interface specific configuration takes precedence and shuts the port down before BPDUFilter can transition the port back to normal.  However, if you enable BPDUFilter using the interface-specific command and BPDUGuard using the interface-specific command, BPDUFilter will catch the BPDU first and transition the port to normal spanning tree mode before BPDUGuard can shut it down.  So, they each will perform their function while locking out the other.  The question becomes where each is configured (globally vs. interface-specific).  For those of you who might be in the unfortunate position to still be running CatOS, the only way to enable BPDUFilter is globally.  In this specific case, BPDUGuard will always win and the ports will be disabled.  You would only use BPDUFilter in this case to prevent ports from transmitting BPDUs.

My Take

Since best practice guidelines tell us that switch-to-switch connections should be trunk links, you should enable Portfast on all your user-facing ports to cut down on delay and trouble tickets.  But, if you have Portfast enabled, you better make sure to have BPDUGuard enabled at a minimum.  It will save your bacon one day.  The case for BPDUFilter is less compelling to me.  If you are in one of the few scenarios where BPDUFilter makes more sense than BPDUGuard, by all means use it.  It’s better than a poke in the eye with a sharp stick.  Personally, I’ve used BPDUFilter once or twice with mixed results.  My network started behaving quite strangely and some poorly-configured switches hanging off unidentified ports stopped responding until I removed the BPDUFilter configuration.  So I mainly stick to BPDUGuard now.  Better to have to re-enable a port after a user plugged in something they weren’t supposed to than to have to frantically unplug connections in the core in a vain effort to stem the raging broadcast storm.

EDIT

Be sure to check out my additional testing and findings over here.