IOS X-Treme!


As a nerd, I’m a huge fan of science fiction. One of my favorite shows was Stargate SG-1. Inside the show, there was a joke involving an in-universe TV program called “Wormhole X-Treme” that a writer unintentionally created based on knowledge of the fictional Stargate program. Essentially, it’s a story that’s almost the same as the one we’re watching, with just enough differences to be a totally unique experience. In many ways, that’s how I feel about the new versions of Cisco’s Internetwork Operating System (IOS) that have been coming out in recent months. They may look very similar to IOS. They may behave similarly to IOS. But to mistake them for IOS isn’t right. In this post, I’m going to talk about the three most popular IOS-like variants – IOS XE, IOS XR, and NX-OS.


IOS XE is the most IOS-like of all the new IOS builds that have been released. That’s because the entire point of the IOS XE project was to rebuild IOS to future proof the technology. Right now, the IOS that runs on routers (which will henceforth be called IOS Classic) is a monolithic kernel that runs all of the necessary modules in the same memory space. This means that if something happens to the routing engine or the LED indicator, it can cause the whole IOS kernel to crash if it runs out of memory. That may have been okay years ago but today’s mission critical networks can’t afford to have a rogue process bringing down an entire chassis switch. Cisco’s software engineers set out on a mission to rebuild the IOS CLI on a more robust platform.

IOS XE runs as a system daemon on a “modern Linux platform.” Which one is anyone’s guess. Cisco also abstracted the system functions out of the main kernel and into separate processes. That means that if one of them goes belly up it won’t take the core kernel with it. One of the other benefits of running the kernel as a system daemon is that you can now balance the workload of the processes across multiple processor cores. This was one of the more exciting things to me when I saw IOS XE for the first time. Thanks to the many folks that pointed out to me that the ASR 1000 was the first device to run IOS XE. The Catalyst 4500 (the first switch to get IOS XE) is using a multi core processor to do very interesting things, like the ability to run inline Wireshark on a processor core while still letting IOS have all the processor power it needs. Here’s a video describing that:

Because you can abstract the whole operation of the IOS feature set, you can begin to do things like offer a true virtual router like the CSR 1000. As many people have recently discovered, the CSR 1000 is built on IOS XE and can be booted and operated in a virtualized environment (like VMware Fusion or ESXi). The RAM requirements are fairly high for a desktop virtualization platform (CSR requires 4GB of RAM to run), but the promise is there for those that don’t want to keep using GNS3/Dynamips or Cisco’s IOU to emulate IOS-like features. IOS XE is the future of IOS development. It won’t be long until the next generation of supervisor engines and devices will be using it exclusively instead of relying on IOS Classic.


In keeping with the sci-fi theme of this post, IOS XR is what the Mirror Universe version of IOS would look like. Much like IOS XE, IOS XR does away with the monolithic kernel and shared memory space of IOS Classic. XR uses an OS from QNX to serve as the base for the IOS functions. XR also segments the ancillary process in IOS into separate memory spaces to prevent system crashes from an errant bug. XR is aimed at the larger service provider platforms like the ASR 9000 and CRS series of routers. You can see that in the way that XR can allow multiple routing protocol processes to be executed at the same time in different memory spaces. That’s a big key to the service provider.

What makes IOS XR so different from IOS Classic? That lies in the configuration method. While the CLI may resemble the IOS that you’re used to, the change methodology is totally foreign to Cisco people. Instead of making live config changes on a live system, the running configuration is forked into a separate memory space. Once you have created all the changes that you need to make, you have to perform a sanity check on the config before it can be moved into live production. That keeps you from screwing something up accidentally. Once you have performed a sanity check, you have to explicitly apply the configuration via a commit command. In the event that the config you applied to the router does indeed contain errors that weren’t caught by the sanity checker (like the wrong IP), you can issue a command to revert to a previous working config in a process known as rollback. All of the previous configuration sets are retained in NVRAM and remain available for reversion.

If you’re keeping track at home, this sounds an awful lot like Junos. Hence my Mirror Universe analogy. IOS XR is aimed at service providers, which is a market dominated by Juniper. SPs have gotten very used to the sanity checking and rollback capabilities provided by Junos. Cisco decided to offer those features in an SP-specific IOS package. There are many that want to see IOS XR ported from the ASR/CSR lines down into more common SP platforms. Only time will tell if that will happen. Jeff Fry has an excellent series of posts on IOS XR that I highly recommend if you want to learn more about the specifics of configuration on that platform.


NX-OS is the odd man out from the IOS family. It originally started life as Cisco’s SAN-OS, which was responsible for running the MDS line of fibre channel switches. Once Cisco started developing the Nexus switching platform, they decided to use SAN-OS as the basis for the operating system, as it already contained much of the code that would be needed to allow networking and storage protocols to interoperate on the device, a necessity for a converged data center switch. Eventually, the new OS became known as NX-OS.

NX-OS looks similar to the IOS Classic interface that most engineers have become accustomed to. However, the underlying OS is very different from what you’re used to. First off, not every feature of classic IOS is available on demand. Yes, a lot of the more esoteric feature sets (like the DHCP server) are just plain unavailable. But even the feature sets that are listed as available in the OS may not be in the actual running code. You need to active each of these via use of the feature keyword when you want to enable them. This “opt in” methodology ensures that the running code only contains essential modules as well as the features you want. That should make the security people happy from an exploit perspective, as it lowers the available attack surface of your OS.

Another unique feature of NX-OS is the interface naming convention. In IOS Classic, each interface is named via the speed. You can have Ethernet, FastEthernet, GigabitEthernet, TenGigabit, and even FortyGigabit interfaces. In NX-OS, you have one – Ethernet. NX-OS treats all interfaces as Ethernet regardless of the underlying speed. That’s great for a modular switch because it allows you to keep the same configuration no matter which line cards are running in the device. It also allows you to easily port the configuration to a newer device, say from Nexus 5500 to Nexus 6000, without needed to do a find/replace operation on the config and risk changing a line you weren’t supposed to. Besides, most of the time the engineer doesn’t care about whether an interface is gigabit or ten gigabit. They just want to program the second port on the third line card.

Tom’s Take

No software program can survive without updates. Especially if it is an operating system. The hardware designed to run version 1.0 is never the same as the hardware that version 5.0 or even 10.0 utilizes. Everything evolves to become more efficient and useful. Think of it like seasons of sci-fi shows. Every season tells a story. There may be some similarities, but people overall want the consistency of the characters they’ve come to love coupled with new stories and opportunities to increase character development. Network operating systems like IOS are no different. Engineers want the IOS-like interface but they also want separated control planes, robust sanity checking, and modularized feature insertion. Much like the writers of sci-fi, Cisco will continue to provide new features and functionality while still retaining the things to which we’ve grown accustomed. However, if Cisco ever comes up with a hare-brained idea like the Ori, I can promise there’s no way I’ll ever run IOS-Origin.

What’s My Cisco ATA Second Line MAC Address?

In the world of voice, not everything is wine and roses.  As much as we might want to transition everything over to digital IP phones and soft clients, the fact remains that there are some analog devices that still need connectivity on a new phone system.  The more common offender of this is the lowly fax machine.  Yes, even in this day and age we still need to rely on the tried-and-true facsimile machine to send photostatic copies of documents across the PSTN to a waiting party.  Never mind email or Dropbox or even carrier pigeon.  Fax machines seem to be the most important device connected to a phone system.  Normally, I leave the fax connections and their POTS lines intact without touching anything.  However, there are times when I don’t have that luxury.

In the case of the Cisco VoIP systems, that means relying on the Analog Terminal Adapter, or ATA.  The ATA allows you to connect an analog device to the unit, whether it be a fax machine or a cordless analog phone or even a fire alarm or postage machine.  It has many uses.  The configuration of the ATA is fairly straightforward under any CUCM system.  However, if you have a multitude of analog devices that you need to connect, you might opt to use the second analog port on the ATA.  The ATA 186 of the past and its current replacement, the ATA 187, both have 2 analog ports on the back.  There’s only one Ethernet port, though.  This is where the interesting part comes in to play.  If there are two analog ports but only one Ethernet port, how to I configure the MAC address for the second port?  All phone devices in CUCM must be identified by MAC address.  On an ATA, the primary MAC address printed on the bottom or the side of the box is the address for the first port.

If you want to use the second port, you’re going to have to do a little bit of disassembly.  Cisco uses a standard method to create a new MAC address:

1.  Take the MAC address for port 1.  For example, 00:00:DE:AD:BE:EF.

2.  Drop the first two digits from the MAC address.  In the example, 00:DE:AD:BE:EF.

3.  Append “01” to the end of the 10-digit address.  Example, 00:DE:AD:BE:EF:01.

Once you’ve completed those steps, take the MAC address you’ve just created and plug it into CUCM as a new ATA device.  Once you’ve completed the necessary steps to create the new device, it will register with the DN you’ve assigned to it.  Then you can start calling or faxing it to your heart’s content.

There’s no mention of the secondary MAC address anywhere on the web interface.  You’d figure it wouldn’t be hard to write some HTML function to read the MAC address and do the above operation.  The Cisco documentation buries this information deep inside the setup document.  I’ve even search Cisco’s very own support forums and found all manner of advice that doesn’t work correctly.  I decided that it was time to jot this information down in a handy place for the next time I need to remember how to configure the ATA’s second port.  I hope you find it useful as well.

The Card Type Command – Don’t Flop

If you’ve ever found yourself staring at a VWIC2-1MFT-T1/E1 or a NM1-T3/E3 module, you know that you’ve got some configuration work ahead of you.  Whether it be for a PRI circuit to hook up that new VoIP system or a DS3 to get a faster network connection, the T1/T3 circuit still exists in many places today.  However, I’ve seen quite a few people that have been stymied in their efforts to get these humble interface cards connected to a router.  I have even returned a T1/E1 card myself when I thought that it was defective.  Imagine the egg on my face when I discovered that the error was mine.

It turns out that ordering the T1/E1 or T3/E3 module from Cisco requires a little more planning on the installation side of things.  These cards can have a dual identity because the delivery mechanism for these circuits is identical.  In the case of a T1/E1, the delivery mechanism is almost always over an unshielded twisted pair (UTP) cable.  Almost all of the T3/E3 circuits that I’ve installed have been delivered over fiber but terminated via coax cables with BNC connectors.  The magic, then, is in the location.  A T1 circuit is typically delivered in North America, while the E1 circuit is European version.  There are also differences in the specifics of each circuit.  A T1 is 24 channels of 64kbits each.  An E1 is 32 channels of the same size.  This means that a T1 has an effective data rate of 1.544 Mbits while an E1 is a bit faster a 2.048 Mbits.  There are also framing differences and a slightly different signaling structure.  The long and short of it is that T1 and E1 circuits are incompatible with each other.  So how does Cisco manage to ship a module that supports both circuit types?

The key is that you must choose which circuit you are going to support when you install the card.  The card can’t automatically flip back and forth based on circuit detection.  Where the majority of issues come from in my line of work is that the card doesn’t show up as a configurable interface until you force a circuit type.  This is accomplished by using the card type command:

RouterA(config)#card type ?
 e1 E1
 e3 E3
 t1 T1
 t3 T3

Choose your circuit type and away you go!  As soon as you enter the card type, the appropriate serial interface is created.  You will still need to enter the controller interface to set parameters like the framing and line code.  However, the controller interface only shows up when the card type has been set as well.  So unless you’ve done the first step, there isn’t going to be a place to enter any additional commands.

Tom’s Take

Sometimes there are things that seem so elementary that you forget to do them.  Checking a power plug, flipping a light switch, or even remembering to look for little blinking lights.  We don’t think about doing all the easy stuff because we’re concentrating on the hard problems.  After all our hard work, we know it has to be something really messed up otherwise it would be fixed by now.  In the case of T1/E1 cards, I made that mistake.  I forgot to check everything before declaring the card dead on arrival.  Now, I find myself spending a lot of time providing that voice of reason for others when they’re sure that it has to be something else.  The little voice of reason doesn’t always have to be loud, sometimes it just has to say something at the right time.

DST Configuration – Just In the Nick of Time

Today is the dreaded day in the US (and other places) when we must sacrifice an hour of our precious time to the sun deity so that he might rise again in the morning.  While this is great for being outdoors and enjoying the sunshine all the way into the late evening hours, it does wreak havoc on our networking equipment that relies on precise timing to let us know when a core dump happened or when that last PRI call came in when running debug isdn q931.  However, getting the right time running on our devices can be a challenge.  In this post, I will cover configuring Daylight Savings Time on Cisco, HP, and Juniper network equipment for the most pervasive OS deployments.  Note that some configurations are more complicated than others.  Also, I will be using Central Time (CST/CDT) for my examples, which is GMT -6 (-5 in DST).  Adjust as necessary for your neck of the woods.  I’m also going to assume that you’ve configured NTP/SNTP on your devices.  If not, read my blog post about it and go do that first.  Don’t worry, I’ll still be here when you get back.  I have free time.


I’ve covered the basics of setting DST config on Cisco IOS before, but I’ll put it here for the sake of completeness.  In IOS (and IOS XR), you must first set the time zone for your device:

R1(config)# clock timezone <name> <GMT offset>
R1(config)# clock timezone CST -6

Easy, right?  Now for the fun part.  Cisco has always required manual configuration of DST on their IOS devices.  This is likely due to them being shipped all around the world and various countries observing DST (or not) and even different regions observing it differently.  At any rate, you must the clock summer-time command to configure your IOS clock to jump when needed.  Note that in the US, DST begins at 2:00 a.m. local time on the second Sunday in March and ends a 2:00 a.m. local time on the first Sunday in November.  That will help you decode this code string:

R1(config)# clock summer-time <name> recurring <week number start> <day> <month> <time to start> <week number end> <day> <month> <time to end>
R1(config)# clock summer-time CDT recurring 2 Sun Mar 2:00 1 Sun Nov 2:00

Now your clock will jump when necessary on the correct day.  Note that this was a really handy configuration requirement to have in 2007, when the US government decided to change DST from the previous requirement of the first Sunday in April at the start and the last Sunday in October to end.  With Cisco, manual reconfiguration was required, but no OS updates were needed.

HP (Procurve/E-Series and H3C/A-Series)

As near as I can tell, all HP Networking devices derive their DST settings from the OS.  That’s great…unless you’re working on an old device or one that hasn’t been updated since the last presidential administration.  It turns out that many old HP Procurve network devices still have the pre-2007 US DST rules hard-coded in the OS.  In order to fix them, you’re going to need to plug in a config change:

ProCurve(config)# time daylight-time-rule user-defined begin-date 3/8 end-date 11/1

I know what you’re thinking.  Isn’t that going to be a pain to change every year if the dates are hard-coded?  Turns out the HP guys were ahead of us on that one too.  The system is smart enough to know that DST always happens on a Sunday.  By configuring the rule to occur on March 8th (the earliest possible second Sunday in March) and November 1st (the earliest possible first Sunday in November), the system will wait until the Sunday that matches or follows that date to enact the DST for the device.  Hooray for logic!  Note that if you upgrade the OS of your device to a release that supports the correct post-2007 DST configuration, you won’t need to remove the above configuration.  It will work correctly.


Juniper configures DST based on the information found in the IANA Timezone Database, often just called tz.  First, you want to get your device configured for NTP.  I’m going to refer you to Rich Lowton’s excellent blog post about that.  After you’ve configured your timezone in Junos, the system will automatically correct your local clock to reflect DST when appropriate.  Very handy, and it makes sense when you consider that Junos is based heavily on BSD for basic OS operation.  One thing that did give me pause about this has nothing to do with Junos itself, but with the fact that there have been issues with the tz database, even as late as last October.  Thankfully, that little petty lawsuit was sidestepped thanks to the IANA taking control of the tz database.  Should you find yourself in need of making major changes to the Junos tz database without the need to do a complete system update, check out these handy instructions for setting a custom timezone over at Juniper’s website.  Just don’t be afraid to get your hands dirty with some BSD commands.

Tom’s Take

Daylight Savings Time is one of my least favorite things.  I can’t see the advantage of having that extra hour of daylight to push the sunlight well past bedtime for my kids.  Likewise, I think waking up to sunrise is overrated.  As a networking professional, DST changes give me heartburn even when everything runs correctly.  And I’m not even going to bring up the issues with phone systems like CallManager 4.x and the “never going to be patched” DST issues with Windows 2000.  Or the Java issues with 79xx phones that still creep up to this day and make DST and confusing couple of weeks for those that won’t upgrade technology. Or even the bugs in the iPhone with DST that cause clocks to spring the wrong way or alarms to fail to fire at the proper time.  In the end though, network enginee…rock stars are required to pull out our magical bags and make everything “just work”.  Thanks to some foresight by major networking vendors, it’s fairly easy to figure out DST changes and have them applied automagically.  It’s also easy to change things when someone decides they want their kids to have an extra hour of daylight to go trick-or-treating at Halloween (I really wish I was kidding).  If you make sure you’ve taken care of everything ahead of time, you won’t have to worry about losing more than just one hour of sleep on the second Sunday in March.

I Have The Power! – Common Electrical Connectors

I’m no electrician.  In fact, the last time I tried to do some electrical work ended up with a minor electrocution and me not being able to taste for an hour.  However, in the IT world electricity is a key to our jobs.  The mightiest Nexus 7k or QFabric deployment can be brought to its knees by inadequate power.  The need to understand power and the the connectors that go along with it are vital.

Fabulous secrets were revealed to me the day I pulled up the Internet and started looking up the various codes for connectors that are used in electrical work.  I figured out pretty quickly that electricians had a language all their own and that I needed to learn how to speak some of it in order to get things accomplished.  After all, describing a connector as “one goes like this, the other goes like that” isn’t really helpful, especially over the phone.  I wanted to pass on a bit of what I learned to all of you.  A note for my international readers: this is going to be focused primarily on US connectors.  However, if you’d like to fly me to your country for a few days, I’ll be more than happy to bring a travel kit and research your electrical code from my hotel room.  Just a thought…

Some of these connectors standardized by the National Electrical Manufacturers Association (NEMA).  They also hold the trademark on the term “twist lock”, so when I use it, know that it belongs to them.  The others are standardized by the International Electrotechnical Commission (IEC) under IEC 60320.


The NEMA 5-series plug and receptacles are the most common found in the US today.  They are three-wire circuits (hot, neutral, and ground) and are rated to carry a maximum of 125 volts, although they usually carry about 110 volts and are referred to as “110 circuits”. All of these connectors will start with a “5” and a dash, followed by the amperage of the circuit, from 15 amps to 30 amps.


By far the most common connector you’ll run into in the US.  A simple 110-125v circuit rated at 15 amps.  Whatever you do, don’t rip the ground plug out if you need to fit this into a NEMA 1-15 two prong outlet.  I’ve seen what happens there and it isn’t pretty.


The NEMA 5-20 connector is more common as you being to start using equipment with high wattage power supplies, like Catalyst 4500 switches.  I’ve always heard the 5-20 described as a “dedicated circuit” plug, but that’s not entirely accurate.  It’s a 110-125v 20 amp circuit.  It can easily be identified by the perpendicular blade.


Here’s where the real fun starts.  This little jewel was the source of my first head scratching moment with NEMA connectors.  This plug is the same as its 5-20 cousin above, however the connectors look all wrong.  This connector is designed to be inserted into the receptacle and then rotated counterclockwise to lock it in.  This way, it can’t simply be pulled out by someone tripping over it or by vibrations from heavy machinery.  Do take care though, as tugging on the connector too hard will cause it to rip the whole receptacle assemble out of the wall with possibly some exposed wires.


The NEMA L5-30 is the big boy of 110-125v circuits.  I found it curious that I have never really seen the non-locking version of this plug, but I’ve since been informed that the majority of devices that use 30 amp circuits prefer this connector to prevent it from being yanked out.  Uninterruptible Power Supplies (UPSs) use this connector frequently in order to ensure their devices are getting enough power.


The NEMA 6 series connectors are the ones you use when you need to provide heavy duty power to a device.  These are typically 208 volt or 240 volt circuits, but I’ve always heard them referred to as “220 circuits”.  There is no neutral wire in this setup, as both non-ground wires are delivering power.  These are the kind of connectors used in the home for things like air conditioners or clothes dryers.  In my experience, the standard versions of these connectors are uncommon.  The locking version are used far more frequently, usually do to the fact that whatever is running off of that much power probably doesn’t want to go offline due to having a power plug kicked out of the wall.  These connectors all begin with a “6” followed by the amperage of the circuit, usually between 15 and 30 amps.


The only non-locking NEMA 6-series connector I run into on a regular basis is the 6-20.  Note that while this plug/receptacle is similar to it’s 5-20 cousin, the perpendicular blade is the opposite on this connector to prevent you from plugging the wrong cord into the wrong plug and getting a nasty surprise.


The locking version of the NEMA 6-20.  Far more common due to the ability to keep whatever this cord is powering plugged in.  Odds are good that if you are using a heavy duty power supply in a device like a Catalyst 4500 or a Catalyst 6500, you’ve got this cord powering your device.


Big devices call for big power.  This 208/240v 30 amp circuit connector is going to power just about anything you can throw at it.  Big UPSes or huge power supplies in a switch like the 8700 watt monster in the Catalyst 6500.  You are most likely to encounter this connector in a rack-mounted PDU where it provides the power coming from the main electrical connection and then the PDU disperses the power to the devices in the rack itself.

IEC 60320

The IEC connectors are fairly common in electronics devices, however I believe that most people couldn’t pick them out of a lineup.  That’s because the IEC connectors are the ends that are on the opposite side of the cord from the power plug.  These ends plug into power transformers and power supplies.  By making them an international standard, the equipment manufacturers need only put one kind of receptacle on their equipment and then manufacture the various country-specific cords when needed.  Also, unlike the NEMA connectors above that use “P” and “R” to denote plugs and receptacles, the IEC connectors use a different number to specify the plug and receptacle, for instance the IEC C19 is the plug and the IEC C20 is the receptacle.


This connector is used for power transformers, laptop power supplies and small appliances, like the original Playstation.  This connector is shaped like a figure 8, unless you have the polarized version when is squared off on the hot end.  I’ve spent many an hour looking for one of these connectors for a forgotten device.


If you’ve ever worked with a computer, you know this power cord.  In fact, for the majority of my career I’ve called it a “computer power cord”.  This particular IEC connector run everything from desktop computer to fixed-configuration switches.  I always carry two or three spare cables with me at all times just in case I need them.  I’m also starting to see more and more higher wattage devices requiring two or more cords to work properly, like server power supplies or the newer power supplies for chassis switches.  The C14 plug is also very common on power strip type PDUs for racks where you plug the C13 end into your server and the C14 end into the PDU.  Saves a bit more space than the NEMA 5-15 connector above.


Odds are good you’ve seen this cable and said “Huh?”.  I’ve started seeing it ship with newer fixed-configuration switches that had Power over Ethernet (PoE) power supplies.  For the life of me, I couldn’t figure out the point of changing this connector.  A chance comment from one of my friends about this so-called “kettle cord” made me start researching what was so special about the connector.  It turns out the C13/C14 connector/cord is only rated to about 70 degrees F.  The C15/C16 was specifically designed for higher temperature devices, like electric kettles and PoE switches with higher wattage power supplies, like those that drive 802.3at or PoE+ 30 watts per port.  This connector will work in the C13/C14 receptacles, but not vice versa.  This prevents the cord melting and causing all kinds of trouble.


If you’ve ever installed a Catalyst 4500 switch, you’ve seen this plug on the power supply.  For a long time, I referred to it as the “chassis switch power connector” without knowing it had a real name.  The C19/C20 connectors are used for devices that require a higher current than that which can be provided by the C13/C14/C15/C16 connectors.  I’ve only ever seen it on chassis switches, but it can also appear in high end workstations or some kinds of UPSes.

Tom’s Take

When you stare at the long list of power options available for a switch or a server, you might find your eyes crossing at the multitude of unfamiliar acronyms you find.  At best you’ll end up with an extra power cord you don’t need.  At worst, you could delay a large project for many days because you ordered an L6-20 cord when you really needed a 5-20 connector.  The same goes for ensuring you have the correct power connections hooked up in your server closet or data center.  Electricians aren’t sure what you’re talking about when you tell them you want a “winking man” connector or “the twisty one” kind of plug.  If you tell them you need an L5-20R, they’ll be able to put one in without asking another question.  Network rock stars always talk about the virtues of standardization and the electrical world is no different.  By knowing the standard name for the NEMA or IEC connector you need, you can be sure that you are the one that has the power.

Nerd v6 – My First IPv6 Tunnel

Home - Courtesy of ThinkGeek (Click to buy this shirt!)

I realized the other day that I’ve done a lot of IPv6-related posts in the past few months, but for one reason or another I keep putting off setting up my own IPv6 presence.  I signed up for a free IPv6 tunnel from Hurricane Electric’s Tunnel Broker service almost a year ago.  However, the Cisco Valet Plus router I have running my home network has zero support for IPv6, let alone tunnels.  Because of that little omission, my tunnel sat unused for a while, gathering 128-bit dust.  As the end of the year approached, I found myself with some extra time on my hands and a bit of spare gear thanks to my local Cisco office.  I decided that maybe it was time to turn up my IPv6 nerdiness to the next level.

I found an old 2821 in my lab that wasn’t serving an immediate purpose.  I erased it and reconfigured it to serve as a gateway for my IPv6 setup.  I logged back into my account at and found that I could create up to 5 tunnels for free.  Who in their right mind would turn that down?  I created a new tunnel for my lab environment.  In this interface, you would create a regular tunnel for basic connectivity.  You input your IPv4 address as one side of the endpoint. provides the IP that you’re coming in on if you wanted to set this up with a home connection, for instance.  In my case, I already had a global IPv4 address ready to go.  However, the router wasn’t all the way up yet.  If can’t ping your IPv4 tunnel endpoint, they won’t reserve the address space.  So:

Tip #1: Don’t start setting up your tunnel until you’ve got your gateway ready.

I picked the closest endpoint to my geographic area – Dallas, TX.  Once the router came all the way up and the ARP caches had settled down, I registered my new tunnel without a hitch. provides you with a /64 for your tunnel interface.  ::1 is an interface on their side, ::2 is an interface to assign to your tunnel.  They also provide you with an entirely different /64 to setup for your client devices behind your router/firewall.  If you’re wanting to bring more than one site online, you can even register for a /48.  That’s 1.20892582 × 10^24 addresses.  Per tunnel.  That’s a lot for nothing!

My first attempt with Dallas didn’t work out so well.  For some reason, I couldn’t ping the other side of my tunnel.  I gave it about 15 minutes and then gave up.  I tore down the tunnel and created a new one.  If you’re going to do that, give the servers about 5 minutes to clean up their side of things, since the tunnel creation script will think that you’re  trying to register an endpoint twice.  I picked a nice little sunny patch of hex addresses in Fremont, CA and this time it worked!  I was able to ping from one side of my tunnel to the other.  For reference, this is the sample config they gave me for the tunnel and it worked quite nicely:

interface Tunnel0
 description Hurricane Electric IPv6 Tunnel Broker
 no ip address
 ipv6 enable
 ipv6 address <Your Tunnel /64 Endpoint>
 tunnel source <Your IPv4 Interface>
 tunnel destination <'s IPv4 Interface>
 tunnel mode ipv6ip
ipv6 route ::/0 Tunnel0

Easy, right?  Once that’s up and running, you can configure the other interface of your router with the routed /64 or /48 they assign you.  I’d suggest starting with the /64 first to get your feet wet.  There is one other piece of configuration you need to enable that seems to be the cause of many issues on’s forums when configuring Cisco devices.  You need to enable IPv6 routing with this command:

ipv6 unicast-routing

Tip #2: Don’t forget to enable IPv6 routing.

Now that you’ve got two sides up and running and routing between each other, you should be able to launch some packets toward the Interwebz v6. sets you up with an IPv6 DNS server you can plug into your devices to test connectivity.  If you want to be able to ping from your router, be sure to enter this command:

ip name-server 2001:470:20::2

Now you can test to your heart’s content.  When you’re sure that your tunnel is going to stay up, you need to concentrate on getting desktops working.  That’s where the routed /64 comes into play. gives you an additional routed /64 that is different than the tunnel address for the purposes of setting up a site for IPv6.  Most networking people know that you must have two different subnets on the router for routing to occur.  Yet, on the forums I see a lot of people that are configuring their routers with addresses from the tunnel /64 subnet.  Save yourself the headache and use the other /64 to get started.

The easiest way to setup your client device with an IPv6 address is good old fashioned static addressing.  This is very easy to do in both Windows and OS X.  Lucky for you that both of the latest versions of those OSes have IPv6 enabled by default.  You just have to click on the option for IPv6 and assign a static IP.  You should also use the IPv6 name server listed above to allow you to resolve addresses like  I tested with an OS X Lion server and was able to get the machine running on a static IP with no real issues.  I gave my Windows 7 workstation an IP in the same range and they were able to happily ping each other with no problems.  I think I’m going to take another post to talk about the fun of configuring DHCPv6 and SLAAC on my tunnel, as that has caused me a bit of heartburn so far making everything play nice with other people’s ideas about security.

Speaking of security…I would be remiss if I didn’t end this little article without a discussion of securing your new tunneling router from the nastiness on the Internet.  I found a great access list on the forums and thought I’d share it with you:

ipv6 access-list internet_inbound_ipv6
 remark Permit IPv6 Link-Local & Multicast
 permit ipv6 FE00::/7 any
 remark Block IPv6 Bogons
 deny ipv6 ::/3 any
 deny ipv6 4000::/2 any
 deny ipv6 8000::/1 any
 remark Block own assigned IPv6 space
 deny ipv6 <Your /64>::/64 any
 remark Block anything going to Windows RPC
 deny tcp any any eq 135
 permit icmp any any
ipv6 access-list internet_outbound_ipv6
 remark Prohibit any contact with Windows RPC-NetBIOS
 deny tcp any any eq 135
 deny tcp any any eq 137
 deny tcp any any eq 138
 deny tcp any any eq 139
 deny udp any any eq 135
 deny udp any any eq netbios-ns
 deny udp any any eq netbios-dgm
 deny udp any any eq netbios-ss
 remark Allow traffic from own assigned IP space
 permit ipv6 <Your /64>::/64 any

Of note here, remember that in IPv6, ICMP does a lot more than just pings.  Read RFC4890 for a lot more info on the subject, but right now I’m just allowing the whole stack in.  If you want to permit traffic to servers for things like HTTP or SMTP, be sure to add those servers at the end of the Inbound ACL so the traffic doesn’t get dropped.

Tom’s Take

One of the things that impressed me when I started troubleshooting some of the issues with my tunnel was the help that people were more than willing to give in the forums.  Especially where they helped their brethren with things like establishing connectivity.  Lots of posts about being able to ping router tunnel endpoints and things like that.  It shows that setting up your own IPv6 presence isn’t really that hard and should allow you to get out on the wider Internet with IPv6 in short order.  Thanks to all the work that other’s have been more than willing to share, I had an easier time than many.  I just hope my examples here help someone to get their tunnels up and running.

Rollover Beethoven – USB’s In Town

Every Cisco engine…rock star in the world should have a rollover cable or two stashed away in their bag/car/pocket just in case.  The rollover serial cable is the hallmark of access to a Cisco device.  The console port is the last resort for configuration when all else has gone wrong.  It is the first thing you should plug into when you boot up a router for the first time and the best way to get info you couldn’t otherwise find.  However, the days of the serial cable are quickly becoming numbered.

It wasn’t all that long ago that every PC manufactured included a 9-pin serial connection.  These ports were handy for all kinds of devices, including printers and modems.  However, with the introduction of Universal Serial Bus (USB) connections, the usefulness of the serial (and parallel) ports has been waning quickly.  By utilizing a higher speed connection that more tightly integrates into the system, the need to configure devices with DIP switches and play COM port roulette have long since passed.  As it is with any transition though, there have been some holdouts in the movement to retire serial ports.  While some of these are understandable due to outdated single-purpose technology, others have never made any sense to me, like the Cisco rollover console cable.  Surely there must be a better way to connect to the serial port of a device than with an outdated technology holdover from the 80s?  I myself am a victim of this kind of thinking, having used an IBM T30 Thinkpad well past its useful life simply because it had an integrated serial port and my replacement laptop wouldn’t.

When Cisco developed the new ISR G2 line of routers, someone in the console access department finally decided to wake up and get with the 2000’s.  Thanks to their efforts, the Cisco routers and switches manufactured today have started including a new console access option:

In the picture above, you can see the familiar RJ-45 console port to the right and the newer USB console port to the left, indicated with the USB icon.  This new port allows those of us that have spent most of our lives using the flat blue rollover serial cables to add a new, exciting cable to our bag, the USB A-to-mini cable.

The new USB port allows the user to access the router’s console with a newer cable instead of relying on the standby rollover cable.  However, you need to take a few steps first.  You have to head out to the Cisco Connections Online (CCO) download page and pull the driver for your particular operating system if you’re running on Windows.  Make sure you specify 32-bit or 64-bit, since this driver will be masquerading as a COM port on your system.  You don’t want to waste time downloading a driver that won’t work.  Once you’ve installed the driver, you can plug in your USB connection to any USB port and then to the router.  It will look like an additional COM port on your system, probably with a high number like COM6 or COM7, so make sure you’ve got a terminal emulator that allows you to choose your COM port.  I tend to use TeraTerm for this very reason, but your terminal program of choice should do nicely.  For those of you in the audience with Macbooks, you don’t need to download any drivers at all.  Seems like OS X already has the right driver built in, so just plug and and get cranking.  As a quick aside, Cisco will attempt to sell you a $30 USB console cable when you order the router. JUST. SAY. NO.  This is a regular USB A-to-Mini cable that can be purchased at Walmart for about $10.  You can even use the USB cable that came with your digital camera or Blackberry or old Motorola RAZR.

Once you get attached to the USB console port, you’ll find that it works pretty much the same as the RJ-45 port that you’ve become attached to over the years.  You can also plug in a regular old serial cable into the RJ-45 port if you need a second connection.  The RJ-45 console port will mirror what’s going on with the USB console port.  However, since their both Console 0, only one of them will have preference on the input.  In this case, that’s the USB port.  So if you have a terminal access server plugged in for reverse telnet connections and someone comes in and attaches a USB connector, you can watch what’s going on but you can’t do anything about it.  You can specify a timeout value if you’d like so you can force a logout after inactivity.  You can do that with the following command:

Router(config)# line con 0
Router(config-line)#usb-inactivity-timeout <value in minutes>

Note that this command doesn’t work on the 2900 series ISR G2 routers for some strange reason.  Oh well, feature request down the road.  For those of you out there that don’t feel comfortable with the idea of having just anyone off the street walking up and consoling into your router via USB, you can always disable the USB console port in favor of the RJ-45 connection as follows:

Router(config)# line con 0
Router(config-line)# media-type rj45

Bingo.  USB port locked out.  Now only those people in possession of a serial-to-USB adapter or a Redpark iOS Console Cable will have access.

Tom’s Take

I have three rollover cables in my various laptop bags.  I keep two for emergencies and one in case someone doesn’t have theirs.  I passed out console cables to all my engineers and technicians once and told them the next time I asked them for their console cable, they’d better present this one to me.  A console cable is an indispensable tool for anyone that works on Cisco equipment.  Having the USB option is always welcome since I no longer have to fumble for my USB-to-serial adapter or worry that the dodgy drivers are going to bluescreen my Windows 7 64-bit laptop over and over again.  Still, there is a lot of Cisco equipment out there with the older RJ-45 cable setup as the only console option.   So you can’t just throw out the old rollover serial cables just yet.  Better to throw a USB cable in your bag for those glorious days where you get to access a newer device.  Then you can await the day when you can bury your rollover cable alongside Beethoven.

Fruit Company Console: My Review of the Cisco Console Companion for iPad/iPhone

One of the major advantages to owning an iPad, or in some cases an iPhone, is that you have a mobile computer at your fingertips that is quite easy to carry around the datacenter or networking closet.  I have an iPad myself, and I find it very useful for documentation purposes.  Whether it be taking notes about the configuration of a specific device or looking up the PDF of a particular feature from Cisco’s website, the iPad has many uses.  However, if I find myself in need of connecting to a device such as a switch or a router, my iPad/iPhone options are limited.  I can use a telnet or SSH client to remote into the system, but if I don’t know the management IP or the username/password combination I can be sunk.  Or worse yet, if the switch has never been properly configured for remote access it becomes a moot point.  If I want to be able to use my trusty Cisco rollover console cable to get into the switch the old fashioned way, I have to lug out my behemoth Lenovo W701 laptop and get it ready, which can be quite an endeavor depending on the amount of room I have to work with or the amount of time that I’m going to spend consoled in, since my laptop has about 1.5 hours of battery life under the best of circumstances.  Add in the difficulties that I’ve faced with USB-to-serial adapters under Windows 7 64-bit and you can see why I’m reluctant to use the console.  However, there is hope for the best of these two worlds.

A company called Redpark has started selling a rollover cable with a 30-pin iDevice connector.  Engadget had a story about it HERE.  Naturally, I decided that I just had to have one of these.  You know…for work and stuff.  Anyway, I jumped right over to the Redpark website.  Hello sticker shock.  This baby is going to set you back a cool $69.  Add in more if you want shipping and handling (whatever that is), so expect to shell out about $80 to get it to your neck of the woods, more if you need to have one tomorrow.  That’s not all, folks!  Even if you do manage to get your hands on one of these little jewels, you still need an app to access the console.  Now those of you that looked at this excellent blog post by Ruhann about console access on a jailbroken iPad are all set.  The rest of us poor saps that haven’t jailbroken our iPads yet are in a bit of a lurch.  Fear not, because the company also has an official app on the App Store called Get Console (or Cisco Console Companion) that will give you console access.  For a measly $9.99.  After all, you’ve already spent $80 already, what’s a few dollars more?

Once my console cable arrived in the mail, I was a little underwhelmed by the packaging:

Not much to look at.  The contents of the box were even worse.  The console cable lovingly encased in bubble wrap, and this instruction sheet:

Bravo for making it straightforward and easy to read.  Off to the App Store to download my new app.  Except…”Cisco Console Companion” isn’t the official title of the app.  It’s “Get Console”, along with a big disclaimer that it is in no way associated with Cisco.  I’m guessing they had to use an alternate title in the app store because of some wonky trademark issues that Uncle John wasn’t too pleased about.  At any rate, it was a fast download and then I was off and running.

For the purposes of this test, I’m consoling into a Cisco Catalyst 3560 8-port switch.  Once I fired up the program, it popped up with a one-time reminder that it was only for Cisco devices and that it would check each device to ensure that it was a genuine Cisco product.  My best guess is this is there to prevent people from trying to use it as an Ethernet cable or something, because most reports I’ve seen says that it works just fine with any kind of device that uses a rollover cable, like Juniper, or HP, or what have you.  I didn’t test this out during my first run, but I will be testing it down the road of some of those devices.  Note that since it is an RJ-45 rollover cable, it can’t be used on RS-232 or null modem devices.  Oh well, time to upgrade those old switches anyway.  The cable itself feels rather thin, almost like a fiber patch cable rather than a flat rollover cable or even a UTP cable.  It’s about 6 feet long, so you don’t have to be right next to the device you’re trying to console into, but don’t expect to be programming from across the room.  Here’s a picture of the cable on top of my test switch:

My first encounter with the Get Console program led me to this screen:

Fairly utilitarian, but that’s fine by me.  I’m not really a “bells and whistles” kind of guy.  The bottom section of the screen is dominated by the on-screen keyboard, but that’s to be expected.  Just above that is a collapsible keyboard bar that lists some very useful control keys.  First is the all-important TAB key, which I’ve found sorely lacking on some of the telnet clients I’ve used.  TAB saves me a ton of time.  Next is the CTRL key, which when tapped toggles on and allows you to use CTRL+ shortcuts for moving around the command line or sending a CTRL+C or CTRL+Z to end.  Next is the BRK key, which sends an immediate break signal to the console.  Useful for those times when you need to enter ROMMON on bootup.  Next is everyone’s favorite question mark key.  Having it here is really helpful so that I don’t have to waste a keystroke getting to the number/symbol keyboard on the iPad.  This is followed by the up and down error keys, which are used to cycle through your command history forward and backward.  Lastly is a Return key, which I didn’t really use, since the iPad keyboard has one built in.

The upper right corner of the app replicates many of the same keys as the collapsible keyboard, along with a paper clip icon.  When you tap this, it pulls out a drawer that contains the contents of the clipboard.  You can paste those contents directly onto the command line.  So if you find yourself typing the same commands in over and over, this is a handy shortcut (there are others we’ll get to in a second).  As a quick note, while you can type in this clipboard, if you don’t copy the contents before pasting it will simply paste what was in the box before.  So be sure to copy before you paste.

The upper left includes the Settings button, the session button, the keyboard show/hide button, a button to show/hide the collapsible keyboard with the TAB and CTRL keys, and a file drawer for storing config files.  The settings button is very feature rich. You can choose to have the program automatically connect when it launches or wait for you to connect manually.  There are also settings to change the baud rate and stop bits, which really helps when you are connecting to some non-standard gear.  You can have the system log all of your console sessions, which can be stored in the filing cabinet for later examination.  You can change the number of columns and rows, as well as the amount of scrollback in the window.  Be aware that adding too many columns will mean you need to scroll the screen left or right to see the output, as it looks like the main window is about 80 columns wide.  You can change the bell that dings when you do something you aren’t supposed to, as well as changing the color scheme to something other than white-on-black text.  The font size slider doesn’t correspond to actual point sizes, so you might need to play around with it to find a comfortable setting.

The session button allows you to disconnect a console session manually as well as offering one of the added benefits of this program.  By signing up at, you can add an option under settings to connect to a remote console server at that website.  You can then tap the session button and obtain a 7-digit access code that allows someone to access your console session from the Get Console webpage.  This is fairly handy if you have a junior administrator on site and need to walk them through a configuration.  Or if that same junior admin is in a network that is down, you can use a 3G iPad to connect to their console session and do some troubleshooting.  I had to play around with the settings in order to test this feature.  It looks like the app connects to the remote console server when you choose to share the session, and the access code allows the user on the website to connect in like a type of reverse telnet connection.  I couldn’t get the app to connect using the North America servers, but the Europe and Asia servers worked just fine.  However, the latency on these connections was pitiful.  Redraw on my screen could be measured in seconds.  I tried entering some commands on the webpage, but careful typing was enough to overrun the keyboard buffer for the app.  And if you’re going to try and look at live debugs, you might as well forget about it.  By the time you could send a break or “un all”, you’d be swamped in messages.  Better to use the web app as a mirroring device for training or for simple troubleshooting.  You can also choose to encrypt the sessions if you want, which is a pretty good idea if you don’t want everyone on the Internet up in your business.

The filing cabinet is another interesting piece.  By uploading configs to the Get Console website, you can store them in your filing cabinet to copy onto the device locally.  That way, if you have a template for your switches, you don’t need to worry about copying and pasting it out of an e-mail, where it may get buggered up by some strange formatting issues.  You can also have those pesky junior admins share an account and copy the configs to the filing cabinet for them, so all they have to do is walk out and plug in to setup the switch with enough config for you to be able to telnet to it.  There is local shortcut storage as well, so you can keep some of your more clever commands on your own iPad safe from those that could use them to do harm.  You can also store console logs for later upload or email.

Out of the box, the font size was downright tiny.  I had to bump the slider up to about 3/4ths of the way just to read it comfortably, and I was holding the iPad less than a foot from my face.  The keyboard was quite responsive, and the scrolling of the information was smooth and easy to follow.  The app is setup to beep at you when you try to use a key that isn’t supported, such as a down arrow at the prompt when there are no more commands to replay.  This feature is nice because it gives some feedback so you know when you’re beating your head against a brick wall.

In case you’re curious, this app is universal for both iPad and iPhone/iPod Touch.  But other than just glancing at the console I’m not sure how useful it’s going to be.  There isn’t much screen real estate to start with, and all the extra pieces don’t give you much room to look at things.  Here’s a screen shot to give you and idea of what I’m talking about:

Tom’s Take

It all comes down to money.  Is there enough utility in this cable and app for you to justify spending $100 on it?  Do you often find yourself in a network room with only your iPad and a switch that won’t respond to any other method of input?  I wouldn’t dream of trying to do any kind of heavy duty debugging on this device.  I’d rather have my full laptop with multiple apps and notepad windows to drag around to interpret console spam.  As well, any kind of programming that would require lots of time at the keyboard would probably get uncomfortable after a while, unless you’re one of those people that happens to like typing on the iPad on-screen keyboard.  I suppose you could haul along a wireless keyboard, but at that point you’re dragging along an awful lot of devices for simple console access.

I could see this being a useful tool for training or for an emergency tool kit.  Throw an iPad and a cable in your kit and you have instant access to the console of a device from anywhere in the world.  You could send the less-skilled network admins out on site and a more senior person could stay in the office and do some simple troubleshooting or configuration in order to get to the equipment through SSH or telnet.  The web piece, in my mind, is just too unresponsive to spend a lot of time on.  Plus, if you are fast typist like I am, you’re going to get rather frustrated with the delay in command execution, if you don’t outright lock the system up with all the characters you’re throwing at it.

The app does what it says, there’s no denying that.  I find it very useful to have on my iPad and I’ll probably use it going forward for many of my walkthroughs and audits.  However, I think the $100 price tag is a little steep for something like this.  I hope that the price of the console cable will come down at some point, because $69 dollars for this is a bit of a stretch, even by Apple standards.  If there is enough demand, we may even see some other vendors get into the market and offer something like this.  If that happens, hopefully the Get Console people will support them as well.  I had hoped that maybe the software people could offer a gift card with the purchase of the cable, but I believe that they are two different companies so that’s probably out of the question.  Redpark could always throw in a $10 iTunes gift card if the want to soften the blow of needing the additional app to use the cable, but marketing isn’t my department.

All in all, I think I’m going to be able to find some use out of this app.  However, you really need to think twice about whether or not a C-note is worth giving up for this type of functionality.  If you want to learn more about these products, you can check out the console cable at and you can check out the software program at

When Is A Trunk Not A Trunk?

When I was an impressionable youth back in my college years, I decided it might be a good idea to take Japanese as a foreign language.  I spent three semesters learning vocabulary and kanji and eventually managed to forget pretty much everything I learned.  One lesson that did stick with me, however, occurred in my first semester.  Our professor was explaining to us gaijins (Westerners) that we needed to be very careful about how we pronounced certain words.  Since words in Japanese are limited by a very small number of vowel sounds, there are many cases were words use the same sounds but have totally different meanings.  Such is the case with shujin. When pronounced as I have written it, it is the word for “husband”.  However, if you hold the “u” sound a little too long, as in shuujin, you instead have referred to that person as a “prisoner”.  As my professor explained, “Well, perhaps those two words really aren’t so different.”  In this case, a small difference in pronunciation can have a profound difference in meaning.

Another place where I see an issue similar to this is when someone starts asking questions about terminology differences between HP Networking (nee Procurve) and Cisco terminology on switches.  Often, these questions boil down to two major terminology differences based around one word: trunk.  It’s pronounced the same in both vendors world, but just like in Japanese, it can have a very different meaning depending on how it’s used.  Allow me to illustrate:

In Ciscoland, when I use the term trunk, I am referring to a port that carries multiple VLANs.  Trunk ports carry information about each of the VLANs on the switch in either Cisco’s ISL proprietary format or in the 802.1q vendor-neutral format.  Newer switches, like the 2960, have no support for ISL and will only form trunks using 802.1q, so I’ll use 802.1q for my examples.  Just keep in mind that if the switch doesn’t support ISL, you don’t need to configure the trunk encapsulation.  When these ports are designated as “trunks”, the frames are tagged with a special 802.1q header that indicates which VLAN they are a part of.  The only VLAN that is not tagged with an 802.1q header (by default) is the native VLAN.  On Cisco equipment, the default native VLAN for an 802.1q trunk is VLAN 1.  The behavior of Cisco IOS is to transmit information about all VLANs present on the switch over the trunk.  You can narrow this behavior through use of the switchport trunk allowed vlan command.  A sample Cisco trunk config might look like this:

Switch(config)#interface gig 0/1
Switch(config-int)#switchport trunk encapsulation dot1q
Switch(config-int)#switchport trunk allowed vlan 1,10,99
Switch(config-int)#switchport mode trunk

This configuration is sufficient to setup an 802.1q trunk and only allow VLANs 1, 10, and 99 to pass traffic on it.

In HPvania, the terminology used for a port that carries multiple VLANs is a tagged port.  On an HP switch, the individual ports are rarely configured directly.  Instead, the VLAN itself is configured and the ports are added to the VLAN configuration.  In order to setup an access port, it is configured as an untagged member of the VLAN it needs to belong to.  Since HP does not support ISL trunking, the terminology is straight from 802.1q.  The untagged ports do not carry 802.1q headers that specify they VLAN information.  This would be known as an “access port” on a Cisco switch.  In order to create a port that carries information for multiple VLANs, we must “tag” those VLANs on that port.   This modifies the packets sent on that port to carry VLAN tags for the VLANs indicated.  A sample configuration for an HP switch connected to the above Cisco switch might look like this:

Switch(config)#vlan 1
Switch(config-vlan)#untagged 1
Switch(config-vlan)#untagged 48
Switch(config-vlan)#vlan 10
Switch(config-vlan)#tagged 48
Switch(config-vlan)#vlan 99
Switch(config-vlan)#tagged 48

Notice that the configuration is done under the VLAN and references the port number on the switch.  Access ports are untagged members of a particular VLAN, and the native VLAN of an 802.1q trunk is also an untagged member.  Even though 802.1q has a native VLAN that doesn’t carry a tag, on an HP switch this VLAN needs to be explicitly set.  The other VLANs must be tagged to the uplink port in order for their information to be carried between switches.  This lends itself to being an additive solution, where VLANs are only present on a trunk if they have been specifically configured.  There is no need to prune VLANs or exclude them from the trunk if they aren’t needed.  They just aren’t configured in the first place.

Which method is better?  This tends to devolve into an OSPF vs. IS-IS type of argument.  If you are more comfortable with one you tend to prefer it.  I tend to use the Cisco terminology in my day-to-day operations, and I have a slight preference for not needing to remember to add each individual VLAN to an uplink port.  However, from a security perspective, I do like the HP idea of only adding VLANs that are needed.  In fact, this same concept of VLAN creation is present in the Force10 OS.  If you’d like to see more of it in action, you should check out Stretch’s excellent intro to Force10 over on

So, if HP refers to an uplink carrying multiple VLANs are a tagged port, then does HP have a “trunk”?  In fact they do.  In HPvania, a trunk is a logical construct that aggregates multiple ports into one logical link.  For those of you that might be out there scratching your heads about this one, this means that when you “trunk” a group of ports on an HP switch, you are creating one LACP link from up to four individual ports.  This kind of configuration should look like this:

Switch(config)#trunk 19-24
Switch(config-trk)#vlan 1
Swtich(config-vlan)#untagged trk1
Swtich(config-vlan)#vlan 10
Swtich(config-vlan)#tagged trk1
Swtich(config-vlan)#vlan 99
Swtich(config-vlan)#tagged trk1

Those of you that are fans of irony will appreciate that the above config sets up this LACP port aggregation to pass multiple VLANs to another switch.  In other words, we are configuring a Cisco “trunk” on top of an HP “trunk”.

In Ciscoland, the idea of aggregating multiple ports into a grouping is referred to by many names, usually “port channeling” or “Etherchanneling”.  The latter is the term that usually describes the Cisco-proprietary Port Aggregation Protocol (PAgP) links that are created by default.  However, this term has more or less become genericized and is used to refer to any group of aggregated ports on a Cisco switch.  Cisco does support LACP on aggregated ports, so let’s see how we’d configure this switch to use LACP and send tagged VLAN traffic back to the HP switch:

Switch(config)#interface range gi 0/19-24
Switch(config-int-range)#switchport trunk encapsulation dot1q
Switch(config-int-range)#switchport trunk allowed vlan 1,10,99
Switch(config-int-range)#switchport mode trunk
Switch(config-int-range)#channel-group 1 mode active
Switch(config-int-range)#channel-protocol lacp

This will set the aggregated ports to use LACP and pass VLANs across the link with 802.1q tags.  Note that you must set the channel-group command to “active” in order to use LACP on the link.  If you set it to “auto” or “desirable”, it will use PagP by default.  If you set the mode to “on”, it will not use LACP or PAgP at all.

There you have it.  The terminology is different, but as long as you know what you are trying to accomplish, you can usually figure out what you need to configure in order to make it all work correctly.  Hopefully you’ll never find yourself married to any particular configuration, much less become a prisoner of it.  In the end, just remember that a trunk is still just a trunk.  The meaning is entirely up to you.

Party like it’s 1993!

I was once told by a consultant that he could figure out whether a client needed his services after about 30 seconds at a router console.  When I asked how he could be so sure after such a short amount of time, he consoled to a router in his lab and typed in “show clock”.  The lab router returned the now-familiar string of Mar 1, 1993 (he’d just booted it).  As soon as he showed me that one command, it all made sense.  Keeping accurate time in a computer environment is very important to systems.  Directory structures depend on an accurate clock to authorize logins and track audit events.  Novell and Windows both utilize systems to ensure that the clocks of all the servers in the network are synchronized.  And if those clocks drift out of sync, heaven help the server admins.

What about network equipment?  In days past, the routers and switches were typically neglected when it came to clock setting.  In fact, most older Cisco routers didn’t even include an on-board battery to keep the clock accurate.  And when the router would reboot, the software clock didn’t have an accurate hardware clock to refer to, so it used 00:00 1 Mar 1993 as the reference point.  But as systems have increased in complexity over the past several years, the need to have all the time on your equipment accurate has become paramount.  When debugging a call hand-off on a voice gateway, an accurate router clock ensures that you can match the time the call was placed with the debug message output.  If there is a security incursion into your network devices, you need to track the time the device was accessed in order to be able to accurately report the event to the proper authorities.

So how do we get our network clocks to report the right time?  Well, the process is fairly easy, provided you’ve done a little homework about a couple of things.  First, you need to know what timezone you are in and what your GMT offset is.  In today’s world, it is far easier to keep the clock of your device synced to GMT, then apply an offset to show you what the local time is.  That way, if you have devices spread all over the world, you never have to worry about a significant time difference because one clock was synced to local time and the other was synced to GMT with an offset applied.  For the purposes of the examples in this post, I’m going to assume the router is located in the central United States and is in the Central Time Zone.  The fact that I myself am in the Central Time Zone and therefore would not need to do any additional thinking to write my examples is purely coincidental.

In the case of my example, the central United States is in the Central Standard Time Zone (CST).  CST is six hours behind the GMT clock (GMT -6).

When you first connect to the router, you will either see the default time of 00:00 1 Mar 1993 (for older routers), or if you’re on an ISR or newer router, it may be synced fairly close to the actual time.  Starting with the ISR, Cisco started keeping the time synced more closely when the router was shipped from the factory, and the battery keeping the hardware clock time when powered off seemed to last a lot longer.

At this point, we need to set the router’s timezone with this command:

R1(config)# clock timezone <name> <GMT offset>
R1(config)# clock timezone CST -6

Once you’ve done that, the system should update with a message telling you that the current time zone has been updated.  Now, the router should know what time zone it’s in and adjust the clock offset appropriately.  You should also set the daylight savings time conditions as well.  For those not familiar, DST is a law that many countries have adopted that force sleepy engineers to reset the clock on the microwave at 2 a.m. on two days during the year.  Just when we had it down, they went and changed it a couple of years ago.  Hence the reason that Cisco doesn’t hard-code the DST settings on their routers.  They are more than happy to let you do it yourself.  In this example, we’re using the U.S. standard of the second Sunday in March and the first Sunday in November:

R1(config)# clock summer-time <name> recurring <week number start> 
        <day> <month> <time to start> <week number end> <day> <month> <time to end>
R1(config)# clock summer-time CDT recurring 2 Sun Mar 2:00 1 Sun Nov 2:00

And just like that, your router will perform just like every other smart device in your house and reset the clock for DST when necessary.  Now, if I could just get my microwave to do that.

Now that your router is in the right time zone, it’s probably a good idea to sync the clock to some kind of external time source.  That’s why Network Time Protocol (NTP) was created.  It allows distributed systems to sync their clocks with a time source directly connected to an atomic clock (the most precise kind).  NTP form as a hierarchy through use of strata. Stratum 0 NTP devices are directly connected to atomic clock sources, usually through RS-232 or other cabling.  The are not connected via a network.  Stratum 1 NTP devices are connected to Stratum 0 devices via a network connection.  These are the systems that are usually referred to as time servers.  These are usually the devices that you will sync one of your clocks to.  Why just one?  Well, while having all your devices synced to external time sources is a great idea, there are two issues that can arise.  First, having that much NTP traffic exiting your network isn’t exactly optimal.  There is no reason for 20 routers to each poll an external NTP server when one router could do the same thing, and the other 19 routers poll the synced router.  You can even designated your NTP-synced router as a Stratum 2 or 3 device if you’d like, then have it start serving time to your other devices.  The second issue with using all external NTP servers deals with security.  Some people are uncomfortable with the idea that all their devices are being told how to manage their clocks by an external device over which they have no control.  By configuring your devices to sync to one clock in your network that you control, there is more consistency and less reliance on systems outside your control that may be overloaded or unreliable.  For more info on what happens when someone starts abusing an NTP server, check out what NETGEAR did to the University of Wisconsin’s NTP server here.

For the sake of your sanity, it’s best to set the clock to something close to the actual time before you set the NTP server.  The reason for this is due to the propensity for NTP to look at your clock and declare it insane if you are too far drifted from the actual time.  I usually try to get my local clock somewhere in the neighborhood of one hour from the actual time, but the closer you are and the faster NTP will sync.  You do this with the following command (note the context)

R1#clock set <HH:MM:SS> <1-31> <Name of Month> <Year>
R1#clock set 01:00:00 10 Jan 2011

You must manually set the clock in enable mode, not global config mode.  Once you’ve gotten the clock close to your time, you can set the NTP server.  If you want to sync your router to an external time source, using the NIST Time Server list is always a good start.  If your router is capable of resolving DNS, a better idea is to use the NTP pool service.  This service is a cluster of NTP servers around the world that is served by round-robin DNS query, so no one server can get overloaded.  If you don’t feel comfortable syncing your clock to a server in France or Japan, you can always narrow down the focus of your query by using narrower DNS entries.  Check out their site for ways to do this.  Once you’ve figured out  how you are going to connect to NTP, and whether you are going to use external or internal servers, the command is pretty easy:

R1(config)# ntp server <name or ip>
R1(config)# ntp server
R1(config)# ntp server

Yep, that’s it.  Simple once you’ve gotten all the planning down.  You can check your NTP sync status with the command show ntp status.  You can see which time servers your are polling with the command show ntp associations.  The commands have a lot of good info, but can be a little cryptic your first couple of trys.

Once you’ve gotten your clock in sync, there’s just two more commands you need to use to ensure that all your logs and debug outputs are using the right clock.  By default, logs and debugs use the system uptime for their output, so you could get a log message that says “1w3d” instead of the real time.  And if you have to start doing math on your debugs to figure out when a call was dropped, you’re going to be a cranky rock star.  From global config mode, go ahead and type in these two commands:

R1(config)# service timestamps log datetime msec
R1(config)# service timestamps debug datetime msec

The “MSEC” keyword on the end tracks the message down to the millisecond level, which I’ve always found very handy when trying to figure out sub-second error messages.  It also helps better match error logs which are all part of the same event, but spread out over multiple messages.

Now, your routers are all in sync and your logs and debugs are all outputting the correct time.  If a consultant tries to access your devices, you will appear to be one cool customer that doesn’t need any help on your networking devices.  You’ll also be able to troubleshoot faster and get all the fame and wealth appropriate to your station as the Network Miracle Worker.

For what it’s worth, I did a lot of searching about why the default time on a Cisco router without a battery backup is March 1, 1993.  No one seemed to have a definitive answer.  According to the Internet, the only really exciting thing that happened that day was George Steinbrenner being reinstated as Yankees owner.  I doubt anyone in San Jose is a Yankees fan, so I didn’t think that was it.  It also didn’t correspond to any neat numbers in the Unix Time Epoch.  1993 was the first year that Cisco acquired a company, but that happened in September of that year.  The only thing that happened early was the release of IOS 10.0.  I guess that Cisco decided that “10” was an important enough number that they wanted to start basing recent history sourced from this date.  The other possibility is that it was just an arbitrary date chosen by IOS engineers so that the router clock didn’t cycle all the way back to 1900.  MS-DOS had a similar functionality, wherein it would see the system BIOS clock set to 1900 and assume that the BIOS had to be wrong.  It would then set the clock to the earliest date it could (January 1, 1980).  Maybe Cisco just decided that 1993 was so early that if they noticed a router clock stuck on that date, it would just be assumed the the clock was not set.  Either than, or they were really big fans of this song…