Category Archives: Configuration

OpenConfig and Wi-Fi – The Winning Combo

Posted on by
1

Wireless isn’t easy by any stretch of the imagination. Most people fixate on the spectrum analysis part of the equation when they think about how hard wireless is. But there are many other moving parts in the whole architecture that make it difficult to manage and maintain. Not the least of which is how the devices talk to each other.

This week at Aruba Atmosphere 2019, I had the opportunity to moderate a panel of wireless and security experts for Mobility Field Day Exclusive. It was a fun discussion, as you can see from the above video. As the moderator, I didn’t really get a change to explain my thoughts on OpenConfig, but I figured now would be a great time to jump in with some color on my side of the conversation.

Yin and YANG

One of the most exciting ideas behind OpenConfig for wireless people should be the common YANG data models. This means that you can use NETCONF to have a common programming language against specific YANG models. That means no more fumbling around to remember esoteric commands. You just tell the system what you want it to do and the rest is easy.

As outlined in the video, this has a huge impact on trying to keep configurations standard across different types of APs. Imagine the nightmare of trying to configure power settings or radio thresholds with 3 or more AP manufacturers in your building. Now, imagine being able to do it across your building or dozens of others with a few simple commands and some programming know-how? Doesn’t seem quite as daunting now, does it? It’s easy because you’re speaking the same language across all those APs.

So, what if you don’t care, like Richard McIntosh (@802TopHat) points out? What if your vendor doesn’t support OpenConfig? Well, that’s fine. Not everyone has to support it today. But if you work on building a model system and setting up the automation and API interfaces, are you just going to throw it out the window during your refresh cycle because the new APs that you’re buying don’t support OpenConfig? Or is the need for OpenConfig going to be a huge driver for you and part of the selection process.

Companies are motived by their customers. If you tell them that they need to develop OpenConfig for their devices, they will do it because they run the risk of losing sales. And if the industry moves toward adopting a standard northbound API, what happens to those that get left out in the cold after a few missed refresh cycles? I bet they’ll quickly realize the lost opportunities more than cover the development costs of supporting OpenConfig.

Telemetry Short-Cuts

The other big piece of OpenConfig and wireless is telemetry. SNMP-based monitoring doesn’t work well in today’s wired networks and it’s downright broken in wireless. There are too many variables out there in the average deployment to be able to account for them with anything other than telemetry. Many vendors are starting to adopt the idea of streaming the data directly to collectors via a subscription model. OpenConfig makes this easy with the ability to subscribe to the data you want using OpenConfig models.

From a manufacturer perspective, this is a huge chance to get into telemetry and offer it as a differentiator. If you’re not tied to using an archaic platform with proprietary data models you can embrace OpenConfig and deliver a modern telemetry infrastructure that users will want to adopt. And if the radio performance is the same between all of the offerings, telemetry could be a the piece that tips the scales in your favor.

Single-Vendor Isn’t So Single

I remember doing a deployment for a wireless system once that was “state of the art” when we put it in. I had my challenges and made everything work well and the customer was happy. Until a month later when the supporting vendor announced they were buying a competing company and using that company as their offering going forward. My customer was upset, naturally, but so was I. I spent a lot of time working out how to build and deploy that system and now it was on the scrap heap.

It’s even worse when you keep buying from single vendors and suddenly find that the new products don’t quite conform to the same syntax or capabilities. Maybe the new model of router or AP has a new board that is 95% compatible with the old one, except of that one command you use all the time.

OpenConfig can change that. Because the capabilities of the device have to be outlined you can easily figure out if there are any missing parts and pieces. You can also be sure that your provisioning scripts and programs aren’t going to break or cause problems because a critical command was deprecated. And since you can keep the models around for old hardware as well as new you aren’t going to be duplicating efforts everywhere trying to keep things moving between the platforms.

Tom’s Take

OpenConfig is a great idea for any system that has distributed components. Even if it never takes off in Wi-Fi, it will force the manufacturers to do a bit better job of documenting their capabilities and making them easy to consume via API. And anything that exposes more functionality to be consumed by automation and programmability is a good thing indeed.

It’s The Change Freeze Season

Posted on by
1

Everyone’s favorite time of the year is almost here! Is it because it’s the holiday season? Perhaps it’s the magic that happens at the end of the year? Or maybe, it’s because there’s an even better reason to get excited!

Change Freeze Season!

That’s right. Some of you reading this started jumping up and down like Buddy the Elf at the thought of having a change freeze. There’s something truly magical about laying down the law about not touching anything in the system until after the end-of-year reports are run and certified. For some, this means a total freeze of non-critical changes from the first of December all the way through the New Year until maybe even February. That’s a long time to have a frozen network? But why?

The Cold Shoulder

Change freezes are an easy thing to explain to the new admins. You simply don’t touch anything in the network during the freeze unless it’s broken. No tweaking. No experimenting. No improvements. Just critical break/fix changes only. There had better be a ticket. There should be someone yelling that something’s not right. Otherwise you’re in for it.

There are a ton of reasons for this. The first is something I remember from my VAR days as Boredom Repellent. When you find yourself at the end of year with nothing to do, you tend to get bored. After you’ve watched Die Hard for the fifteenth time this year you decide it’s time to clear out your project backlog. Or maybe you’ve been doing some learning modules instead. You find a great blog post from one of your favorite writers about a Great Awesome Amazing Feature That Will Save You Days Of Work If You Just Enable This One Simple Command!

In either case, the Boredom Repellent becomes like pheromones for problems. Those backlogged projects take more time than you expected. That simple feature you just need to enable isn’t so simple. It might even involve an entire code upgrade train to enable it. Pretty soon you find yourself buried in a CLI mess with people screaming about very real downtime. Now, instead of being bored you’re working until the wee hours of the night because of something you did.

The second reason for change freezes at the end of the year is management. You know, the people that call and scream at you as soon as their email appears to be running slow. The people that run reports once a month at 6:00pm and then call you because they get a funny warning message on their screen. Those folks. Guess what? End-of-year is their time to shine in all their glory.

This is usually the time they are under the most stress. Those reports have to be reprinted. All the financials from the year need to be consolidated and verified. The taxes will need to be paid. And all that paperwork and pressure adds up to stress. The kind of stress that makes any imperfections in the network seem ten times more important than before. Report screen not show success within 10 ms? Problem. Printer run out of yellow toner? Network problem. Laptop go to sleep while someone went to lunch and now the entire report is gone? Must be your problem. And guess who gets to work around the clock to solve it with someone bearing down on them from on high?

Don’t Let It Go

The fact is that we can’t have people doing things in the network without tracking those changes back to reasons. That applies for adventurous architects wanting to squeeze out the last ounce of performance from that amazing new switch. And it goes double for the CFO demanding you put his traffic into AF41 so it gets to the server faster so his reports don’t take six hours to print.

It all comes back to the simple fact that we have no way to track changes in our network and we have no way of knowing what will happen when we make one live. It feels an awful lot like this GIF:

Crazy, right? Yet every time we hit the Enter key, we are amazed at the results. Even for “modern” OSes with sanity checking, like Junos or IOS-XR, you have no way of knowing if a change you make on one device somewhere in the branch is going to crash OSPF or BGP for the entire organization. And even if there was a big loud warning popup that said, “ALERT: YOU ARE GOING TO BREAK EVERYTHING!!!”, odds are good we would just click past it.

Network automation and orchestration systems can prevent this. They can take the control of change management out of the hands of bored engineers and wrap it in process and policy. And if the policy says Change Freeze then that’s what you get. No changes. Likewise, if there is a critical need, like patching out a backdoor or something, that policy can be overridden and noted so that if there is a bug eight months from now in that code train that causes issues you can have documentation of the reason for the change when someone comes to chew you out.

Likewise, there are other solutions out there that try to prototype the entire network to figure out what will happen when you make a change. Companies like Forward Networks and Veriflow can prototype your network in a model that can assess the impact of a change before you commit to it. It’s the dream of a bored engineer because you can run simulations to your heart’s content to find out if two hours of code upgrades will really get you that 2% performance increase promised in that blog post. And for the CFO/CEO/CIO screaming at you to prioritize their traffic, these solutions can remind them that most of their traffic is Youtube and Spotify and having that at AF41 will cause massive issues for them.

What’s important is that you and the rest of the team realize that change freezes aren’t a solution to the problem of an unstable network. Instead, they are treating the symptoms that crop up from the underlying disease of the network not being a deterministic system. Unlike some other machines, networks run just fine at sub-optimal performance levels. You can make massive mistakes that will live in a network for years and never show their ugly face. That is, until you make a small change that upsets equilibrium and causes the whole system to fail, cascade style, and leave you holding the keyboard as it were.

Tom’s Take

I both love and hate Change Freeze season. I know it’s for the best because any changes that get made during this time will ultimately result in long hours at work undoing those changes. I also know that the temptation to experiment with things is very, very strong this time of year. But I feel like Change Freeze season will soon go the way of the aluminum Christmas tree when we get change management and deterministic network modeling systems in place to verify changes on a system-wide basis and not just sanity checking configs at a device level. Tracking, prototyping, and verification will solve our change freeze problems eventually. And that will make it the most wonderful time of the year all year long.

On Old Configs and Automation

Posted on by
2

I used to work with a guy that would configure servers for us and always include an extra SCSI card in the order. When I asked him about it one day, he told me, “I left it out once and it delayed the project. So now I just put them on every order.” Even after I explained that we didn’t need it over and over again, he assured me one day we might.

Later, when I started configuring networking gear I would always set a telnet password for every VTY line going into the switch. One day, a junior network admin asked me why I configured all 15 instead of just the first 5 like they learn in the Cisco guides. I shrugged my shoulders and just said, “That’s how I’ve always done it.”

The Old Ways

There’s no more dangerous phrase than “That’s the way it’s always been.”

Time and time again we find ourselves falling back on the old rule of thumb or an old working configuration that we’ve made work for us. It’s comfortable for the human mind to work from a point of reference toward new things. We find ourselves doing it all the time. Whether it’s basing a new configuration on something we’ve used before or trying to understand a new technology by comparing it to something we’ve worked on in the past.

But how many times have those old configurations caused us grief? How many times have we been troubleshooting a problem only to find that we configured something that shouldn’t have been configured in the way that it was. Maybe it was an old method of doing hunt groups. Or perhaps it was a legacy QoS configuration that isn’t supported any more.

Part of our issue as networking professionals is that we want to concentrate on the important things. We want to implement solutions and ideas that work for our needs and not concentrate on the minutia of configuration. Sure, the idea of configuration a switch from bare metal to working config is awesome the first time you do it. But the fifteenth time you have to configure one in a row is less awesome. That’s why copy-and-paste configurations are so popular with people that just want to get the job done.

New Hotness

This idea of using old configurations for new things takes even more importance when you start replacing the boring old configuration methods with new automation and API-driven configuration models. Yes, APIs make it a lot easier to configure a switch programmatically. And automation tools like Puppet and Ansible make it much faster to bring a switch online from nothing to running in the minimum amount of time.

However, even with this faster configuration method, are we still using old, outdated configurations to solve problems? Sure, I don’t have to worry about configuring VLANs on the switch one at a time. But if my configuration is still referencing VLANs that are no longer in the system that makes it very difficult to keep the newer switches running optimally. And that’s just assuming the configuration is old and outdated. What if we’re still using deprecated commands?

APIs are great because they won’t support unsupported things. But if we don’t scrub the configuration now and then to remove these old lines and protocols we’ll quickly find ourselves in a world of trouble. Because those outdated and broken things will bring the API to a halt. Yes, the valid commands will still be entered correctly, but if those valid commands rely on something invalid to work properly you’re going find things broken very fast.

What makes this whole thing even more irritating is that most configurations need to be validated and approved before being implemented. Which makes the whole process even more difficult to manage. Part of the reason why old configs live for so long is that they need weeks or months of validation to be implemented effectively. When new platforms or configuration methods crop up it could delay new equipment installation. This sometimes leads to people installing new gear with “approved” configs that might not be the best fit in order to get that new equipment into service.

Tom’s Take

So, how do we fix all this? What’s the trick? Well, it’s really a combination of things. We need to make sure we audit configs regularly to keep the old stuff from continuing on past the expiration dates. We also need to continually resubmit new configurations to the approvals process. Just like disaster recovery documentation, configurations are living, breathing documents that should always be current and refreshed. The more current your configurations, the less likely you are to have old cruft keeping your systems running at subpar performance. And the less likely you are to have to worry about breaking new things like APIs and automation systems in the future.

IOS X-Treme!

Posted on by
8

IOSXtreme

As a nerd, I’m a huge fan of science fiction. One of my favorite shows was Stargate SG-1. Inside the show, there was a joke involving an in-universe TV program called “Wormhole X-Treme” that a writer unintentionally created based on knowledge of the fictional Stargate program. Essentially, it’s a story that’s almost the same as the one we’re watching, with just enough differences to be a totally unique experience. In many ways, that’s how I feel about the new versions of Cisco’s Internetwork Operating System (IOS) that have been coming out in recent months. They may look very similar to IOS. They may behave similarly to IOS. But to mistake them for IOS isn’t right. In this post, I’m going to talk about the three most popular IOS-like variants – IOS XE, IOS XR, and NX-OS.

IOS XE

IOS XE is the most IOS-like of all the new IOS builds that have been released. That’s because the entire point of the IOS XE project was to rebuild IOS to future proof the technology. Right now, the IOS that runs on routers (which will henceforth be called IOS Classic) is a monolithic kernel that runs all of the necessary modules in the same memory space. This means that if something happens to the routing engine or the LED indicator, it can cause the whole IOS kernel to crash if it runs out of memory. That may have been okay years ago but today’s mission critical networks can’t afford to have a rogue process bringing down an entire chassis switch. Cisco’s software engineers set out on a mission to rebuild the IOS CLI on a more robust platform.

IOS XE runs as a system daemon on a “modern Linux platform.” Which one is anyone’s guess. Cisco also abstracted the system functions out of the main kernel and into separate processes. That means that if one of them goes belly up it won’t take the core kernel with it. One of the other benefits of running the kernel as a system daemon is that you can now balance the workload of the processes across multiple processor cores. This was one of the more exciting things to me when I saw IOS XE for the first time. Thanks to the many folks that pointed out to me that the ASR 1000 was the first device to run IOS XE. The Catalyst 4500 (the first switch to get IOS XE) is using a multi core processor to do very interesting things, like the ability to run inline Wireshark on a processor core while still letting IOS have all the processor power it needs. Here’s a video describing that:

Because you can abstract the whole operation of the IOS feature set, you can begin to do things like offer a true virtual router like the CSR 1000. As many people have recently discovered, the CSR 1000 is built on IOS XE and can be booted and operated in a virtualized environment (like VMware Fusion or ESXi). The RAM requirements are fairly high for a desktop virtualization platform (CSR requires 4GB of RAM to run), but the promise is there for those that don’t want to keep using GNS3/Dynamips or Cisco’s IOU to emulate IOS-like features. IOS XE is the future of IOS development. It won’t be long until the next generation of supervisor engines and devices will be using it exclusively instead of relying on IOS Classic.

IOS XR

In keeping with the sci-fi theme of this post, IOS XR is what the Mirror Universe version of IOS would look like. Much like IOS XE, IOS XR does away with the monolithic kernel and shared memory space of IOS Classic. XR uses an OS from QNX to serve as the base for the IOS functions. XR also segments the ancillary process in IOS into separate memory spaces to prevent system crashes from an errant bug. XR is aimed at the larger service provider platforms like the ASR 9000 and CRS series of routers. You can see that in the way that XR can allow multiple routing protocol processes to be executed at the same time in different memory spaces. That’s a big key to the service provider.

What makes IOS XR so different from IOS Classic? That lies in the configuration method. While the CLI may resemble the IOS that you’re used to, the change methodology is totally foreign to Cisco people. Instead of making live config changes on a live system, the running configuration is forked into a separate memory space. Once you have created all the changes that you need to make, you have to perform a sanity check on the config before it can be moved into live production. That keeps you from screwing something up accidentally. Once you have performed a sanity check, you have to explicitly apply the configuration via a commit command. In the event that the config you applied to the router does indeed contain errors that weren’t caught by the sanity checker (like the wrong IP), you can issue a command to revert to a previous working config in a process known as rollback. All of the previous configuration sets are retained in NVRAM and remain available for reversion.

If you’re keeping track at home, this sounds an awful lot like Junos. Hence my Mirror Universe analogy. IOS XR is aimed at service providers, which is a market dominated by Juniper. SPs have gotten very used to the sanity checking and rollback capabilities provided by Junos. Cisco decided to offer those features in an SP-specific IOS package. There are many that want to see IOS XR ported from the ASR/CSR lines down into more common SP platforms. Only time will tell if that will happen. Jeff Fry has an excellent series of posts on IOS XR that I highly recommend if you want to learn more about the specifics of configuration on that platform.

NX-OS

NX-OS is the odd man out from the IOS family. It originally started life as Cisco’s SAN-OS, which was responsible for running the MDS line of fibre channel switches. Once Cisco started developing the Nexus switching platform, they decided to use SAN-OS as the basis for the operating system, as it already contained much of the code that would be needed to allow networking and storage protocols to interoperate on the device, a necessity for a converged data center switch. Eventually, the new OS became known as NX-OS.

NX-OS looks similar to the IOS Classic interface that most engineers have become accustomed to. However, the underlying OS is very different from what you’re used to. First off, not every feature of classic IOS is available on demand. Yes, a lot of the more esoteric feature sets (like the DHCP server) are just plain unavailable. But even the feature sets that are listed as available in the OS may not be in the actual running code. You need to active each of these via use of the feature keyword when you want to enable them. This “opt in” methodology ensures that the running code only contains essential modules as well as the features you want. That should make the security people happy from an exploit perspective, as it lowers the available attack surface of your OS.

Another unique feature of NX-OS is the interface naming convention. In IOS Classic, each interface is named via the speed. You can have Ethernet, FastEthernet, GigabitEthernet, TenGigabit, and even FortyGigabit interfaces. In NX-OS, you have one – Ethernet. NX-OS treats all interfaces as Ethernet regardless of the underlying speed. That’s great for a modular switch because it allows you to keep the same configuration no matter which line cards are running in the device. It also allows you to easily port the configuration to a newer device, say from Nexus 5500 to Nexus 6000, without needed to do a find/replace operation on the config and risk changing a line you weren’t supposed to. Besides, most of the time the engineer doesn’t care about whether an interface is gigabit or ten gigabit. They just want to program the second port on the third line card.

Tom’s Take

No software program can survive without updates. Especially if it is an operating system. The hardware designed to run version 1.0 is never the same as the hardware that version 5.0 or even 10.0 utilizes. Everything evolves to become more efficient and useful. Think of it like seasons of sci-fi shows. Every season tells a story. There may be some similarities, but people overall want the consistency of the characters they’ve come to love coupled with new stories and opportunities to increase character development. Network operating systems like IOS are no different. Engineers want the IOS-like interface but they also want separated control planes, robust sanity checking, and modularized feature insertion. Much like the writers of sci-fi, Cisco will continue to provide new features and functionality while still retaining the things to which we’ve grown accustomed. However, if Cisco ever comes up with a hare-brained idea like the Ori, I can promise there’s no way I’ll ever run IOS-Origin.

What’s My Cisco ATA Second Line MAC Address?

Posted on by
1

In the world of voice, not everything is wine and roses.  As much as we might want to transition everything over to digital IP phones and soft clients, the fact remains that there are some analog devices that still need connectivity on a new phone system.  The more common offender of this is the lowly fax machine.  Yes, even in this day and age we still need to rely on the tried-and-true facsimile machine to send photostatic copies of documents across the PSTN to a waiting party.  Never mind email or Dropbox or even carrier pigeon.  Fax machines seem to be the most important device connected to a phone system.  Normally, I leave the fax connections and their POTS lines intact without touching anything.  However, there are times when I don’t have that luxury.

In the case of the Cisco VoIP systems, that means relying on the Analog Terminal Adapter, or ATA.  The ATA allows you to connect an analog device to the unit, whether it be a fax machine or a cordless analog phone or even a fire alarm or postage machine.  It has many uses.  The configuration of the ATA is fairly straightforward under any CUCM system.  However, if you have a multitude of analog devices that you need to connect, you might opt to use the second analog port on the ATA.  The ATA 186 of the past and its current replacement, the ATA 187, both have 2 analog ports on the back.  There’s only one Ethernet port, though.  This is where the interesting part comes in to play.  If there are two analog ports but only one Ethernet port, how to I configure the MAC address for the second port?  All phone devices in CUCM must be identified by MAC address.  On an ATA, the primary MAC address printed on the bottom or the side of the box is the address for the first port.

If you want to use the second port, you’re going to have to do a little bit of disassembly.  Cisco uses a standard method to create a new MAC address:

1.  Take the MAC address for port 1.  For example, 00:00:DE:AD:BE:EF.

2.  Drop the first two digits from the MAC address.  In the example, 00:DE:AD:BE:EF.

3.  Append “01” to the end of the 10-digit address.  Example, 00:DE:AD:BE:EF:01.

Once you’ve completed those steps, take the MAC address you’ve just created and plug it into CUCM as a new ATA device.  Once you’ve completed the necessary steps to create the new device, it will register with the DN you’ve assigned to it.  Then you can start calling or faxing it to your heart’s content.

There’s no mention of the secondary MAC address anywhere on the web interface.  You’d figure it wouldn’t be hard to write some HTML function to read the MAC address and do the above operation.  The Cisco documentation buries this information deep inside the setup document.  I’ve even search Cisco’s very own support forums and found all manner of advice that doesn’t work correctly.  I decided that it was time to jot this information down in a handy place for the next time I need to remember how to configure the ATA’s second port.  I hope you find it useful as well.

The Card Type Command – Don’t Flop

Posted on by
1

If you’ve ever found yourself staring at a VWIC2-1MFT-T1/E1 or a NM1-T3/E3 module, you know that you’ve got some configuration work ahead of you.  Whether it be for a PRI circuit to hook up that new VoIP system or a DS3 to get a faster network connection, the T1/T3 circuit still exists in many places today.  However, I’ve seen quite a few people that have been stymied in their efforts to get these humble interface cards connected to a router.  I have even returned a T1/E1 card myself when I thought that it was defective.  Imagine the egg on my face when I discovered that the error was mine.

It turns out that ordering the T1/E1 or T3/E3 module from Cisco requires a little more planning on the installation side of things.  These cards can have a dual identity because the delivery mechanism for these circuits is identical.  In the case of a T1/E1, the delivery mechanism is almost always over an unshielded twisted pair (UTP) cable.  Almost all of the T3/E3 circuits that I’ve installed have been delivered over fiber but terminated via coax cables with BNC connectors.  The magic, then, is in the location.  A T1 circuit is typically delivered in North America, while the E1 circuit is European version.  There are also differences in the specifics of each circuit.  A T1 is 24 channels of 64kbits each.  An E1 is 32 channels of the same size.  This means that a T1 has an effective data rate of 1.544 Mbits while an E1 is a bit faster a 2.048 Mbits.  There are also framing differences and a slightly different signaling structure.  The long and short of it is that T1 and E1 circuits are incompatible with each other.  So how does Cisco manage to ship a module that supports both circuit types?

The key is that you must choose which circuit you are going to support when you install the card.  The card can’t automatically flip back and forth based on circuit detection.  Where the majority of issues come from in my line of work is that the card doesn’t show up as a configurable interface until you force a circuit type.  This is accomplished by using the card type command:

RouterA(config)#card type ?
 e1 E1
 e3 E3
 t1 T1
 t3 T3

Choose your circuit type and away you go!  As soon as you enter the card type, the appropriate serial interface is created.  You will still need to enter the controller interface to set parameters like the framing and line code.  However, the controller interface only shows up when the card type has been set as well.  So unless you’ve done the first step, there isn’t going to be a place to enter any additional commands.

Tom’s Take

Sometimes there are things that seem so elementary that you forget to do them.  Checking a power plug, flipping a light switch, or even remembering to look for little blinking lights.  We don’t think about doing all the easy stuff because we’re concentrating on the hard problems.  After all our hard work, we know it has to be something really messed up otherwise it would be fixed by now.  In the case of T1/E1 cards, I made that mistake.  I forgot to check everything before declaring the card dead on arrival.  Now, I find myself spending a lot of time providing that voice of reason for others when they’re sure that it has to be something else.  The little voice of reason doesn’t always have to be loud, sometimes it just has to say something at the right time.

DST Configuration – Just In the Nick of Time

Posted on by
Reply

Today is the dreaded day in the US (and other places) when we must sacrifice an hour of our precious time to the sun deity so that he might rise again in the morning.  While this is great for being outdoors and enjoying the sunshine all the way into the late evening hours, it does wreak havoc on our networking equipment that relies on precise timing to let us know when a core dump happened or when that last PRI call came in when running debug isdn q931.  However, getting the right time running on our devices can be a challenge.  In this post, I will cover configuring Daylight Savings Time on Cisco, HP, and Juniper network equipment for the most pervasive OS deployments.  Note that some configurations are more complicated than others.  Also, I will be using Central Time (CST/CDT) for my examples, which is GMT -6 (-5 in DST).  Adjust as necessary for your neck of the woods.  I’m also going to assume that you’ve configured NTP/SNTP on your devices.  If not, read my blog post about it and go do that first.  Don’t worry, I’ll still be here when you get back.  I have free time.

Cisco

I’ve covered the basics of setting DST config on Cisco IOS before, but I’ll put it here for the sake of completeness.  In IOS (and IOS XR), you must first set the time zone for your device:

R1(config)# clock timezone <name> <GMT offset>
R1(config)# clock timezone CST -6

Easy, right?  Now for the fun part.  Cisco has always required manual configuration of DST on their IOS devices.  This is likely due to them being shipped all around the world and various countries observing DST (or not) and even different regions observing it differently.  At any rate, you must the clock summer-time command to configure your IOS clock to jump when needed.  Note that in the US, DST begins at 2:00 a.m. local time on the second Sunday in March and ends a 2:00 a.m. local time on the first Sunday in November.  That will help you decode this code string:

R1(config)# clock summer-time <name> recurring <week number start> <day> <month> <time to start> <week number end> <day> <month> <time to end>
R1(config)# clock summer-time CDT recurring 2 Sun Mar 2:00 1 Sun Nov 2:00

Now your clock will jump when necessary on the correct day.  Note that this was a really handy configuration requirement to have in 2007, when the US government decided to change DST from the previous requirement of the first Sunday in April at the start and the last Sunday in October to end.  With Cisco, manual reconfiguration was required, but no OS updates were needed.

HP (Procurve/E-Series and H3C/A-Series)

As near as I can tell, all HP Networking devices derive their DST settings from the OS.  That’s great…unless you’re working on an old device or one that hasn’t been updated since the last presidential administration.  It turns out that many old HP Procurve network devices still have the pre-2007 US DST rules hard-coded in the OS.  In order to fix them, you’re going to need to plug in a config change:

ProCurve(config)# time daylight-time-rule user-defined begin-date 3/8 end-date 11/1

I know what you’re thinking.  Isn’t that going to be a pain to change every year if the dates are hard-coded?  Turns out the HP guys were ahead of us on that one too.  The system is smart enough to know that DST always happens on a Sunday.  By configuring the rule to occur on March 8th (the earliest possible second Sunday in March) and November 1st (the earliest possible first Sunday in November), the system will wait until the Sunday that matches or follows that date to enact the DST for the device.  Hooray for logic!  Note that if you upgrade the OS of your device to a release that supports the correct post-2007 DST configuration, you won’t need to remove the above configuration.  It will work correctly.

Juniper

Juniper configures DST based on the information found in the IANA Timezone Database, often just called tz.  First, you want to get your device configured for NTP.  I’m going to refer you to Rich Lowton’s excellent blog post about that.  After you’ve configured your timezone in Junos, the system will automatically correct your local clock to reflect DST when appropriate.  Very handy, and it makes sense when you consider that Junos is based heavily on BSD for basic OS operation.  One thing that did give me pause about this has nothing to do with Junos itself, but with the fact that there have been issues with the tz database, even as late as last October.  Thankfully, that little petty lawsuit was sidestepped thanks to the IANA taking control of the tz database.  Should you find yourself in need of making major changes to the Junos tz database without the need to do a complete system update, check out these handy instructions for setting a custom timezone over at Juniper’s website.  Just don’t be afraid to get your hands dirty with some BSD commands.

Tom’s Take

Daylight Savings Time is one of my least favorite things.  I can’t see the advantage of having that extra hour of daylight to push the sunlight well past bedtime for my kids.  Likewise, I think waking up to sunrise is overrated.  As a networking professional, DST changes give me heartburn even when everything runs correctly.  And I’m not even going to bring up the issues with phone systems like CallManager 4.x and the “never going to be patched” DST issues with Windows 2000.  Or the Java issues with 79xx phones that still creep up to this day and make DST and confusing couple of weeks for those that won’t upgrade technology. Or even the bugs in the iPhone with DST that cause clocks to spring the wrong way or alarms to fail to fire at the proper time.  In the end though, network enginee…rock stars are required to pull out our magical bags and make everything “just work”.  Thanks to some foresight by major networking vendors, it’s fairly easy to figure out DST changes and have them applied automagically.  It’s also easy to change things when someone decides they want their kids to have an extra hour of daylight to go trick-or-treating at Halloween (I really wish I was kidding).  If you make sure you’ve taken care of everything ahead of time, you won’t have to worry about losing more than just one hour of sleep on the second Sunday in March.

I Have The Power! – Common Electrical Connectors

Posted on by
7

I’m no electrician.  In fact, the last time I tried to do some electrical work ended up with a minor electrocution and me not being able to taste for an hour.  However, in the IT world electricity is a key to our jobs.  The mightiest Nexus 7k or QFabric deployment can be brought to its knees by inadequate power.  The need to understand power and the the connectors that go along with it are vital.

Fabulous secrets were revealed to me the day I pulled up the Internet and started looking up the various codes for connectors that are used in electrical work.  I figured out pretty quickly that electricians had a language all their own and that I needed to learn how to speak some of it in order to get things accomplished.  After all, describing a connector as “one goes like this, the other goes like that” isn’t really helpful, especially over the phone.  I wanted to pass on a bit of what I learned to all of you.  A note for my international readers: this is going to be focused primarily on US connectors.  However, if you’d like to fly me to your country for a few days, I’ll be more than happy to bring a travel kit and research your electrical code from my hotel room.  Just a thought…

Some of these connectors standardized by the National Electrical Manufacturers Association (NEMA).  They also hold the trademark on the term “twist lock”, so when I use it, know that it belongs to them.  The others are standardized by the International Electrotechnical Commission (IEC) under IEC 60320.

NEMA 5

The NEMA 5-series plug and receptacles are the most common found in the US today.  They are three-wire circuits (hot, neutral, and ground) and are rated to carry a maximum of 125 volts, although they usually carry about 110 volts and are referred to as “110 circuits”. All of these connectors will start with a “5” and a dash, followed by the amperage of the circuit, from 15 amps to 30 amps.

5-15

By far the most common connector you’ll run into in the US.  A simple 110-125v circuit rated at 15 amps.  Whatever you do, don’t rip the ground plug out if you need to fit this into a NEMA 1-15 two prong outlet.  I’ve seen what happens there and it isn’t pretty.

5-20

The NEMA 5-20 connector is more common as you being to start using equipment with high wattage power supplies, like Catalyst 4500 switches.  I’ve always heard the 5-20 described as a “dedicated circuit” plug, but that’s not entirely accurate.  It’s a 110-125v 20 amp circuit.  It can easily be identified by the perpendicular blade.

L5-20

Here’s where the real fun starts.  This little jewel was the source of my first head scratching moment with NEMA connectors.  This plug is the same as its 5-20 cousin above, however the connectors look all wrong.  This connector is designed to be inserted into the receptacle and then rotated counterclockwise to lock it in.  This way, it can’t simply be pulled out by someone tripping over it or by vibrations from heavy machinery.  Do take care though, as tugging on the connector too hard will cause it to rip the whole receptacle assemble out of the wall with possibly some exposed wires.

L5-30

The NEMA L5-30 is the big boy of 110-125v circuits.  I found it curious that I have never really seen the non-locking version of this plug, but I’ve since been informed that the majority of devices that use 30 amp circuits prefer this connector to prevent it from being yanked out.  Uninterruptible Power Supplies (UPSs) use this connector frequently in order to ensure their devices are getting enough power.

NEMA 6

The NEMA 6 series connectors are the ones you use when you need to provide heavy duty power to a device.  These are typically 208 volt or 240 volt circuits, but I’ve always heard them referred to as “220 circuits”.  There is no neutral wire in this setup, as both non-ground wires are delivering power.  These are the kind of connectors used in the home for things like air conditioners or clothes dryers.  In my experience, the standard versions of these connectors are uncommon.  The locking version are used far more frequently, usually do to the fact that whatever is running off of that much power probably doesn’t want to go offline due to having a power plug kicked out of the wall.  These connectors all begin with a “6” followed by the amperage of the circuit, usually between 15 and 30 amps.

6-20

The only non-locking NEMA 6-series connector I run into on a regular basis is the 6-20.  Note that while this plug/receptacle is similar to it’s 5-20 cousin, the perpendicular blade is the opposite on this connector to prevent you from plugging the wrong cord into the wrong plug and getting a nasty surprise.

L6-20

The locking version of the NEMA 6-20.  Far more common due to the ability to keep whatever this cord is powering plugged in.  Odds are good that if you are using a heavy duty power supply in a device like a Catalyst 4500 or a Catalyst 6500, you’ve got this cord powering your device.

L6-30

Big devices call for big power.  This 208/240v 30 amp circuit connector is going to power just about anything you can throw at it.  Big UPSes or huge power supplies in a switch like the 8700 watt monster in the Catalyst 6500.  You are most likely to encounter this connector in a rack-mounted PDU where it provides the power coming from the main electrical connection and then the PDU disperses the power to the devices in the rack itself.

IEC 60320

The IEC connectors are fairly common in electronics devices, however I believe that most people couldn’t pick them out of a lineup.  That’s because the IEC connectors are the ends that are on the opposite side of the cord from the power plug.  These ends plug into power transformers and power supplies.  By making them an international standard, the equipment manufacturers need only put one kind of receptacle on their equipment and then manufacture the various country-specific cords when needed.  Also, unlike the NEMA connectors above that use “P” and “R” to denote plugs and receptacles, the IEC connectors use a different number to specify the plug and receptacle, for instance the IEC C19 is the plug and the IEC C20 is the receptacle.

C7/C8

This connector is used for power transformers, laptop power supplies and small appliances, like the original Playstation.  This connector is shaped like a figure 8, unless you have the polarized version when is squared off on the hot end.  I’ve spent many an hour looking for one of these connectors for a forgotten device.

C13/C14

If you’ve ever worked with a computer, you know this power cord.  In fact, for the majority of my career I’ve called it a “computer power cord”.  This particular IEC connector run everything from desktop computer to fixed-configuration switches.  I always carry two or three spare cables with me at all times just in case I need them.  I’m also starting to see more and more higher wattage devices requiring two or more cords to work properly, like server power supplies or the newer power supplies for chassis switches.  The C14 plug is also very common on power strip type PDUs for racks where you plug the C13 end into your server and the C14 end into the PDU.  Saves a bit more space than the NEMA 5-15 connector above.

C15/C16

Odds are good you’ve seen this cable and said “Huh?”.  I’ve started seeing it ship with newer fixed-configuration switches that had Power over Ethernet (PoE) power supplies.  For the life of me, I couldn’t figure out the point of changing this connector.  A chance comment from one of my friends about this so-called “kettle cord” made me start researching what was so special about the connector.  It turns out the C13/C14 connector/cord is only rated to about 70 degrees F.  The C15/C16 was specifically designed for higher temperature devices, like electric kettles and PoE switches with higher wattage power supplies, like those that drive 802.3at or PoE+ 30 watts per port.  This connector will work in the C13/C14 receptacles, but not vice versa.  This prevents the cord melting and causing all kinds of trouble.

C19/C20

If you’ve ever installed a Catalyst 4500 switch, you’ve seen this plug on the power supply.  For a long time, I referred to it as the “chassis switch power connector” without knowing it had a real name.  The C19/C20 connectors are used for devices that require a higher current than that which can be provided by the C13/C14/C15/C16 connectors.  I’ve only ever seen it on chassis switches, but it can also appear in high end workstations or some kinds of UPSes.

Tom’s Take

When you stare at the long list of power options available for a switch or a server, you might find your eyes crossing at the multitude of unfamiliar acronyms you find.  At best you’ll end up with an extra power cord you don’t need.  At worst, you could delay a large project for many days because you ordered an L6-20 cord when you really needed a 5-20 connector.  The same goes for ensuring you have the correct power connections hooked up in your server closet or data center.  Electricians aren’t sure what you’re talking about when you tell them you want a “winking man” connector or “the twisty one” kind of plug.  If you tell them you need an L5-20R, they’ll be able to put one in without asking another question.  Network rock stars always talk about the virtues of standardization and the electrical world is no different.  By knowing the standard name for the NEMA or IEC connector you need, you can be sure that you are the one that has the power.

Nerd v6 – My First IPv6 Tunnel

Posted on by
7

Home - Courtesy of ThinkGeek (Click to buy this shirt!)

I realized the other day that I’ve done a lot of IPv6-related posts in the past few months, but for one reason or another I keep putting off setting up my own IPv6 presence.  I signed up for a free IPv6 tunnel from Hurricane Electric’s Tunnel Broker service almost a year ago.  However, the Cisco Valet Plus router I have running my home network has zero support for IPv6, let alone tunnels.  Because of that little omission, my tunnel sat unused for a while, gathering 128-bit dust.  As the end of the year approached, I found myself with some extra time on my hands and a bit of spare gear thanks to my local Cisco office.  I decided that maybe it was time to turn up my IPv6 nerdiness to the next level.

I found an old 2821 in my lab that wasn’t serving an immediate purpose.  I erased it and reconfigured it to serve as a gateway for my IPv6 setup.  I logged back into my account at tunnelbroker.net and found that I could create up to 5 tunnels for free.  Who in their right mind would turn that down?  I created a new tunnel for my lab environment.  In this interface, you would create a regular tunnel for basic connectivity.  You input your IPv4 address as one side of the endpoint.  HE.net provides the IP that you’re coming in on if you wanted to set this up with a home connection, for instance.  In my case, I already had a global IPv4 address ready to go.  However, the router wasn’t all the way up yet.  If HE.net can’t ping your IPv4 tunnel endpoint, they won’t reserve the address space.  So:

Tip #1: Don’t start setting up your tunnel until you’ve got your gateway ready.

I picked the closest endpoint to my geographic area – Dallas, TX.  Once the router came all the way up and the ARP caches had settled down, I registered my new tunnel without a hitch.  HE.net provides you with a /64 for your tunnel interface.  ::1 is an interface on their side, ::2 is an interface to assign to your tunnel.  They also provide you with an entirely different /64 to setup for your client devices behind your router/firewall.  If you’re wanting to bring more than one site online, you can even register for a /48.  That’s 1.20892582 × 10^24 addresses.  Per tunnel.  That’s a lot for nothing!

My first attempt with Dallas didn’t work out so well.  For some reason, I couldn’t ping the other side of my tunnel.  I gave it about 15 minutes and then gave up.  I tore down the tunnel and created a new one.  If you’re going to do that, give the HE.net servers about 5 minutes to clean up their side of things, since the tunnel creation script will think that you’re  trying to register an endpoint twice.  I picked a nice little sunny patch of hex addresses in Fremont, CA and this time it worked!  I was able to ping from one side of my tunnel to the other.  For reference, this is the sample config they gave me for the tunnel and it worked quite nicely:

interface Tunnel0
 description Hurricane Electric IPv6 Tunnel Broker
 no ip address
 ipv6 enable
 ipv6 address <Your Tunnel /64 Endpoint>
 tunnel source <Your IPv4 Interface>
 tunnel destination <HE.net's IPv4 Interface>
 tunnel mode ipv6ip
ipv6 route ::/0 Tunnel0

Easy, right?  Once that’s up and running, you can configure the other interface of your router with the routed /64 or /48 they assign you.  I’d suggest starting with the /64 first to get your feet wet.  There is one other piece of configuration you need to enable that seems to be the cause of many issues on HE.net’s forums when configuring Cisco devices.  You need to enable IPv6 routing with this command:

ipv6 unicast-routing

Tip #2: Don’t forget to enable IPv6 routing.

Now that you’ve got two sides up and running and routing between each other, you should be able to launch some packets toward the Interwebz v6.  HE.net sets you up with an IPv6 DNS server you can plug into your devices to test connectivity.  If you want to be able to ping ipv6.google.com from your router, be sure to enter this command:

ip name-server 2001:470:20::2

Now you can test to your heart’s content.  When you’re sure that your tunnel is going to stay up, you need to concentrate on getting desktops working.  That’s where the routed /64 comes into play.

HE.net gives you an additional routed /64 that is different than the tunnel address for the purposes of setting up a site for IPv6.  Most networking people know that you must have two different subnets on the router for routing to occur.  Yet, on the HE.net forums I see a lot of people that are configuring their routers with addresses from the tunnel /64 subnet.  Save yourself the headache and use the other /64 to get started.

The easiest way to setup your client device with an IPv6 address is good old fashioned static addressing.  This is very easy to do in both Windows and OS X.  Lucky for you that both of the latest versions of those OSes have IPv6 enabled by default.  You just have to click on the option for IPv6 and assign a static IP.  You should also use the HE.net IPv6 name server listed above to allow you to resolve addresses like ipv6.google.com.  I tested with an OS X Lion server and was able to get the machine running on a static IP with no real issues.  I gave my Windows 7 workstation an IP in the same range and they were able to happily ping each other with no problems.  I think I’m going to take another post to talk about the fun of configuring DHCPv6 and SLAAC on my tunnel, as that has caused me a bit of heartburn so far making everything play nice with other people’s ideas about security.

Speaking of security…I would be remiss if I didn’t end this little article without a discussion of securing your new tunneling router from the nastiness on the Internet.  I found a great access list on the HE.net forums and thought I’d share it with you:

ipv6 access-list internet_inbound_ipv6
 remark Permit IPv6 Link-Local & Multicast
 permit ipv6 FE00::/7 any
 remark Block IPv6 Bogons
 deny ipv6 ::/3 any
 deny ipv6 4000::/2 any
 deny ipv6 8000::/1 any
 remark Block own assigned IPv6 space
 deny ipv6 <Your HE.net /64>::/64 any
 remark Block anything going to Windows RPC
 deny tcp any any eq 135
 permit icmp any any
!
ipv6 access-list internet_outbound_ipv6
 remark Prohibit any contact with Windows RPC-NetBIOS
 deny tcp any any eq 135
 deny tcp any any eq 137
 deny tcp any any eq 138
 deny tcp any any eq 139
 deny udp any any eq 135
 deny udp any any eq netbios-ns
 deny udp any any eq netbios-dgm
 deny udp any any eq netbios-ss
 remark Allow traffic from own assigned IP space
 permit ipv6 <Your HE.net /64>::/64 any

Of note here, remember that in IPv6, ICMP does a lot more than just pings.  Read RFC4890 for a lot more info on the subject, but right now I’m just allowing the whole stack in.  If you want to permit traffic to servers for things like HTTP or SMTP, be sure to add those servers at the end of the Inbound ACL so the traffic doesn’t get dropped.

Tom’s Take

One of the things that impressed me when I started troubleshooting some of the issues with my HE.net tunnel was the help that people were more than willing to give in the HE.net forums.  Especially where they helped their brethren with things like establishing connectivity.  Lots of posts about being able to ping router tunnel endpoints and things like that.  It shows that setting up your own IPv6 presence isn’t really that hard and should allow you to get out on the wider Internet with IPv6 in short order.  Thanks to all the work that other’s have been more than willing to share, I had an easier time than many.  I just hope my examples here help someone to get their tunnels up and running.

Rollover Beethoven – USB’s In Town

Posted on by
1

Every Cisco engine…rock star in the world should have a rollover cable or two stashed away in their bag/car/pocket just in case.  The rollover serial cable is the hallmark of access to a Cisco device.  The console port is the last resort for configuration when all else has gone wrong.  It is the first thing you should plug into when you boot up a router for the first time and the best way to get info you couldn’t otherwise find.  However, the days of the serial cable are quickly becoming numbered.

It wasn’t all that long ago that every PC manufactured included a 9-pin serial connection.  These ports were handy for all kinds of devices, including printers and modems.  However, with the introduction of Universal Serial Bus (USB) connections, the usefulness of the serial (and parallel) ports has been waning quickly.  By utilizing a higher speed connection that more tightly integrates into the system, the need to configure devices with DIP switches and play COM port roulette have long since passed.  As it is with any transition though, there have been some holdouts in the movement to retire serial ports.  While some of these are understandable due to outdated single-purpose technology, others have never made any sense to me, like the Cisco rollover console cable.  Surely there must be a better way to connect to the serial port of a device than with an outdated technology holdover from the 80s?  I myself am a victim of this kind of thinking, having used an IBM T30 Thinkpad well past its useful life simply because it had an integrated serial port and my replacement laptop wouldn’t.

When Cisco developed the new ISR G2 line of routers, someone in the console access department finally decided to wake up and get with the 2000’s.  Thanks to their efforts, the Cisco routers and switches manufactured today have started including a new console access option:

In the picture above, you can see the familiar RJ-45 console port to the right and the newer USB console port to the left, indicated with the USB icon.  This new port allows those of us that have spent most of our lives using the flat blue rollover serial cables to add a new, exciting cable to our bag, the USB A-to-mini cable.

The new USB port allows the user to access the router’s console with a newer cable instead of relying on the standby rollover cable.  However, you need to take a few steps first.  You have to head out to the Cisco Connections Online (CCO) download page and pull the driver for your particular operating system if you’re running on Windows.  Make sure you specify 32-bit or 64-bit, since this driver will be masquerading as a COM port on your system.  You don’t want to waste time downloading a driver that won’t work.  Once you’ve installed the driver, you can plug in your USB connection to any USB port and then to the router.  It will look like an additional COM port on your system, probably with a high number like COM6 or COM7, so make sure you’ve got a terminal emulator that allows you to choose your COM port.  I tend to use TeraTerm for this very reason, but your terminal program of choice should do nicely.  For those of you in the audience with Macbooks, you don’t need to download any drivers at all.  Seems like OS X already has the right driver built in, so just plug and and get cranking.  As a quick aside, Cisco will attempt to sell you a $30 USB console cable when you order the router. JUST. SAY. NO.  This is a regular USB A-to-Mini cable that can be purchased at Walmart for about $10.  You can even use the USB cable that came with your digital camera or Blackberry or old Motorola RAZR.

Once you get attached to the USB console port, you’ll find that it works pretty much the same as the RJ-45 port that you’ve become attached to over the years.  You can also plug in a regular old serial cable into the RJ-45 port if you need a second connection.  The RJ-45 console port will mirror what’s going on with the USB console port.  However, since their both Console 0, only one of them will have preference on the input.  In this case, that’s the USB port.  So if you have a terminal access server plugged in for reverse telnet connections and someone comes in and attaches a USB connector, you can watch what’s going on but you can’t do anything about it.  You can specify a timeout value if you’d like so you can force a logout after inactivity.  You can do that with the following command:

Router(config)# line con 0
Router(config-line)#usb-inactivity-timeout <value in minutes>

Note that this command doesn’t work on the 2900 series ISR G2 routers for some strange reason.  Oh well, feature request down the road.  For those of you out there that don’t feel comfortable with the idea of having just anyone off the street walking up and consoling into your router via USB, you can always disable the USB console port in favor of the RJ-45 connection as follows:

Router(config)# line con 0
Router(config-line)# media-type rj45

Bingo.  USB port locked out.  Now only those people in possession of a serial-to-USB adapter or a Redpark iOS Console Cable will have access.

Tom’s Take

I have three rollover cables in my various laptop bags.  I keep two for emergencies and one in case someone doesn’t have theirs.  I passed out console cables to all my engineers and technicians once and told them the next time I asked them for their console cable, they’d better present this one to me.  A console cable is an indispensable tool for anyone that works on Cisco equipment.  Having the USB option is always welcome since I no longer have to fumble for my USB-to-serial adapter or worry that the dodgy drivers are going to bluescreen my Windows 7 64-bit laptop over and over again.  Still, there is a lot of Cisco equipment out there with the older RJ-45 cable setup as the only console option.   So you can’t just throw out the old rollover serial cables just yet.  Better to throw a USB cable in your bag for those glorious days where you get to access a newer device.  Then you can await the day when you can bury your rollover cable alongside Beethoven.