CCIE Version 5: Out With The Old

Cisco announced this week that they are upgrading the venerable CCIE certification to version five.  It’s been about three years since Cisco last refreshed the exam and several thousand people have gotten their digits.  However, technology marches on.  Cisco talked to several subject matter experts (SMEs) and decided that some changes were in order.  Here are a few of the ones that I found the most interesting.

CCIEv5 Lab Schedule

Time Is On My Side

The v5 lab exam has two pacing changes that reflect reality a bit better.  The first is the ability to take some extra time on the troubleshooting section.  One of my biggest peeves about the TS section was the hard 2-hour time limit.  One of my failing attempts had me right on the verge of solving an issue when the time limit slammed shut on me.  If I only had five more minutes, I could have solved that problem.  Now, I can take those five minutes.

The TS section has an available 30 minute overflow window that can be used to extend your time.  Be aware that time has to come from somewhere, since the overall exam is still eight hours.  You’re borrowing time from the configuration section.  Be sure you aren’t doing yourself a disservice at the beginning.  In many cases, the candidates know the lab config cold.  It’s the troubleshooting the need a little more time with.  This is a welcome change in my eyes.


The biggest addition is the new 30-minute Diagnostic section.  Rather than focusing on problem solving, this section is more about problem determination.  There’s no CLI.  Only a set of artifacts from a system with a problem: emails, log files, etc.  The idea is that the CCIE candidate should be an expert at figuring out what is wrong, not just how to fix it.  This is more in line with the troubleshooting sections in the Voice and Security labs.  Parsing log files for errors is a much larger part of my time than implementing routing.  Teaching candidates what to look for will prevent problems in the future with newly minted CCIEs that can diagnose issues in front of customers.

Some are wondering if the Diagnostic section is going to be the new “weed out” addition, like the Open Ended Questions (OEQs) from v3 and early v4.  I see the Diagnostic section as an attempt to temper the CCIE with more real world needs.  While the exam has never been a test of ideal design, knowing how to fix a non-ideal design when problems occur is important.  Knowing how to find out what’s screwed up is the first step.  It’s high time people learned how to do that.

Be Careful What You Wish For

The CCIE v5 is seeing a lot of technology changes.  The written exam is getting a new section, Network Principles.  This serves to refocus candidates away from Cisco specific solutions and more toward making sure they are experts in networking.  There’s a lot of opportunity to reinforce networking here and not idle trivia about config minimums and maximums.  Let’s hope this pays off.

The content of the written is also being updated.  Cisco is going to make sure candidates know the difference between IOS and IOS XE.  Cisco Express Forwarding is going to get a focus, as is ISIS (again).  Given that ISIS is important in TRILL this could be an indication of where FabricPath development is headed.  The written is also getting more IPv6 topics.  I’ll cover IPv6 in just a bit.

The biggest change in content is the complete removal of frame relay.  It’s been banished to the same pile as ATM and ISDN.  No written, no lab.  In it’s place, we get Dynamic Multipoint VPN (DMVPN).  I’ve talked about why Frame Relay is on the lab before.  People still complained about it.  Now, you get your wish.  DMVPN with OSPF serves the same purpose as Frame Relay with OSPF.  It’s all about Stupid Router Tricks.  Using OSPF with DMVPN requires use of mGRE, which is a Non-Broadcast Multi-Access (NBMA) network.  Just like Frame Relay.  The fact that almost every guide today recommends you use EIGRP with DMVPN should tell you how hard it is to do.  And now you’re forced to use OSPF to simulate NBMA instead of Frame Relay.  Hope all you candidates are happy now.


The lab is also 100% virtual now.  No physical equipment in either the TS or lab config sections.  This is a big change.  Cisco wants to reduce the amount of equipment that needs to be physically present to build a lab.  They also want to be able to offer the lab in more places than San Jose and RTP.  Now, with everything being software, they could offer the lab at any secured PearsonVUE testing center.  They’ve tried in the past, but the access requirements caused some disaster.  Now, it’s all delivered in a browser window.  This will make remote labs possible.  I can see a huge expansion of the testing sites around the time of the launch.

This also means that hardware-specific questions are out.  Like layer 2 QoS on switches.  The last reason to have a physical switch (WRR and SRR queueing) is gone.  Now, all you are going to get quizzed on is software functionality.  Which probably means the loss of a few easy points.  With the removal of Frame Relay and L2 QoS, I bet that services section of the lab is going to be really fun now.

IPv6 Is Real

Now, for my favorite part.  The JNCIE has had a robust IPv6 section for years.  All routing protocols need to be configured for IPv4 and IPv6.  The CCIE has always had a separate IPv6 section.  Not any more.  Going forward in version 5, all routing tasks will be configured for v4 and v6.  Given that RIPng has been retired to the written exam only (finally), it’s a safe bet that you’re going to love working with OSPFv3 and EIGRP for IPv6.

I think it’s great that Cisco has finally caught up to the reality of the world.  If CCIEs are well versed in IPv6, we should start seeing adoption numbers rise significantly.  Ensuring that engineers know to configure v4 and v6 simultaneously means dual stack is going to be the preferred transition method.  The only IPv6-related thing that worries me is the inclusion of an item on the written exam: IPv6 Network Address Translation.  You all know I’m a huge fan of NAT.  Especially NAT66, which is what I’ve been told will be the tested knowledge.

Um, why?!? 

You’ve removed RIPng to the trivia section.  You collapsed multicast into the main routing portions.  You’re moving forward with IPv6 and making it a critical topic on the test.  And now you’re dredging up NAT?!? We don’t NAT IPv6.  Especially to another IPv6 address.  Unique Local Addresses (ULA) is about the only thing I could see using NAT66.  Ed Horley (@EHorley) thinks it’s a bad idea.  Ivan Pepelnjak (@IOSHints) doesn’t think fondly of it either, but admits it may have a use in SMBs.  And you want CCIEs and enterprise network engineers to understand it?  Why not use LISP instead?  Or maybe a better network design for enterprises that doesn’t need NAT66?  Next time you need an IPv6 SME to tell you how bad this idea is, call me.  I’ve got a list of people.

Tom’s Take

I’m glad to see the CCIE update.  Getting rid of Frame Relay and adding more IPv6 is a great thing.  I’m curious to see how the Diagnostic section will play out.  The flexible time for the TS section is way overdue.  The CCIE v5 looks to be pretty solid on paper.  People are going to start complaining about DMVPN.  Or the lack of SDN-related content.  Or the fact that EIGRP is still tested.  But overall, this update should carry the CCIE far enough into the future that we’ll see CCIE 60,000 before it’s refreshed again.

More CCIE v5 Coverage:

Bob McCouch (@BobMcCouch) – Some Thoughts on CCIE R&S v5

Anthony Burke (@Pandom_) – Cisco CCIE v5

Daniel Dib (@DanielDibSWE) – RS v5 – My Thoughts

INE – CCIE R&S Version 5 Updates Now Official

IPExpert – The CCIE Routing and Switching (R&S) 5.0 Lab Is FINALLY Here!

Betting the Farm on IPv6


IPv6 seems to have taken a back seat to discussions about more marketing-friendly topics like software defined networking and the Internet of Things.  People have said that IPv6 is an integral part of these initiatives and any discussion of them implies IPv6 use.  Yet, as I look around at discussions about SDN host routes or NATed devices running home automation for washing machines and refrigerators, I wonder if people really understand the fundamental shift in thinking.

One area that I recently learned has been investing heavily in IPv6 is agriculture.  When people think of a farm, they tend to imagine someone in a field with a plow being pulled by a horse.  Or, in a more modern environment you might imagine a tractor pulling a huge disc plow across hundreds of acres of fallow land.  The reality of faming today is as far removed from the second example as the second example is from the first.

Farming In The East

Modern farmers embrace all kinds of technology to assist in providing maximum yields, both in the western world as well as the east.  The biggest strides in information technology assistance for farmers has been in East Asia.  Especially in China, a country that has to produce massive amounts of food to feed 1.3 billion people.

Chinese farmers have embraced technologies that allow them to increase productivity.  Think about how many tractors are necessary to cultivate the huge amount of land needed to grow food.  Each of those tractors now comes equipped with a GPS transmitter capable of relaying exact positioning.  This ensures the right land is being worked and the area is ideal for planting certain types of crops.  All that telemetry data needs to be accumulated somewhere in order to analyze and give recommendations.

Think also about livestock.  In the old days, people hired workers to ensure that livestock didn’t escape or wander away from the herd.  It was a process that was both time and labor intensive.  With modern technology, those same cattle can be tagged with a small GPS transmitter.  A system can poll each animal in a given interval to determine herd count and location.  Geofences can be erected to ensure that no animal moves outside of a safe area.  When that occurs, alarms can be sent to monitoring stations where a smaller number of farm hands can drive out and rescue the errant animal.

Those two examples alone show the power of integrating traditional agriculture with information technology.  However, an unstated problem does exist: Where are we going to get those addresses?  We joke about giving addresses to game consoles and television sets and how that’s depleting the global IPv4 pool.  What happens when I do the same to dairy farmer’s herd?  Even my uncle and his modest dairy years ago had around one hundred cattle in his herd.  What happens when your herd is bigger than a /24?

IPv6 Rides To The Rescue

China has already solved this problem.  They don’t have any more IPv4 prefixes available.  They have to connect their devices.  The only answer is IPv6.  Tractors can exist as IPv6 endpoints in the monitoring station.  They can be tracked globally via monitoring stations.  Farm workers and supervisors can determine where the unit is at any given time.  Maintenance information can be relayed back to the manufacturer to alert when a part is on the verge of failure.  Heavy equipment can stay in working condition longer and be used more efficiently with this type of tracking.

Livestock herds can be monitored for position to ensure they are not trespassing on another farmers land.  The same telemetry can be used to monitor vital statistics to discover when animals have become ill.  That allows the farm workers to isolate those animals to prevent the herd from contracting illness that will slow production or impact yields.  Keeping better track of these animals ensures they will be as productive as possible, whether that be in a dairy case or a butcher shop.

Tom’s Take

I grew up on a farm.  I have gathered eggs, bottle fed calves, and milked cows.  Two of my uncles owned dairies.  The biggest complaint that I’ve heard from them was the lack of information they had on their products.  Whether it be a wheat crop or a herd of dairy cattle, they always wanted to know more about their resources.  With IPv6, we can start connecting more and more things to the Internet to provide access to the data that’s been locked away for so long, inaccessible to the systems that can provide insight.  Advancing technology to the point where a tractor or a bull can have a 2001::/16 address is probably the safest bet a farmer will make in his entire career.

IPv4? That Will Cost You


After my recent articles on Network Computing, I got an email from Fred Baker.  To say I was caught off guard was an understatement.  We proceeded to have a bit of back and forth about IPv6 deployment by enterprises.  Well, it was mostly me listening to Fred tell me what he sees in the real world.  I wrote about some of it over on Network Computing.

One thing that Fred mentioned in a paragraph got me thinking.  When I heard John Curran of ARIN speak at the Texas IPv6 Task Force meeting last December, he mentioned that the original plan for IPv6 (then IPng) deployment involved rolling it out in parallel with IPv4 slowly to ensure that we had all the kinks worked out before we ran out of IPv4 prefixes.  This was around the time the World Wide Web was starting to take off but before RFC 1918 and NAT extended the lifetime of IPv4.  Network engineers took a long hard look at the plans for IPv6 and rightfully concluded that it was more expensive to run IPv6 in conjunction with IPv4 and instead it was more time and cost effective to just keep running IPv4 until the day came that IPv6 transition was necessary.

You’ve probably heard me quote my old Intro to Database professor, Dr. Traci Carte.  One of my favorite lessons from her was “The only way to motivate people is by fear or by greed.”  Fred mentioned that an engineer at an ISP mentioned to him that he wanted to find a way to charge IPv4 costs back to the vendors.  This engineer wants to move to a pure IPv6 offering unless there is a protocol or service that requires IPv4.  In that case, he will be more than willing to enable it – for a cost.  That’s where the greed motivator comes into play.  Today, IPv6 is quickly becoming equivalent in cost to IPv4.  The increased complexity is balanced out by the lack of IPv4 prefixes.

What if we could unbalance the scales by increasing the cost of IPv4?  It doesn’t have to cost $1,000,000 per prefix.  But it does have to be a cost big enough to make people seriously question their use of IPv4.  Some protocols are never going to be ported to have IPv6 versions.  By making the cost of using them higher, ISPs and providers can force enterprises and small-to-medium enterprises (SMEs) to take a long hard look at why they are using a particular protocol and whether or not a new v6-enabled version would be a better use of resources.  In the end, cheaper complexity will win out over expensive ease.  The people in charge of the decisions don’t typically look at man-hours or support time.  They merely check the bottom line.  If that bottom line looks better with IPv6, then we all win in the end.

I know that some of you will say that this is a hair-brained idea.  I would counter with things like Carrier-Grade NAT (CGN).  CGN is an expensive, complicated solution that is guaranteed to break things, at least according to Verizon.  Why would you knowingly implement a hotfix to IPv4 knowing what will break simply to keep the status quo around for another year or two?  I would much rather invest the time and effort in a scaling solution that will be with us for another 10 years or more.  Yes, things my break by moving to IPv6.  But we can work those out through troubleshooting.  We know how things are supposed to work when everything is operating correctly.  Even in the best case CGN scenario we know a lot of things are going to break.  And end-to-end communications between nodes becomes one step further removed from the ideal.  If IPv4 continuance solutions are going to drain my time and effort they become as costly (or moreso) that implementing IPv6.  Again, those aren’t costs that are typically tracked by bean counters unless they are attached to a billable rate or to an opportunity cost of having good engineering talent unavailable for key projects.

Tom’s Take

Dr. Carte’s saying also included a final line about motivating people via a “well reasoned argument”.  As much as I love those, I think the time for reason is just about done.  We’ve cajoled and threatened all we can to convince people that the IPv4 sky has fallen.  I think maybe it’s time to start aiming for the pocketbook to get IPv6 moving.  While the numbers for IPv6 adoption are increasing, I’m afraid that if we rest on our laurels that there will be a plateau and eventually the momentum will be lost.  I would much rather spend my time scheming and planning to eradicate IPv4 through increased costs than I would trying to figure out how to make IPv4 coexist with IPv6 any longer.

IP Addresses in Entertainment

Fake IP

Every time I sit down to watch a TV show or movie and they mention computers or hacking, I get amused.  I know that I’m probably going to see some attempt to make computer hacking look cool or downright scary.  Whether it be highly stylized like Hackers or fairly accurate like the power plant hack in The Matrix Reloaded, there are always little details that get glossed over.  In many cases, one of these is the IP addressing of the systems themselves.  If the producers and writers of the film even choose to show an IP address on the screen, it’s usually so wrong that I laugh at a totally inappropriate moment of drama.

The practice of using fictitious numbering schemes for things in entertainment goes back several decades.  The first known instance of a movie using a fake number for something was in Panic in Year Zero back in 1962.  For the first time, the writers used a fictitious phone number starting with 555 instead of a real telephone number.  Even though 555 prefixes were used for things like directory assistance, they weren’t widely deployed.  As such, the 555 prefix became synonymous with a “fake” phone number.  555-0100 through 555-0199 are the only official numbers in that range set aside for fictitious use, however many people still associate that prefix with a phone number that won’t work in the real world.

Hollywood has been trying for some time to come up with IP addresses that look real enough to pass the sniff test but are totally false.  Sometimes that works.  Other times, you end up with Law and Order.  In particular, the SVU flavor of that show has been known to produce IP address ranges that don’t even come close to looking real.  This page documents a couple of the winners from that show when the police start tracing an offender by their IP address.  Some of them look almost real.  Others seem to have an octet that jumps above 255.  Still others have 4-digit octets or other oddities that don’t quite measure up.  Sure, it heightens the suspense when people can see what the detectives are doing, but for those of us that know enough to be dangerous, it pulls you out of the moment.  It would be like watching ER and hearing the doctors start talking about brain surgery, only to start cutting open a patient’s arm to get to it.

TCP/IP has a large number of address ranges that can be used in a fictitious manner. For instance, Class E experimental addresses ( were set aside and hard coded into most OSes as unavailable.  The address range for example use and documentation purposes can also serve as a safe fictitious range.  Then there’s RFC 1918.  These addresses are used for private network ranges and must be NATed to work correctly on the public internet due to their non-routability.  These would be perfect for use in movies, as they represent networks that most people use daily.  They would look believable to those of us that know what to look for.  However, I think the producers and writers avoid doing that because of the inherent curiosity of people.

The greatest example of this comes courtesy of Tommy Tutone.  The band hit radio gold with their song “867-5309/Jenny” back in 1982.  Unlike 555, 867 is a widely used prefix code in the North American Numbering Plan (NANP).  There are numerous stories of people that have received that phone number and been cursed with popularity.  One story from Brown university tells of unsuspecting freshmen that move into the dorm room with that telephone number.  The phone calls never stop until a request is made to shut down the line.  Even back in 1982, the regional Bell companies were seeing huge spikes in telephone calls to that one number.  In many cases, they had to disconnect it in order to keep the traffic to a reasonable level.  If you’re curious, you can hear some of the messages left for the unfortunate possessors of that cursed number over at

People are compelled to try things they see in movies.  This article in the Chicago Tribune talks about the writer memorizing a realistic looking number from a movie and going home to call it several times before giving up.  The movie Magnolia included the real number 877-TAME-HER which the movie studio used to record Tom Cruise giving an in-character speech about his system for the purposes of marketing.  That’s all well and good in the real world when someone gets a few occasional prank calls or other harmless issues.  What happens in a computer network when someone sees a address on TV and then decides to try and hack it?  What if they call the police and say that the computer address of a murder or a predator is on their network?  This can cause huge issues for network admins.  The nightmare of trying to explain to people that just because the Gibson in Hackers 3 is at doesn’t mean they get to assault the mail server every day would get old really fast.  And when it comes to IPv6, the opportunity for even more trouble arises.

I was a long-time player of the MMORPG City of Heroes.  One of the reasons that I liked playing it so much was the lore and back story to the world.  I was one of the players that read all of the fluff text to get a better sense of what the writers were trying to do.  Imagine my surprise when I was playing a new mission a several months ago and ran across a little Easter egg.  One of the writers decided that the imaginary world of Paragon City had long ago ran out of IPv4 addresses and decided to upgrade to IPv6.  One of the consoles in the game had a reference to an IPv6 address – 3015:db6:97c4:9e1:2420:9b3f:073:8347.  I was excited.  Finally, someone in the entertainment industry realized we were running out of IPv4!  Then I started thinking.  Right now, the allocations to the RIRs all start with 2001.  Eventually, once we get the intergalactic Internet up and running, we might end up getting into the 3000 range.  It might be a hundred years before the address above is allocated to someone.  By then, most everyone will have forgotten City of Heroes ever existed.  Putting real IPv6 addresses in movies and on TV does run the risk of having people “hacking the Gibson” when you least expect it.  I think you’ll see that even in those far-flung ranges, the odds of a fake address on TV coinciding with a real IPv6 server or workstation address, even on a global scale, is pretty slim.  Despite the fact that all our systems will be globally reachable, the IPv6 address space is so large that no two systems are likely to even overlap.  Add in neighbor discovery, duplicate address detection, and the uniqueness of a MAC address (which forms the basis of EUI-64 addressing and SLAAC) and you can see how difficult it would be.

Tom’s Take

In case the name of my blog didn’t warn you…I’m a nerd.  When I see something inaccurate in a movie, I tend to point it out.  That’s why I don’t watch Armageddon any more.  I understand that writers and directors are trying to entertain people.  When you’re trying to do that, sometimes the details get sacrificed for the sake of telling a good story.  However, when it comes to something that can represented easily for the most realistic look possible, the creative team involved should do that.  Whether it be the night sky in Titanic or the address of the mainframe in a techno thriller, I want the people that care about the production values of a movie to show me how much they care.  With the advent of IPv6, I think creating fake addresses to put in movies and other entertainment will be easier.  Given the vast range of available space it doesn’t take too much effort to pull out something “techy sounding” to put in a movie script.  Trust me, the nerds out there will thank you for it.

2012 Depleted, Time to Adopt ::2013

It’s been 366 days since my last post about goals for 2012.  How’d I do on my list for the past year?

1. Juniper – Dropped the ball on this one.  I spent more time seeing Juniper gear being installed all over the place and didn’t get my opportunity to fire up the JNCIA-Junos liked I wanted.  I’m planning to change all that sooner rather than later.  Doug Hanks even gave me a good head start on immersion learning of the MX Series.

2. Data Center – I did get a little more time on some Nexus gear, but not nearly enough to call it good for this goal.  Every time I sat down to start looking at UCS, I kept getting pulled away on some other project.  If the rumblings I’m hearing in the DC arena are close to accurate, I’m going to wish I’d spent more time on this.

3. Advanced Virtualization – While I didn’t get around to taking either of the VCAP tests in 2012, I did spend some more time on virtualization.  I was named a vExpert for 2012, gave a virtualization primer presentation, and even attended my first VMUG meeting.  I also started listening to the vBrownBag podcast put on by ProfessionalVMware.  They have a ton of material that I’m going to start reviewing so I can go out and at least take the DCD test soon.

4. Moving to the Cloud – Ah ha! At last something that I nailed.  I moved a lot of my documents and data into cloud-based storage.  I leveraged Dropbox, Skydrive, and Google Docs to keep my documentation consistent across multiple platforms.  As I continue forward, I’m going to keep storing my stuff in the big scary cloud so I can find it whenever I need it.

Looks like I’ve got two fails, one tie, and one win.  Still not the 50% that I had hoped for, but it’s funny how real life tends to pull you in a different direction that you anticipate.  Beyond attending a few more Tech Field Day events and Cisco Live, I also attended a Cisco Unified Communications Partner Beta Training launch event and the Texas IPv6 Task Force Winter Summit.  It was this last event that really got me thinking about what I wanted to do in the coming year.

I think that 2013 is going to be a huge year for IPv6 adoption on the Internet.  We’ve been living in the final depletion phase of IPv4 for a whole year now.  We can no longer ignore the fact that IPv6 is the future.  I think the major issue with IPv6 adoption is getting the word out to people.  Some of the best and brightest are doing their part to talk to people about enabling IPv6.  The Texas IPv6 Task Force meeting showed me that a lot of great people are putting in the time and effort to try and drive people into the future.  However, a lot of this discussion is happening outside of people’s view.  Mailing lists aren’t exactly browsing-friendly.  Not everyone can drop what they’re doing for a day or two to go to a task force meeting.  However, people do have the spare time to read a blog post on occasion.  That’s where I come in.

In 2013, I’m going to do my part to get the word out about IPv6.  I’m going to spend more time writing about it.  I’m going to write posts about enabling it on all manner of things.  Hypervisors, appliances, firewalls, routers, and even desktops are on the plate.  I want to take the things I’m learning about IPv6 and apply them to the world that I work in.  I don’t know how service providers are going to to enable IPv6.  However, I can talk about enabling CallManager to use IPv6 and register IP phones without IPv4 addresses.  I can work out the hard parts and the gotchas so that you won’t have to.  I’ve already decided that any presentation that I give in 2013 will be focused on IPv6.  I’ve already signed up for one slot later in the year with a possibility of having a second.  I applied for a presentation slot at the Rocky Mountain IPv6 Task Force meeting in April.  I want to hone my skills talking to people about IPv6.  I’m also going to try and make a lot more blog posts about IPv6 in the coming year.  I want to take away all the scary uncertainty behind the protocol and make it more agreeable to people that want to learn about it without getting scared off by the litany of RFCs surrounding it.  To that end, I’m going to start referring to this year as ::2013.  The more we get familiar with seeing IPv6 notation in our world, the better off we’ll be in the long run.  Plus, it gives me a tag that I can use to show how important IPv6 is to me.

A shorter set of goals this year doesn’t mean a more modest one.  Focus is a good thing in the long run for me.  Being an agent of change when it comes to IPv6 is something that I’m passionate about.  Sure, I’m still going to make the occasional NAT post.  I may even have some unnice things to say about vendors and IPv6 support.  The overall idea is that we keep the discussion focused on moving forward and making IPv6 more widely adopted.  It’s the least I can do to try and leave my mark on the Internet in some other way besides posting cat pictures or snarky memes.  It’s also a goal that is going to keep progressing and never really be finished until the lights are turned out on the last IPv4 webserver out there.  Until that fateful day, here’s hoping that ::2013 is a good year for all.

Unlearning IPv4

"You must unlearn what you have learned." -Yoda

“You must unlearn what you have learned.” -Yoda

As network rock stars, we’ve all spent the majority of our careers learning how to do things.  We learn how to address interfaces and configure routing protocols.  For many of those out there, technology has changed often enough that we often find ourselves need to retrain.  Whether it be a new version of an old protocol or an entirely new way of thinking about things, there will always come a time when it’s necessary to pick up new knowledge.  However, in the case of updated knowledge it’s often difficult to process.  That’s because the old way of doing things interposes itself in our brains while we’re learning to do it the new way.  How many times have you been practicing something only to hear a little voice in the back of your head saying, “That’s not right.  You should be doing it this way.”  In many ways, it’s like trying to reprogram the pathway in your brain that leads to the correct solution to your problem.

This is very apparent to me when it comes to learning how to configure and setup IPv6 on a network.  Those just starting out in the big wide world of IPv6 need to have some kind of reference point to start configuring things, so they tend to lean back on their IPv4 training in order to get started.  This can work for some applications.  For others, though, it can be quite detrimental to getting IPv6 running the way it should.  Instead of carrying forward the old way of doing things because “that’s just the way they should be done,” you need to start unlearning IPv4.  The little green guy in Empire Strikes Back hit the nail on the head.  The whiney farm boy had spent so much of his life convinced that something was impossible that he couldn’t conceive that someone could lift his starship out of a swamp with the Force.  He had to unlearn that lifting things with his mind was impossible.  Once you take that little step, nothing can stop you from accomplishing anything.

With that in mind, here are a few things that need to be unlearned from our days working with IPv4.  Note that this won’t be easy.  But nothing worth doing is ever easy.

Address Conservation – This one is the biggest stumbling block today.  Look at all the discussion we’ve got around point-to-point links and whether to address them with a /64 or a /127 bit mask.  People claim that addressing this link with a /64 wastes addresses.  To  quote the old guy in the desert from Star Wars, “It’s true, depending on your point of view.”  In a given /64, there are approximately 18 quadrillion addresses available (I’m rounding to make the math easy).  If you address a point-to-point link with a /64, you’re only going to be using 0.0000000000000000001% of those addresses (thats 1 * 10^-19).  To many, that’s a pretty big waste.  But with numbers that big, your frame of reference gets screwed up.  By example, take a subnet with 4,094 hosts, which today would need /20 in IPv4.  That’s about the biggest single subnet I can imagine creating.  If you address that 4,094 host subnet with a /64 in IPv6, you’d end up using 0.0000000000000002% (2 * 10^-16) of the address space.  Waste is all a matter of perspective.  On the other hand, by addressing a link with a bit mask beyond a /64, we break neighbor discovery and secure neighbor discovery and PIM sparse mode with embedded RP among other things.  We need to unlearn the address conservation mentality and instead concentrate on making our networks easier to configure and manage.

Memorizing IP addresses – I’m guilty of this.  I spend a lot of time working at the command line with IPv4, whether it be via telnet or SSH or even just plugging numbers into a GUI.  My CUCM systems are setup to use IP only.  I memorize the addresses of my servers, or in many cases try to make this as similar mnemonically to other systems to jog my memory about where to find them in IP space.  In IPv6, memorizing addresses is going to be impossible.  It’s hard enough for me to remember non-RFC1918 address space as it is with 4 octets of decimal numbers.  Now quadruple that and add in hex addressing.  And when it comes to workstations with SLAAC or DHCPv6 assigned addresses?  Forget about it.  Rather than memorizing address space, we’re going to need to start using DNS for communications between endpoints.  Yes, that means setting up DNS for all your routers and CUCM servers too.  It’s going to be a lot of extra work up front.  It’ll pay off in the long run, though.  I’m sure you’d much rather refer to CUCM1.local rather than trying to remember fe80::ba8d:12ff:fe0b:8aff every time you want to get to the phone server.

Subnet Masks – Never again will you need to see 255 in an IPv6 address unless it’s part of the address.  Subnet masking is dead and buried.  Instead, bit masks and slash notation rule the day.  This is going to be one of the most welcome changes in IPv6, but I think it’s going to take a long time to unlearn.  Not really as much for network engineers, but mainly for the people that have ancillary involvement with networking, such as the server people.  Think about the number of server admins that you’ve talked to that have memorized that the subnet mask of their network card is  Now, ask them what that means. Odds are good they can’t tell you.  Worse, some of them might say that it’s a Class C subnet mask.  It’s a little piece of anecdotal information that they heard once when the network folks were talking that they just picked up.  Granted, most of the time the servers are going to be addresses with a /64 bit mask on the IPv6 address.  That’s still going to take a while to explain to the non-networking people.  No, you don’t need any more 255s in your address.  Yes, the /64 is the same as that, sort of.  No, there’s math involved.  Yes, I’ll take care of all the math.

Ships in the Night – As I said on my recent appearance on the Class C block podcast, I think it’s high time that networking vendors stop treating IPv4 and IPv6 like they are separate entities.  I know that I’ve spent the better part of this blog post talking about how IPv4 and IPv6 require a difference in application and not carrying across old habits and conventions.  The two protocols are more alike that they are different.  That means that we need to stop thinking of IPv6 as an afterthought.  Take a look at the CCIE.  There’s still a separate section for IPv6.  It feels like it was just a piece that was added on to the end of the exam instead of being integrated into the core lab.  Look at Kurt Bales’ review of the JNCIE lab that he took.  Specifically, the last bullet point.  You could be asked to configure something on either IPv4 or IPv6, or even both!  Juniper understands that the people taking the JNCIE today aren’t going to have the luxury of concentrating on just IPv4.  The world is going to require us to use IPv6, so I think it’s only fair that our certification programs start doing the same.  IPv6 should be integrated into every level of certification from CCNA/JNCIA all the way up to CCIE/JNCIE.

Tom’s Take

Working with IPv6 is a big change from the way we’ve done things in the past.  With SLAAC and integrated IPSec, the designers have done a great job of making our lives easier with things that we’ve needed for a long time.  However, we’re doing our best to preclude our transition to IPv6 by carrying over a lot of baggage from IPv4.  I know that our brains look for patterns and like to settle on familiarity as a way to help train for new challenges.  If we aren’t careful, we’re going to carry over too much of the old familiar networking and make IPv6 difficult to work with.  Unlearning what we think we know about networking is a good first step.  A person may learn something quickly with familiarity, but they can learn even faster when they approach it with a blank slate and a keen interest to learn.  With that approach, even the impossible won’t keep you from succeeding.

The Five Stages of IPv6 and NAT

I think it’s time to put up a new post on IPv6 and NAT.  Mainly because I’m still getting comments on my old NAT66 post from last year.  I figured it would be nice to create a new place for people to start telling me how necessary NAT is for the Internet of the future.

In the interim, though, I finally had a chance to attend the Texas IPv6 Task Force Winter 2012 meeting.  I got to hear wonderful presentations from luminaries such as John Curran of ARIN, Owen DeLong of Hurricane Electric, and even Jeff Doyle of Routing TCP/IP book fame.  There was a lot of great discussion about IPv6 and the direction that we need to be steering adoption of the new address paradigm.  I also got some very interesting background about the formation of IPv6.  When RFC 1550 was written to start soliciting ideas about a new version of IP, the Internet was a much different place.  Tim Berners-Lee was just beginning to experiment with HTTP.  The majority of computers connected to the Internet used FTP and Telnet.  Protocols that we take for granted today didn’t exist.  I knew IPSec was a creation of the IPv6 working group.  But I didn’t know that DHCP wasn’t created yet (RFC 2131).  Guess what?  NAT wasn’t created yet either (RFC 1631).  Granted, the IPng (IPv6) informational RFC 1669 was published after NAT was created, but NAT as we know and use it today wasn’t really formalized until RFC 2663.  That’s right, folks.

The reason NAT66 doesn’t exist is because IPv6 was built at a time when NAT didn’t exist.

It’s like someone turned on a lightbulb.  That’s why NAT66 has always felt so wrong to me. Because the people that created IPv6 had no need for something that didn’t exist.  IPv6 was about creating a new protocol with advanced features like automatic address configuration and automatic network detection and assignment.  I mean, take a look at the two IPv6 numbering methods.  Stateless Automatic Autoconfiguration (SLAAC) can assign all manner of network information to a host.  I can provide prefixes and gateways and even default routes.  However, the one thing that I can’t provide in basic SLAAC is a DNS server entry.  In fact, I can’t provide any of the commonly assigned DHCP options, such as NTP server or other vendor-specific fields.  SLAAC is focused solely on helping hosts assign addresses to themselves and get basic IP connectivity to the global Internet.  Now, take DHCPv6.  This stateful protocol can keep track of options like DNS server or NTP server.  It can also provide a database of assignments so I know which machine has which IP.  But you know what critical piece of information it can’t provide?  A default router.  That’s right, DHCPv6 has no method of assigning a default router or gateway to an end node.  I’m sure that’s due to the designers of DHCPv6 knowing that SLAAC and router advertisements (RA) handle the network portion of things.  The two protocols need to work together to get hosts onto the internet.  In 1995, that was some pretty advanced stuff.  Today, we think auto addressing and network prefix assignment is pretty passé.

Instead of concentrating on solving the dilemma of increasing the adoption rate of IPv6 past the 1% mark where it currently resides, we’ve instead turned to the Anger and Bargaining phases of the Küber-Ross model, otherwise known as the Five Stages of Grief. The need for IPv6 can no longer be denied.  The reality of running out of IPv4 addresses is upon us.  Instead, we lash out against that which we don’t understand or threatens us.  IPv6 isn’t ready for real networking.  There are security risks.  End-to-end communications aren’t important.  IPv6 is too expensive to maintain.  People aren’t smart enough to implement it.  Any of those sound familiar?  Maybe not those exact words, but I’ve heard arguments very similar to that leveled at IPv6 in just the two short years that I’ve been writing.  Think about how John Curran of ARIN must feel twenty years after he started working on the protocol.

Anger is something I can handle.  Getting yelled at or called expletives is all part of networking.  It’s the Bargaining phase that scares me.  Now, armed with a quiver of use cases that perhaps 5% of the population will ever take advantage of, we now must delay adoption or move to something entirely different to support those use cases.  It’s the equivalent of being afraid to jump off a diving board because there is a possibility that the water will drain out of the pool on the way down.  The most diabolical is Carrier Grade NAT.  Let’s NAT our NATed networks to keep IPv4 around just a little longer.  It won’t cause that many problems, really.  After all, we’ve only got 65,536 ports that we can assign for any given PAT setup.  So if we take that limit and extend it yet another level, we have 65,536 PATed PAT translations that we can assign per CGN gateway.  That has real potential to break applications, and not just from an end-to-end connectivity point of view. To prove my point, fire up any connection manager and go to  See how many separate connection requests are spawned when those map tiles start loading.  Now, imagine what would happen if you could only load ten or fifteen of them.  There’s going to be a lot of blank spots on the that map.

Now, for the fun part.  I’ve been accused of hating NAT.  Yes, it’s true.  I dislike any protocol that breaks basic connectivity and causes headaches for troubleshooting and end-to-end communications.  I have to live with it in IPv4.  I’d rather not see it carried forward.  That’s the feeling of many IPv6 evangelists.  If you think I dislike NAT, ask Owen DeLong his feelings on the subject.  However, to say that I dislike NAT for no good reason is silly.  People are angry at me for saying the emperor has no clothes.  Every time I discuss the lack of need for NAT66, the same argument gets thrown in my face.  Ivan Pepelnjak wrote an article about about using network prefix translation (NPT) in a very specific case.  If you are multihoming your network to two different providers and not using BGP then a case for NPT can be made.  It’s not the best solution, but it’s the easiest.  Much like Godwin’s Law, as the length of any NAT66 argument increases, the probability of someone bringing up Ivan’s article approaches approaches one.

So, I’ve found a solution to the problem.  I’m going to fix this one scenario.  I’m going to dedicate my time to solving the multihoming without BGP issue.  When I do that, I expect choirs of angels to sing and a chariot pulled by unicorns to arrive at my home to escort me to my new position of Savior of IPv6 Adoption.  More realistically, I expect someone else to find a corner case rationale for why IPv6 isn’t the answer.  Of course, that’s just another attempt at bargaining.  By that point, I’ll have enough free time to solve the next issue.  Until then, I suggest the following course of action:

Start Menus and NAT – An Experiment

Fresh off my recent fame from my NAT66 articles (older and newer), I decided first thing Monday morning that a little experiment was in order.  I wanted to express my displeasure for sullying something like IPv6 with something I consider to be at best a bad idea.  The only thing I could come up with was this:

The response was interesting to say the least.  Questions were raised.  Some asked if I was playing a late April Fools joke.  Others rounded up the pitchforks and torches and threatened to burn down my house if I didn’t recant on the spot.  Mostly though, people made sure to express their displeasure by educating me to the fact that I should use something else to do what I wanted rather than rely on the tried-and-true metaphor of a Start Menu.

Now do you see what I’m talking about with NAT66?  Some people think that NAT is needed not because it’s a technological necessity.  Not because it’s solving fifteen problems that IPv6 has right now.  They want NAT because they really don’t understand how things work in IPv6.  It’s the same as bolting a Start Menu on to OS X.  When I started using my new MacBook a few months ago, I took the time to figure out how to use things like Spotlight and Alfred.  They weren’t my Start Menu, but they worked almost the same way (in many cases better).  I didn’t protest the lack of a metaphor I clearly didn’t need.  I adapted and overcame.  And in the end I found myself happier because I found something that worked better than I had hoped.

In much the same way, people that crave NAT on IPv6 are just looking for familiar metaphors for addressing.  I’m going to cast aside the multihoming argument right now because we’ve done that one to death.  Yes, it exists.  Yes, it needs to be addressed.  Yes, NPT is the best solution we’ve got right now.  However, when I started going through all the comments on my NAT66 blog post after the link from the Register article, I noticed that some of the commenters weren’t entirely sure how IPv6 worked.  They did understand that the addresses being assigned to the adapters were globally routable.  But some seemed to believe that a globally routable address meant that every device was going to need a firewall along with DDoS protection and ruleset monitoring.  Besides the fact that every OS has had a firewall since 2002, let me ask one question.  Are you tearing out your WAN firewall when you move to IPv6?  Because as far as I know, you still one have one (maybe two) WAN connections that are terminated on some device.  That could be a router or a firewall.  In the IPv4 world, that device is doing NAT in addition to controlling which devices on the outside can talk to the inside.  Configuring a service to traverse the firewall is generally a two-stage process today.  You must configure a static NAT entry for the device in question and then allow one or more ports to pass through the firewall.  It’s not too difficult, but it is time consuming.  In IPv6, with the same firewall and no NAT, there isn’t a need to create a static NAT entry.  You just permit the ports to access the devices on the inside.  No NAT required.  If you don’t want anyone to talk to the devices on the inside, you don’t configure any inbound rules.  Simple as that.  When you need to poke holes in the firewall for things like web servers, email servers, and so on, all you need to do is poke the hole and be done.

Perhaps what we really need to end this NAT issue is wildcard masking for IPv6 addresses in firewalls.  I have no doubt that eventually any simple home network device that support DHCPv4 today will eventually support DHCPv6 or SLAAC in the near future.  As fast as new chipsets are created to increase the processing power we install into small office/home office devices, it’s inevitable that support will come.  But to support the “easy” argument, what we likely need to do is create a field in the firewall that says “Network Address”.  That would be the higher ordered 48 bits of the IPv6 address.  Once it’s plugged in, the hosts will use DHCPv6 or SLAAC to address themselves.  Then, we select the devices from a list based on DNS name and click a couple of checkboxes to allow ports to open for inbound and outbound traffic.  If a customer is forced to change their address allocation, all they need to do is change the “Network Address” field.  Then, software on the backend would script changes to DHCPv6/SLAAC and all the firewall rules.  DNS would update automatically and all would work again.  Perhaps this idea is too far fetched right now and the scripting necessary would be difficult to write at the present time.  But if it answers the “easy” outcry about IPv6 addressing without the need to add NAT to the protocol, I’m all for it.  Who knows?  Maybe Apple will come up with something just this easy.

Tom’s Take

For the record, I really don’t think there needs to be a Start Menu in OS X.  I think Spotlight is a perfectly fine way to launch programs not located on the dock and find files on your computer.  Even alternatives like Alfred and Quicksilver are fine for me.  The point of my tweet and subsequent replies wasn’t meant to advocate for screwing up the UI of OS X.  It was meant to show that while some people think that my distaste for NAT is silly, all it takes is the right combination of silliness to get people up in arms.  To all of you that were quick to jump and offer alternatives and education for my apparent lack of vision, I say that we need to focus effort like that into educating people about how IPv6 works or spend our time figuring out how to remove the roadblocks standing in the way of adoption.  If that means time writing scripting for low-end devices or figuring out easy UI options, so be it.  After all, someone else has already figured out how to create a Start Menu on a Mac:

IPv6, NAT, and the SME – A Response

I think my distaste for NAT is pretty well known by now. I’ve talked time and again about how I believe that NAT is a bad idea, especially where IPv6 is concerned. I’d said my peace and had time for good conversations with my friends Ivan Pepelnjak (@ioshints) and Ed Horley (@ehorley) about the subject. I was content to just wear my “I HATE NAT” t-shirts to conferences and let bygones be bygones. Then, suddenly…

IPv6 Sucks for SMEs – The Register

Some of you have seen my responses before. Maybe you’ve even been amused by them. Coupled with the fact that I tend to lean toward the snarky side of things, I can see where I might come off as a bit of a smart ass. But “belitted?” “chastened?” Wow. Maybe I’ve let my passion blind me to the plight of the Small-to-Medium Enterprise/Business (SME/B) network/server folks. Maybe we really should stop trying to undo years of duct tape patches to networking and embrace the fact that NAT is a great thing because it allows the little guys to spend less time changing ISPs and deciding to renumber their internal networks on a whim. In fact, I’m even considering calling all my buddies at the IETF and rescinding the whole idea of IPv6. I mean, after all what good is renumbering the Internet if it breaks such a fundamentally important protocol like NAT?

Oh, sorry. I just couldn’t keep a straight face anymore…

In all seriousness, Trevor Pott (@cakeis_not_alie) brings up some very interesting points in his discourse on the impact of IPv6 for the Small-to-Medium Enterprise/Business (SME/B). I’m even willing to admit that we might have glazed over some of the lower-end aspects of what a change like this will mean to people on the razor’s edge of IT. But in the article, the painting of networking professionals as uncaring dictators of fiat laws is almost as silly as characterizing me as a belittling jackass. Yes, I write some pretty pointed posts about NAT and IPv6. Sure, I have a lot of fun playing the heel. But I would hope that my points are made from somewhat sound networking reasoning and not simply blind hatred of NAT and those that use it.  Yes, especially those on the edge of networking.

When I was an intern at IBM Global Services in 2001, I had my first real exposure to the way networking operates. I spent a lot of time configuring static IP addresses on devices and using DHCP on others. I got a real eye opening experience. It even colored my perception of networking for a few years to come, although I didn’t know it at the time. You see, as one of the “old guard” networking companies, IBM has their own registered /8 (Class A) network prefix. Everything inside IBM runs on Apple similarly has These folks have the luxury of a globally routable IP space large enough that they never (realistically) have to worry about running out. For many years afterwards, I couldn’t understand why I was unable to reach my network at home from my university network. It wasn’t until I really started learning more about networking that I realized that RFC1918 existed and NAT was in place to allow ever-growing LANs to have Internet access in absence of registered IP space like I had enjoyed at IBM. As time moved on and I started becoming involved with more and more network services that are affected by NAT, I began to see what IBM’s /8 could offer an enterprise. The flexibility of being static. By having their own IP space, IBM didn’t have to change their address structure to suit the needs of users. They never had to worry about changing ISPs and renumbering their network. Everything just stayed the same and we went on with our lives. But, as Trevor Pott pointed out in the article, IBM and Cisco and Juniper and Apple are enterprises. They aren’t…small.

On the polar opposite end of the scale, we have the little guys. The folks that keep law offices running on a SOHO router/firewall/DHCP server. The accounting offices that can only get a /28 or a /29 IPv4 block from their ISP. Folks that look at duct tape as a solution and not a patch. The “cost conscious” customer as one might say. I can identify with many of these customers because their are my customers in my day job. I’ve had to renumber a publicly addressed network on the fly on a Saturday morning. I’ve had to reconfigure firewalls because the ISP decided to reclaim IP space from a customer. It’s a giant pain in the exhaust port. It’s not glamorous or fun. It’s a necessary evil. But is it a reason to rail against IPv6?

In my previous posts, I talked about the issues with IPv6 on the small end of the scale. Sure, we’ve got a lot of addresses to hand out. We’ve also got a lot of configuration to do. We have to reexamine our stand on firewalls and routes and DNS and a lot of other things. Yes, I will freely admit that this isn’t going to be cheap. I’ve already started building the costs of these analyses into the contracts I sign with my customers for the coming year because I know it will need to be done and I don’t want them to be surprised when they get the message from their provider that the time has come to renumber. But I’ve also got another solution in mind for some of my most “cost conscious” customers and readers. I don’t really want to spill the secret sauce, but here goes:

If it’s going to bother you that much, don’t use IPv6.

Plain and simple in black and white (and red). Unless your ISP is going to start charging you an inordinately high monthly fee to keep an IP block you’ve had for years, don’t change. Stay on IPv4. There’s a whole world out there that is never going to move from IPv4 and be perfectly happy. People who run equipment that will never have enough memory or CPU power to process a naked IPv6 packet (let alone a NATed or NPTed packet). People who are mandated to use translated addresses because of some kind of regulatory oversight like the Payment Card Industry (PCI). I really don’t mean to sound like I’m snorting derisively with this advice. If the additional cost of maintaining an IPv6 network with things like link local addresses and proper DNS resolution and multiple firewall translations isn’t worth it to you and your customer base, then stay where you are. No one will come to your office and put a gun to your head to make you move. The issues we have with address space exhaustion inside enterprises are a wholly different animal than keeping the small office going. Honestly, you folks will stay in business for years to come serving a subset of the Internet populace. There may come a time when there is an  opportunity cost of being able to reach new customers that are IPv6-only, but that cost will either be balanced by the need to trade out your “low cost” equipment for something that will run newer IPv6 features or it will be balanced out by your need to get any business to offset falling revenues due to IPv6-only customers no longer being able to reach your site on IPv4.

If you’re an SME/B network admin that’s still reading this, I’d highly recommend that you take a moment to think about something though. Is IPv6’s insistence on one-to-one communications and move away from NAT really disrupting the way the Internet works? Or is it moving back to the way things were before? Setting right what once went wrong? One of the funny things about information technology that I’ve noticed can best be summed up by a quote from the new Battlestar Galactica, “All of this has happened before. All of this will happen again.” Think about mainframes. We used to do all of our work from a dumb terminal that gave us a window to a large processing system that housed everything. Then we decided we didn’t like that and wanted all the processing power to live locally in minicomputers and client/server architecture. Now, with the advent of things like virtualization and virtual desktop infrastructure (VDI), we’ve once again come back to using dumb terminals to access resources on large central computers. All of this has happened before. And when we get constrained on our big hypervsior/VDI servers, we’ll go right back to decentralized processing in a minicomputer or client/server model once more. All of this will happen again.

In networking, we moved from globally routable address space on all of our nodes to running them all behind a translated boundary. At first we did it to prevent exhaustion of the address pool before a suitable replacement was ready. But as often is the case in networking (and information technology for that matter), we patched something and forgot to really fix the problem. NAT became a convenient crutch that allowed network admins to not have to worry about address renumbering and creating complex (even if appropriate) firewall rules. I’m just as guilty of this as anyone. It was only when I realized that many of the things that I want to do with networking, such as video conferencing, require more effort to work with NAT than they would otherwise. We spent so much time trying to patch things to work with the patch that we forgot what things looked like before the patch. I’d argue that getting back to end-to-end communications to “fix” protocols like SIP and FTP is just as important as anything. Relying on Skype to do VoIP/video communications just because it doesn’t care about NAT and firewalls isn’t good design. It’s just an inexpensive way to avoid the problem for a little longer. The funny thing about IPv6 is that while there is a huge amount of configuration up front and a lot of design work, when things are configured correctly, stuff just works. Absent of a firewall in the middle, I can easily configure an end-to-end connection directly to a system. Before you say that something like that is only important to an enterprise, think about something like remotely supporting a small office. I no longer have to poke holes in firewalls and create one-to-one NAT translations to remotely connect to servers. I can just fire up my RDP client (or your screen sharing tool of choice) and connect easily. No fuss, no muss, and no NAT needed.

I’ve also said before that I now see that there is a use case for Network Prefix Translation (NPT). Ivan has talked about it before and showed me the light from the networking side. Ed Horley has also given me a perspective from the Microsoft side of things that has changed my mind. But exhorting NPT as a solution to all of our NAT problems in IPv6 is like using a butter knife as a screwdriver. NPT was designed to solve one really huge issue – IPv6 multihoming. Announcing address space to two different upstream providers, which is easier to do with NAT in IPv4 than it currently is in IPv6 absent of the solution provided in RFC6296. NPT for multihoming is a good idea in my mind because of the inherent issues with advertising multiple address spaces to different providers and configuring those addresses on all the internal links in an organization. I also believe that NPT is a transition mechanism and will allow us to start “doing it right” down the road when we’ve overcome some of the thinking that we’ve used with IPv4 for so long. One-to-one NAT makes no sense to me in IPv6. Why are you hiding your address? The idea is that the device is reachable, whether it be a web server or a video conferencing unit. Why force a translation at the edge for no apparent reason? Is it because you don’t want to have to re-address your internal network devices?

Absent the aforementioned multihoming issues, let’s talk about renumbering for a second. How often to you really renumber your internal network? At the company that I work for, we’ve done it once in ten years. That’s not because we were forced to. It was because we ran out of space and needed to move from a /24 to a /23 (and now maybe to a /22). We didn’t even renumber half the devices in the network. We just changed a couple of subnet masks and started adding things in new subnets that were created. Now, granted, that was with an RFC1918 private address space internally. However, with SLAAC/DHCPv6 and IPv6, renumbering isn’t that big of a pain. You just change the network ID that is being handed to your end nodes. Thanks to EUI-64 addressing, the host portion of the address won’t change one bit. And Trevor Pott points out in the article that enterprises assume that DNS resolution will take care of the changeover just before he snorts derisively about how no one has managed to make it work yet. I’d argue that he’s more right than he knows. I have the IP addresses of hundreds of customers memorized. Most of them are RFC1918. Some are not. All of them are dotted decimal octets. I know that when I move these customers to IPv6, I will be relying on DNS resolution to reach these end nodes. My days of memorizing IP addresses are most definitely coming to a middle. And for those that might scoff at the ability of a DHCP server to register and maintain a database of DNS-to-host address mappings, you might take a look at what Active Directory has been doing for the last twelve years. I say that because in my experience, many SME/B networks run some form of Microsoft operating systems, even if it is just for directory services.

I’d like to take a moment to talk about “small” enterprises versus “large” enterprises. For most people, the breaking point is usually measure in costs or in devices. As an example, if you have more than 1000 devices, you’re large. If you have less than 50, you’re small. Otherwise, you’re in the middle (medium). Me? I don’t like those definitions. 10,000 devices in a flat Layer 2 network is (relatively) simple. A 10-person shop doing BGP multihoming and DMVPN is more of an enterprise than the previous example. For those networking admins that are running tens or even hundreds of servers, ask yourself what you really consider yourself to be. Are you a small enterprise because you have a Linksys/D-Link/Netgear Swiss Army Box at your edge? Or are you really a medium-to-large enterprise because of what you’re doing with all that horsepower? Now ask yourself if you want your network to be easy to configure because that’s the way networks should be, or is it really because you’re understaffed and running far more infrastructure that you should be? I’m not going to sit here and say that you just throw more people at the problem. That’s never the right answer. In fact, it’s usually the one that gets you laughed at (or worse). Instead, you should examine what you’re doing to see why wholesale renumbering or network changes are even necessary in the first place. One of the main points of the article is that IPv6 will allow network admins to finally be able to create hundreds of VMs on a single physical server and make them reachable from the global Internet. I would counter with the idea that if the only thing truly holding you back from doing that has been address space, the SME/B that you work for has really been a large enterprise wolf in small enterprise sheep’s clothing all along.

Now, if you’re still with me this far you should congratulate yourself. I’ve expounded a lot of thoughts about the technical reasons behind the way IPv6 behaves and why there are difficulties in applying it to the SME/B. I also wrote all that in isolation on an airplane. As soon as I stepped off and got my Internet lifeline back, I checked up on the original article and noticed that Trevor Pott had clarified his original intent at the bottom of the post with a long comment. Being no stranger to this myself, I read on with measured intent. What I came away with galvanized my original thoughts even further. Allow me to restate my original point a little more pointedly:

If “cheap” and “simple” are your two primary design goals, IPv6 probably isn’t for you.

We’ve gone through this whole problem before in the infancy on the Internet. Last year, Vint Cerf gave a talk at Interop about the problem of protocol adoption.  One of the stories I love from this talk involved Mr. Cerf’s attempt to spur the adoption of TCP/IP over the then-dominant NCP protocol. He needed to drive people away from NCP, which wouldn’t scale into the future, and force them to adopt TCP/IP. But adoption rates plateaued quite often as network operators just became comfortable that NCP would always be there to do all the work. Mr. Cerf eventually solved his adoption issues. How did he do it? He turned off NCP for a couple of hours. Then for a day. Then for a week. He drove adoption of a better protocol through sheer force of will and an on/off switch. Now, we all know that we can’t do that today. The Internet is too vital to our global economy to just start shutting things off willy-nilly. Despite that, “cheap” and “simple” aren’t design goals for the Internet core or even the ISP distribution layer. We have to have a protocol that will scale out to support the explosion of connected devices both now and in the coming years. Enterprise providers like Cisco and Juniper and Brocade are leading the charge to provide equipment and services to support this in-state transition. There will be no shutdown of IPv4. This is a steady-state parallel migration to IPv6. These kinds of things don’t come without a cost of some kind. It may not be in the form of a purchase order for a new network core. It may not even be in the form of a service contract to a consultant to help engineer a renumbering and migration plan. The cost may be extra hours reconfiguring servers. It may be taking more time to read RFCs and understand the challenges inherent in reconfiguring the largest single creation in the history of mankind at a fundamental level.

Economies of scale are a good thing. They bring us amazing products every day. They also enable us to spend less time configuring or working and spend more time on creating solutions. The first time you tried to ride a bicycle was probably difficult. As you practiced and progressed it became easier. Soon, you could ride a bike without thinking about it. You might even be able to ride a bike with holding the handlebars or ride it standing on the seat (I never could). That kind of practice and refinement is what is needed in IPv6. We have to make it work on a large scale first to get the kinks worked out. Every network vendor does this. Yes, even the ones that only sell their wares at the local big box retailer. Once you can make something work on a big scale, you can start winnowing down the pieces that are necessary to make it work on the small scale. That’s where “cheap” and “simple” come from. No magic wand. No easy button. Just hard work and investment of time and money.

Tom’s Take

Spurring us “priestly” networking people to change the way things work is a very valid goal and should be lauded. Doing it by accusing us of being obstinate and condescending is the wrong way to do things. I don’t consider myself to be a member of the Cabal of IETF High Priests. I’m not even a member of the IETF. Or the IEEE. I’m a solutions guy. I take what the academics come up with and I make it work in real life. Yes, much like Trevor Potts, I’m a blogger. I like to take positions on things and write interesting articles. Yes, I lampoon those that would seek to hobble a protocol I have high hopes for with thinking from fifteen years ago for the sake of making things “simple”. I’d rather be spending my time working on ways to reduce the time and effort needed to roll out IPv6 everywhere. I’d rather focus on ways to make it easier to renumber the “hundreds” of VMs I typically see at my local small business. In the end, I want what everyone else wants. I want an Internet that works. I know that it may take the rest of my career to get there. But at the end of the day, if I’m forced to choose between making the best Internet I can for the sake of everyone or making it “cheap” or “simple”, then I’ll sacrifice and pay a little more in time and costs. It’s the least I can do.

IPv6 Wireless Support – The Broadcast Problem

When I was at Wireless Field Day 2, my standard question to all the vendors concerned IPv6 support.  Since I’m a huge proponent of IPv6 and the Internet will be arriving at IPv6 rather soon, I wanted to know what kind of plans the various wireless companies had for their particular flavor of access devices.  Most of the answers were the same: it’s coming…soon.  The generic response of “soon” usually means that there isn’t much demand for it.  It could also mean that there are some tricky technical challenges.  My first thought was about the operating system kernels being run on these access points.  Since most APs run some flavor of BSD/Linux, kernel space can be a premium.  Based on my own experiments trying to load DD-WRT on Linksys wireless routers, I know that the meager amount of memory on these little things can really restrict the feature sets available to network rock stars.  So it was that I went on thinking about this until I had a chance conversation with Matthew Gast (@MatthewSGast) from Aerohive.  Matthew is the chair for the IEEE 802.11 committee.  Yes, that means he’s in charge of running the ship for all the little subletters that drive wireless standards.  I’d say he’s somewhat familiar with wireless.  I spent some time at a party one night talking to him about the challenges of shoehorning IPv6 support into a wireless AP.  His answers were rather enlightening and may have caused one of my brain cells to explode.

Matthew started things off by telling me about wireless keys.  If you pick up Matthew’s book 802.11 Wireless Networks: The Definitive Guideyou can flip over to page 465 to read about the use of keys in wireless.  Keys are used to ensure that all the traffic flying around in the air between clients stays separated.  That’s a rather important thing when you consider how much data gets pushed around via wireless.  However, the frames that carry those keys are limited in the amount of space they have to carry key information.  So some time ago, the architects of 802.11 took a shortcut.  Rather than duplicating key information over and over again for every possible scenario, they decided to make the broadcast key for each wireless client identical.  This saved space in the packet headers and allowed the AP to send broadcasts to all clients connected to the AP.  They relied on the higher layer mechanisms inherent in ARP and layer 3 broadcast control to prune away unnecessary traffic.  Typically, clients will not respond to a broadcast for a different subnet than the one they are attached to.  The major side effect is that clients may hear broadcasts for VLANs for which they are not a member of.  For the most part, this hasn’t been a very big deal.  That is, until IPv6 came about.

Recall, if you will, that IPv6 uses multicast mechanisms to propagate advertisements about neighbor discovery and router advertisement (RA).  In particular, these RAs tell the IPv6-enabled clients about available routers that can be used to exit the local network.  Mulitcast is a purely layer 3 construct.  At layer 2 (and below), multicasts turn into broadcasts.  This is the mechanism that ensures that non-layer 3 aware devices can receive the traffic.  Now, think about the issue above.  Broadcast keys are all the same for clients no matter which VLAN they may be attached to.  Multicast RAs get converted to broadcasts at layer 2.  Starting to see a problem yet?

Let’s say that we have 3 VLANs in a site, VLAN 21, VLAN 42, and VLAN 63.  We are a member of VLAN 63, but we use the same SSID for all 3 VLANs.  If we turn on IPv6 for each of these three VLANs, we now have 3 different devices sending out RAs and SLAAC packets for addressing hosts.  If these multicast packets are converted into broadcast packets for the SSID, all three VLANs are going to see the same broadcast.  The VLAN information is inconsequential to the broadcast key on the AP.  We’re going to see the RAs for the routers in VLAN 21 and VLAN 42 on top of the one in VLAN 63.  All of these are going to get installed as valid exit points off the local network.  As well, the end system may even assign a SLAAC address to itself with a router from a different VLAN.  According to the end system, it heard about all of these networks, so they must all be valid, right?  The system doesn’t know that it won’t have a layer 2 path to them.  Worse yet, if one of those RAs has the best metric for getting off the local LAN, it’s going to be the preferred exit point.  The end system will be unable to communicate with the exit point.  Bummer.

How do we fix this problem?  Well, the current thinking revolves around suppressing the broadcasts at layer 2.  Cisco does this by default in their wireless controllers.  The WLAN controller acts as a DHCP relay and provides proxy ARP while ignoring all other broadcast traffic.  That’s great to prevent the problem from happening right now.  What happens when the problem grows in the future and we can no longer simply ignore these multicast/broadcast packets.  Thankfully, Matthew had the answer for that as well.  In 802.11ac, the new specification for gigabit speed wireless, they’ve overhauled all the old key mechanisms.  No longer will the broadcast key be shared among all clients on the same AP.  Here’s hoping that we can get some 802.11ac clients and APs out there and supported when the time comes to flip the big switch to IPv6.

I’d like to thank Matthew Gast for his help in creating this blog post and pointing out the problems inherent in broadcast key caching.  I’d also like to thank Andrew von Nagy (@revolutionwifi) for translating Matthew’s discussion into terms a non-wireless guy like me can understand.