Q And A Should Include The E

The IT world is cyclical for sure. I’ve seen trends and topics repeating themselves over and over again in my relatively short time here. I find it interesting that we keep solving similar problems over and over again. I also find it fascinating that this particular issue leads to the reason why blogs are so important.

Any Questions?

Questions abound in IT. It’s the nature of the industry. However, it’s not just new questions that we create when technology leaps past us. We keep asking the same questions over and over again. This is the field of study that created the FAQ, remember?

In recent memory, I find the same questions being asked over and over again:

  • What is SDN?
  • How can SDN help me?
  • What makes this different from what we’ve done before?

You’ve probably asked those very same questions. Perhaps you found the answers you were looking for. Perhaps you’re still trying to figure it out. The problem is that those questions are still being asked. The industry should have evolved to the point where the simple questions have been answered with simple answers. Complex questions, or those questions that need more in-depth discussion, should be treated as such. Yes, the question of what SDN really is would take more than a cursory paragraph on a blog, but we should be able to at least answer it with enough specificity to make the user not feel like they been slighted.

Questions will never stop coming in IT. But how should we handle them?

Any Answers?

Questions may abound in IT, but the answers drive IT. People make a career out of being the person with the answers. It’s in all the marketing jargon. It’s why we create blogs. Even though most of my writing in the last year has been focused on industry trends or non-technical focused posts, the top three posts on my blog are still answers to simple questions:

  1. When Is A Trunk Not A Trunk?
  2. Switchport Voice VLAN – What Does It Do?
  3. Why Is My SFP Not Working?

These posts are far and away the most popular. I even saw this a few months ago and it made me smile:

This would make it seem like people are in need of answers. Any blogger can look at the incoming search terms for their blog and see all the things that brought readers to them. People want answers and they will keep looking until they find them. But why?

Explain It

I never understood why people kept searching for answers until I thought about satisfaction. I think Randall Munroe summed up the satisfaction (or lack thereof) angle here:

Who are you, DenverCoder9?!? (Thanks XKCD)
Who are you, DenverCoder9?!? (Thanks XKCD)

People can find answers easily. But they won’t stop looking until they are satisfied with the answer. It’s easy to find people saying things like “That’s not supported” or “RTFM” when you’re looking for an answer to a particularly difficult problem. And if you’ve ever called a tech support line, you know how unfulfilling the unsupported answer can feel.

That’s when explanation comes into play for me. First, an admission: I’m a chronic explainer. If you’ve ever met me and had a conversation with me for more than three minutes, you know I explain things. I talk about comic books and movies and technical topics in more depth than I should. That’s because I want things explained to me. Explaining how OSPF area calculations are done is as important as explaining how Captain America ended up wielding Mjolnir.

Think about the following answers:

This is unsupported.

or

This is unsupported on that platform because the CPU doesn’t have enough horsepower to process the packets in real time. We tried cutting down on the processing time but it just overwhelmed the unit no matter how much we tried. So rather than dealing with poor performance, we marked it as unsupported.

Both answers are technically correct. But the second is much more satisfying because the explanation is there instead of just the distilled answer.

The IT world needs more explanation. We need to know why things work the way they do instead of just getting a response of a few words. The explanation has the keys to understanding the answer to the question in its totality. It prevents us from asking the same questions over and over again. It leaves us fulfilled and ready to seek out the next question that needs to be asked.

Expiring The Internet

Embed from Getty Images

An article came out this week that really made me sigh.  The title was “Six Aging Protocols That Could Cripple The Internet“.  I dove right in, expecting to see how things like Finger were old and needed to be disabled and removed.  Imagine my surprise when I saw things like BGP4 and SMTP on the list.  I really tried not to smack my own forehead as I flipped through the slideshow of how the foundation of the Internet is old and is at risk of meltdown.

If It Ain’t Broke

Engineers love the old adage “If it ain’t broke, don’t fix it!”.  We spend our careers planning and implementing.  We also spend a lot of time not touching things afterwards in order to prevent it from collapsing in a big heap.  Once something is put in place, it tends to stay that way until something necessitates a change.

BGP is a perfect example.  The basics of BGP remain largely the same from when it was first implemented years ago.  BGP4 has been in use since 1994 even though RFC 4271 didn’t officially formalize it until 2006.  It remains a critical part of how the Internet operates.  According to the article, BGP is fundamentally flawed because it’s insecure and trust based.  BGP hijacking has been occurring with more frequency, even as resources to combat it are being hotly debated.  Is BGP to blame for the issue?  Or is it something more deeply rooted?

Don’t Fix It

The issues with BGP and other protocols mentioned in the article, including IPv6, aren’t due to the way the protocol was constructed.  It is due in large part to the humans that implement those protocols.  BGP is still in use in the current insecure form because it works.  And because no one has proposed a simple replacement that accomplishes the goal of fixing all the problems.

Look at IPv6.  It solves the address exhaustion issue.  It solves hierarchical addressing issues.  It restores end-to-end connectivity on the Internet.  And yet adoption numbers still languish in the single digit percentage.  Why?  Is it because IPv6 isn’t technically superior? Or because people don’t want to spend the time to implement it?  It’s expensive.  It’s difficult to learn.  Reconfiguring infrastructures to support new protocols takes time and effort.  Things that are better spent on answering user problems or taking on additional tasks as directed by management that doesn’t care about BGP insecurity until the Internet goes down.

It Hurts When I Do This

Instead of complaining about how protocols are insecure, the solution to the problem should be two fold: First, we need to start building security into protocols and expiring their older, insecure versions.  POODLE exploited SSLv3, an older version that served as a fallback to TLS.  While some old browsers still used SSLv3, the simple easy solution was to disable SSL and force people to upgrade to TLS-capable clients.  In much the same way, protocols like NTP and BGP can be modified to use more security.  Instead of suggesting that people use those versions, architects and engineers need to implement those versions and discourage use of the old insecure protocols by disabling them.  It’s not going to be easy at first.  But as the movement gains momentum, the solution will work.

The next step in the process is to build easy-to-configure replacements.  Bolting security onto a protocol after the fact does stop the bleeding.  But to fix the underlying symptoms, the security needs to be baked into the protocol from the beginning.  But doing this with an entirely new protocol that has no backwards compatibility will be the death of that new protocol.  Just look at how horrible the transition to IPv6 has been.  Lack of an easy transition coupled with no monetary incentive and lack of an imminent problem caused the migration to drag out until the eleventh hour.  And even then there is significant pushback against an issue that can no longer be ignored.

Building the next generation of secure Internet protocols is going to take time and transparent effort.  People need to see what’s going into something to understand why it’s important.  The next generation of engineers needs to understand why things are being built the way they are.  We’re lucky in that many of the people responsible for building the modern Internet are still around.  When asked about limitations in protocols the answer remains remarkably the same – “We never thought it would be around this long.”

The longevity of quick fixes seems to be the real issue.  When the next generation of Internet protocols is built there needs to be a built-in expiration date.  A point-of-no-return beyond which the protocol will cease to function.  And there should be no method for extending the shelf life of a protocol to forestall it’s demise.  In order to ensure that security can’t be compromised we have to resign ourselves to the fact that old things need to be put out to pasture.  And the best way to ensure that new things are put in place to supplant them is to make sure the old things go away on time.


Tom’s Take

The Internet isn’t fundamentally broken.  It’s a collection of things that work well in their roles that maybe have been continued a little longer than necessary.  The probability of an exploit being created for something rises with every passing day it is still in use.  We can solve the issues of the current Internet with some security engineering.  But to make sure the problem never comes back again, we have to make a hard choice to expire protocols on a regular basis.  It will mean work.  It will create strife.  And in the end we’ll all be better for it.

Cisco Just Killed The CLI

DeadCLI

Gallons of virtual ink have been committed to virtual paper in the last few days with regards to Cisco’s lawsuit against Arista Networks.  Some of it is speculating on the posturing by both companies.  Other writers talk about the old market vs. the new market.  Still others look at SDN as a driver.

I didn’t just want to talk about the lawsuit.  Given that Arista has marketed EOS as a “better IOS than IOS” for a while now, I figured Cisco finally decided to bite back.  They are fiercely protective of IOS and they have to be because of the way the trademark laws in the US work.  If you don’t go after people that infringe you lose your standing to do so and invite others to do it as well.  Is Cisco’s timing suspect? One does have to wonder.  Is this about knocking out a competitor? It’s tough to say.  But one thing is sure to me.  Cisco has effectively killed the command line interface (CLI).

“Industry Standards”

EOS is certainly IOS-like.  While it does introduce some unique features (see the NFD3 video here), the command syntax is very much IOS.  That is purposeful.  There are two broad categories of CLIs in the market:

  • IOS-like – EOS, HP Procurve, Brocade, FTOS, etc
  • Not IOS-like – Junos, FortiOS, D-Link OS, etc

What’s funny is that the IOS-like interfaces have always been marketed as such.  Sure, there’s the famous “industry standard” CLI comment, followed by a wink and a nudge.  Everyone knows what OS is being discussed.  It is a plus point for both sides.

The non-Cisco vendors can sell to networking teams by saying that their CLI won’t change.  Everything will be just as easy to configure with just a few minor syntax changes.  Almost like speaking a different dialect of a language.  Cisco gains because more and more engineers become familiar with the IOS syntax.  Down the line, those engineers may choose to buy Cisco based on familiarity with the product.

If you don’t believe that being IOS-like is a strong selling point, take a look PIX and Airespace.  The old PIX OS was transformed into something that looked a lot more like traditional IOS.  In ASA 8.2 they even changed the NAT code to look like IOS.  With Airespace it took a little longer to transform the alien CLI into something IOS-like.  They even lost functionality in doing so, simply to give networking teams an interface that is more friendly to them.  Cisco wants all their devices to run a CLI that is IOS-like.  Junos fans are probably snickering right now.

In calling out Arista for infringing on the “generic command line interface” in patent #7,047,526, Cisco has effectively said that they will start going after companies that copy the IOS interface too well.  This leaves companies in a bit of conundrum.  How can you continue to produce an OS with an “industry standard” CLI and hope that you don’t become popular enough to get noticed by Cisco?  Granted, it seems that all network switching vendors are #2 in the market somehow.  But at what point does being a big enough #2 get the legal hammer brought to bear?  Do you have to be snarky in marketing messages? Attack the 800-pound gorilla enough that you anger them?  Or do you just have to have a wildly successful quarter?

Laid To REST

Instead, what will happen is a tough choice.  Either continue to produce the same CLI year and year and hope that you don’t get noticed or overhaul the whole system.  Those that choose not to play Russian Roulette with the legal system have a further choice to make.  Should we create a new, non-infringing CLI from the ground up? Or scrap the whole idea of a CLI moving forward?  Both of those second choices are going to involve a lot of pain and effort.  One of them has a future.

Rewriting the CLI is a dead-end road.  By the time you’ve finished your Herculean task you’ll find the market has moved on to bigger and better things.  The SDN revolution is about making complex networks easier to program and manage.  Is that going to be accomplished via yet another syntax?  Or will it happen because of REST APIs and programing interfaces?  Given an equal amount of time and effort on both sides, the smart networking company will focus their efforts on scrapping the CLI and building programmability into their devices.  Sure, the 1.0 release is going to sting a little.  It’s going to require a controller and some rough interface conventions.  But building the seeds of a programmable system now means it will be growing while other CLIs are withering on the vine.

It won’t be easy.  It won’t be fun.  And it’s a risk to alienate your existing customer base.  But if your options are to get sued or spend all your effort on a project that will eventually go the way of the dodo your options don’t look all that appealing anyway.  If you’re going to have to go through the upheaval of rewriting something from the ground up, why not choose to do it with an eye to the future?


Tom’s Take

Cisco and Arista won’t be finished for a while.  There will probably be a settlement or a licensing agreement or some kind of capitulation on both sides in a few years time.  But by that point, the fallout from the legal action will have finally finished off the CLI for good.  There’s no sense in gambling that you won’t be the next target of a process server.  The solution will involve innovative thinking, blood, sweat, and tears on the part of your entire development team.  But in the end you’ll have a modern system that works with the new wave of the network.  If nothing else, you can stop relying on the “industry standard” ploy when selling your interface and start telling your customers that you are setting the new standard.

 

Vendor Whitebox Switches – Better Together?

ChocoPeanut

Whitebox switching has moved past the realm of original device manufacturers and has been taken up by traditional networking vendors. Andre Kindness (@AndreKindness) of Forrester recently posted that he fields several calls from his customers every day asking about a particular vendor’s approach to whitebox switching. But what do these vendor offerings look like? And can we predict how a given vendor will address the whitebox market?

Chocolate In My Peanut Butter

Dell was one of the first traditional networking vendors to announce a whitebox switch offering that decoupled the operating system from the switching hardware. Dell offered packages from Cumulus Linux and Big Switch Networks alongside their PowerConnect lineup. This makes sense when you consider that the operating system on the switch has never been the strong suit of Dell. The PowerConnect OS is not very popular with network engineers, being very dissimilar from more popular CLIs such as Cisco IOS and its look-alikes.  Their attempts to capitalize on the popularity of Force Ten OS (FTOS) and adapt it or use on PowerConnect switches has been difficult at best, due to the divide been hardware architecture of the two platforms.

What Dell is very good at is offering hardware at a greatly reduced cost. By utilizing this strength, they can enter the whitebox market successfully by partnering with OS vendors to provide customer options. This also gives them time to adapt FTOS to more switches and attempt to drive acquisition posts down once the port of FTOS to PowerConnect is complete.

Peanut Butter In My Chocolate

What happens when a vendor sees software as their strength? You get an announcement like the one last week from Juniper Networks. Juniper has put a significant amount of time and effort into Junos. The FreeBSD base of the system gives it the adaptability that Cumulus enjoys. Since Juniper sees Junos as a huge advantage, their oath to whitebox switching was to offer hardware that reduces the acquisition cost. Porting Junos to run on the OCP-based OCX1100 allows Juniper to use silicon that is more in line with merchant offering price points. The value to the customer comes from existing experience with Junos allowing for reduced learning time on the new platform.

So how will the rest of the market adopt whitebox switching offerings? HP will likely go the same route as Dell, as their software picture is murky with products split evenly between HP Procurve OS and 3Com/H3C Comware. HP has existing silicon manufacturing facilities that allow for economy of scale to reduce acquisition costs to the customer. Conversely, Brocade will likely leverage existing Vyatta development and investment in projects like OpenDaylight to standardize their whitebox offerings on software while offering OCP-style hardware platforms.

The 800-pound Whitebox Gorilla

And what of Cisco? Cisco had invested significant time and effort into both hardware and software. IOS is being renovated with API access and being ported into containers to broaden the platforms on which it can operate. The Cisco investment in custom silicon development is significant as well, with only the Nexus 3000 and 9000 series using merchant offerings from Broadcom. Their eventual whitebox offering could take any form.

Cisco feels very strongly about keeping IOS and its variants exclusive to Cisco hardware. Given that they sued Arista Networks late last week for patent infringement in EOS, it should be apparent how strongly they feel about IOS. That will be the impetus that pushes them to offering some limited custom silicon that is capable of running third-party operating systems. This allows Cisco to partner closely with one of those developers to ensure peak performance and tight integrations with whatever hardware Cisco includes.  They would likely offer this platform with a bundle of SmartNET support services, recouping the costs of producing the switch with some very high margin services.

The possibility of porting IOS to an OCP-like reference platform is remote at best. A whitebox IOS offering would still carry a high price tag to reflect Cisco R&D and would be priced too high above what customers would be willing to pay for total acquisition cost.  It would also open the door for someone to “port” that version of IOS to run on platforms that it shouldn’t be running on.  At the very least, it will expose Cisco in the market as having too high a price tag on their intellectual property in IOS and give competitors like Juniper and Big Switch ammunition to fight back.


Tom’s Take

When evaluating vendor whitebox offerings, be sure your assessment of the strengths matches theirs. Wide adoption of a given strategy will solidify that approach in the future. Be sure to give feedback to your local account teams and tell them the critical features you need to be supported. That will ensure the vendor has you in mind when the time comes to produce a whitebox offering.  And remember that you always have the option of going your own way.  Nothing says that you have to buy a solution with bundled services from traditional networking vendors.  If you’re willing to fly without a safety net for a while, you can find some great deals on ODM switches and OSes to run on them.

HP Networking – Hitting The Right Notes

HP has quietly been making waves recently with their networking strategies.  They recently showed off their technology around software defined networking (SDN) applications at Interop New York.  Here’s a video:

It would seem that HP has been doing a lot of hard work on the back end with SDN.  So why haven’t we heard about it?

Trumpet and Bugle

HP Networking hasn’t been in the news as much as Cisco and VMware as of late.  When you consider that both of those companies are pushing agendas related to redefining the paradigm of networking around policy and virtualization their trumpeting of those agendas makes total sense.  But even members of the League of Non-Aligned Vendors like Brocade are talking a lot about their SDN strategy with the Vyatta Controller and OpenStack integrations.  Vendors have layers and layers of plans for the “new” networking.  But HP has actually been doing it!  Why haven’t we known until now?

HP has been content to play the role of the bugler to the trumpeters of the bigger organizations.  Rather than talking over and over again about what they are planning on doing, HP waits until they’ve actually done it to talk about it.  It’s a sound strategy.  I love making everything work first and then discussing what you’ve done rather than spending week after week, month after month, talking about a plan that may or may not come to fruition.

The issue with HP is that they need to bugle a little more often to stay afloat in the space.  Only making announcements won’t cut it.  The breakneck pace of innovation and adoption is disrupting the ability of laggard developers to stay afloat.  New technologies are being supplanted by upstarts.  Docker is old news.  Now we’re talking about SocketPlane and Rocket.  You’d be forgiven if you haven’t been keeping up as a blogger or engineer.  But if you’ve missed the boat as a vendor, you’re going to have a hard time treading water.

The Tijuana Brass

How can HP solve their problem?  Technically, they need to keep doing what they’ve been doing all along.  They are making good decisions and innovating around ideas like the HP SDN App Store.  What they need to do it tell more people about it.  Get the word out.  Start some discussions around what you’re doing.  Don’t be afraid to engage.  The more you talk to people about your solutions, the more your name will come up in conversation. You need to be loud and on-key.  Herb Alpert and the Tijuana Brass weren’t popular right away.  It took years of recording and playing before the mainstream “discovered” them and popularized their music.

HP Networking has spent considerable time building SDN infrastructure.  The fact that their are OpenFlow images for a wide variety of their existing switch infrastructure is proof they are concerned about making everything fit together.  Now it’s time to tell the story.  With the impending divestiture of HP’s enterprise businesses from the consumer line, it will be far too easy to get lost in the shuffle of reorganization.  They way to prevent that is to step out and make yourself known.  Write blogs, record podcasts, and interact with the community.  Don’t be afraid to toot your own horn a little.


Disclaimer

HP invited me to attend HP Discover Barcelona as their guest.  They provided travel and lodging expenses during my time in Europe.  They did not require any blog posts or consideration for this invitation, nor where they offered any on my part.  The opinions and analysis expressed herein represents my thoughts alone.