Cisco vs. Arista: Shades of Gray

Posted on by
5

CiscoVArista

Yesterday was D-Day for Arista in their fight with Cisco over the SysDB patent. I’ve covered this a bit for Network Computing in the past, but I wanted to cover some new things here and put a bit more opinion into my thoughts.

Cisco Designates The Competition

As the great Stephen Foskett (@SFoskett) says, you always have to punch above your weight. When you are a large company, any attempt to pick on the “little guy” looks bad. When you’re at the top of the market it’s even tougher. If you attempt to fight back against anyone you’re going to legitimize them in the eye of everyone else wanting to take a shot at you.

Cisco has effectively designated Arista as their number one competitor by way of this lawsuit. Arista represents a larger threat that HPE, Brocade, or Juniper. Yes, I agree that it is easy to argue that the infringement constituted a material problem to their business. But at the same time, Cisco very publicly just said that Arista is causing a problem for Cisco. Enough of a problem that Cisco is going to take them to court. Not make Arista license the patent. That’s telling.

Also, Cisco’s route of going through the ITC looks curious. Why not try to get damages in court instead of making the ITC ban them from importing devices? I thought about this for a while and realized that even if there was a court case pending it wouldn’t work to Cisco’s advantage in the short term. Cisco doesn’t just want to prove that Arista is copying patents. They want to hurt Arista. That’s why they don’t want to file an injunction to get the switches banned. That could take years and involve lots of appeals. Instead, the ITC can just simply say “You cannot bring these devices into the country”, which effectively bans them.

Cisco has gotten what it wants short term: Arista is going to have to make changes to SysDB to get around the patent. They are going to have to ramp up domestic production of devices to get around the import ban. Their train of development is disrupted. And Cisco’s general counsel gets to write lots of posts about how they won.

Yet, even if Arista did blatantly copy the SysDB stuff and run with it, now Cisco looks like the 800-pound gorilla stomping on the little guy through the courts. Not by making better products. Not by innovating and making something that eclipses the need for software like this. No, Cisco won by playing the playground game of “You stole my idea and I’m going to tell!!!”

Arista’s Three-Edged Sword

Arista isn’t exactly coming out of this on the moral high ground either. Arista has gotten a black eye from a lot of the quotes being presented as evidence in this case. Ken Duda said that Arista “slavishly copied” Cisco’s CLI. There have been other comments about “secret sauce” and the way that SysDB is used. A few have mentioned to me privately that the copying done by Arista was pretty blatant.

Understanding is a three-edged sword: Your side, their side, and the truth.

Arista’s side is that they didn’t copy anything important. Cisco’s side is that EOS has enough things that have been copied that it should be shut down and burned to the ground. The truth lies somewhere in the middle of it all.

Arista didn’t copy everything from IOS. They hired people who worked on IOS and likely saw things they’d like to implement. Those people took ideas and ran with them to come up with a better solution. Those ideas may or may not have come from things that were worked on at Cisco. But if you hire a bunch of employees from a competitor, how do you ensure that their ideas aren’t coming from something they were supposed to have “forgotten”?

Arista most likely did what any other company in that situation would do: they gambled. Maybe SysDB was more copied that created. But so long as Arista made money and didn’t really become a blip on Cisco radar. That’s telling. Listen to this video, which starts at 4:40 and goes to about 6:40:

Doug Gourlay said something that has stuck with me for the last four years: “Everyone that ever set out to compete against Cisco and said, ‘We’re going to do it and be everything to everyone’ has failed. Utterly.”

Arista knew exactly which market they wanted to attack: 10Gig and 40Gig data center switches. They made the best switch they could with the best software they could and attacked that market with all the force they could muster. But, the gamble would eventually have to either pay off or come due. Arista had to know at some point that a strategy shift would bring them under the crosshairs of Cisco. And Cisco doesn’t forgive if you take what’s theirs. Even if, and I’m quoting from both a Cisco 10-K from 1996 and a 2014 Annual Report:

[It is] not economically practical or even possible to determine in advance whether a product or any of its components infringes or will infringe on the patent rights of others.

So Arista built the best switch they could with the knowledge that some of their software may not have been 100% clean. Maybe they had plans to clean it up later. Or iterate it out of existence. Who knows? Now, Arista has to face up to that choice and make some new ones to keep selling their products. Whether or not they intended to fight the 800-pound gorilla of networking at the start, they certainly stumbled into a fight here.

Tom’s Take

I’m not a lawyer. I don’t even pretend to be one. I do know that the fate of a technology company now rests in the hands of non-technical people that are very good and wringing nuance out of words. Tech people would look at this and shake their heads. Did Arista copy something? Probably? Was is something Cisco wanted copied? Probably not? Should Cisco have unloaded the legal equivalent of a thermonuclear warhead on them? Doubtful.

Cisco is punishing Arista to ensure no one every copies their ideas again. As I said before, the outcome of this case will doom the Command Line Interface. No one is going to want to tangle with Cisco again. Which also means that no one is going to want to develop things along the Cisco way again. Which means Cisco is going to be less relevant in the minds of networking engineers as REST APIs and other programming architectures become more important that remembering to type conf t every time.

Arista will survive. They will make changes that mean their switches will live on for customers. Cisco will survive. They get to blare their trumpets and tell the whole world they vanquished an unworthy foe. But the battle isn’t over yet. And instead of it being fought over patents, it’s going to be fought as the industry moves away from CLI and toward a model that doesn’t favor those who don’t innovate.

Repeat After Me

Posted on by
1

callandresponse

Thanks to Tech Field Day, I fly a lot. As Southwest is my airline of choice and I have status, I tend to find myself sitting the slightly more comfortable exit row seating. One of the things that any air passenger sitting in the exit row knows by heart is the exit row briefing. You must listen to the flight attendant brief you on the exit door operation and the evacuation plan. You are also required to answer with a verbal acknowledgment.

I know that verbal acknowledgment is a federal law. I’ve also seen some people blatantly disregard the need to verbal accept responsibility for their seating choice, leading to some hilarious stories. But it also made me think about why making people talk to you is the best way to make them understand what you’re saying

Sotto Voce

Today’s society full of distractions from auto-play videos on Facebook to Pokemon Go parks at midnight is designed to capture the attention span of a human for a few fleeting seconds. Even playing a mini-trailer before a movie trailer is designed to capture someone’s attention for a moment. That’s fine in a world where distraction is assumed and people try to multitask several different streams of information at once.

People are also competing for noise attention as well. Pleasant voices are everywhere trying to sell us things. High volume voices are trying to cut through the din to sell us even more things. People walk around the headphones in most of the day. Those that don’t pollute what’s left with cute videos that sound like they were recorded in an echo chamber.

This has also led to a larger amount of non-verbal behavior being misinterpreted. I’ve experienced this myself on many occasions. People distracted by a song on their phone or thinking about lyrics in their mind may nod or shake their head in rhythm. If you ask them a question just before the “good part” and they don’t hear you clearly, they may appear to agree or disagree even though they don’t know what they just heard.

Even worse is when you ask someone to do something for you and they agree only to turn around and ask, “What was it you wanted again?” or “Sorry, I didn’t catch that.” It’s become acceptable in society to agree to things without understanding their meaning. This leads to breakdowns in communication and pieces of the job left undone because you assume someone was going to do something when they agreed, yet they agreed and then didn’t understand what they were supposed to do.

Fortississimo!

I’ve found that the most effective way to get someone to understand what you’ve told them is to ask you to repeat it back in their own words. It may sound a bit silly to hear what you just told them, but think about the steps that they must go through:

  • They have to stop for moment and think about what you said.
  • They then have to internalize the concepts so they understand them.
  • They then must repeat back to you those concepts in their own phrasing.

Those three steps mean that the ideas behind what you are stating or asking must be considered for a period of time. It means that the ideas will register and be remembered because they were considered when repeating them back to the speaker.

Think about this in a troubleshooting example. A junior admin is supposed to go down the hall and tell you when a link light comes for port 38. If the admin just nods and doesn’t pay attention, ask them to repeat those instructions back. The admin will need to remember that port 38 is the right port and that they need to wait until the link light is on before saying something. It’s only two pieces of information, but it does require thought and timing. By making the admin repeat the instructions, you make sure they have them down right.

Tom’s Take

Think about all the times recently when someone has repeated something back to you. A food order or an amount of money given to you to pay for something. Perhaps it was a long list of items to accomplish for an event or a task. Repetition is important to internalize things. It builds neural pathways that force the information into longer-term memory. That’s why a couple of seconds of repetition are time well invested.

Networking Needs Information, Not Data

Posted on by
2

GameAfoot

Networking Field Day 12 starts today. There are a lot of great presenters lined up. As I talk to more and more networking companies, it’s becoming obvious that simply moving packets is not the way to go now. Instead, the real sizzle is in telling you all about those packets instead. Not packet inspection but analytics.

Tell Me More, Tell Me More

Ask any networking professional and they’ll tell you that the systems they manage have a wealth of information. SNMP can give you monitoring data for a set of points defined in database files. Other protocols like NetFlow or sFlow can give you more granular data about a particular packet group of data flow in your network. Even more advanced projects like Intel’s Snap are building on the idea of using telemetry to collect disparate data sources and build collection methodologies to do something with them.

The concern that becomes quickly apparent is the overwhelming amount of data being received from all these sources. It reminds me a bit of this scene:

How can you drink from this firehose? Maybe you should be asking if you should instead?

Order From Chaos

Data is useless. We need to perform analysis on it to get information. That’s where a new wave of companies is coming into the networking market. They are building on the frameworks and systems that are aggregating data and presenting it in a way that makes it useful information. Instead of random data points about NetFlow, these solutions tell you that you’ve got a huge problem with outbound traffic of a specific type that is sent at a specific time with a specific payload. The difference is that instead of sorting through data to make sense of it, you’ve got a tool delivering the analysis instead of the raw data.

Sometimes it’s as simple as color-coding lines of Wireshark captures. Resets are bad, so they show up red. Properly torn down connections are good so they are green. You can instantly figure out how good things are going by looking for the colors. That’s analysis from raw data. The real trick in modern networking monitoring is to find a way to analyze and provide context for massive amounts of data that may not have an immediate correlation.

Networking professionals are smart people. They can intuit a lot of potential issues from a given data set. They can make the logical leap to a specific issue given time. What reduces that ability is the sheer amount of things that can go wrong with a particular system and the speed at which those problems must be fixed, especially at scale. A hiccup on one end of the network can be catastrophic on the others if allowed to persist.

Analytics can give us the context we need. It can provide confidence levels for common problems. It can ensure that symptoms are indeed happening above a given baseline or threshold. It can help us narrow the symptoms and potential issues before we even look at the data. Analytics can exclude the impossible while highlighting the more probably causes and outcomes. Analytics can give us peace of mind.

Tom’s Take

Analytics isn’t doing our job for us. Instead, it’s giving us the ability to concentrate. Anyone that spends their time sifting through data to try and find patterns is losing the signal in the noise. Patterns are things that software can find easily. We need to leverage the work being put into network analytics systems to help us track down the issues before they blow up into full problems. We need to apply the thing that makes network professionals the best suited to look at the best information we can gather about a situation. Our concentration on what matters is where our job will be in five years. Let’s take the knowledge we have and apply it.

The People Versus Security

Posted on by
Reply

PinkLock

It all comes back to people. People are the users of the system. They are the source of great imagination and great innovation. They are also the reason why security professionals pull their hair out day in and day out. Because computer systems don’t have the capability to bypass, invalidated, and otherwise screw up security quite like a living, breathing human being.

Climb Every Mountain

Security is designed to make us feel safe. Door locks keep out casual prowlers. Alarm systems alert us when our home or business is violated. That warm fuzzy feeling we get when we know the locks are engaged and we are truly secure is one of bliss.

But when security gets in our way, it’s annoying. Think of all the things in your life that would be easier if people just stopped trying to make you secure. Airport security is the first that comes to mind. Or the annoying habit of needing to show your ID when you make a credit card purchase. How about systems that scan your email for data loss prevention (DLP) purposes and kick back emails with sensitive data that you absolutely need to share?

Security only benefits us when it’s unobtrusive yet visibly reassuring. People want security that works yet doesn’t get in their way. And when it does, they will go out of their way to do anything they can to bypass it. Some of the most elaborate procedures I’ve ever seen to get around security lockouts happened because people pushed back against the system.

Cases in point? The US Air Force was forced to put a code on nuclear missiles to protect them from being accidentally launched at the height of the cold war. What did they make that code? 00000000. No, really. How about the more recent spate of issues with the US transition to Chip-and-Signature credit card authentication as opposed to the old swipe method? Just today I was confronted with a card reader that had a piece of paper shoved in the chip reader slot saying “Please Swipe”. Reportedly it’s because transactions are taking 10 seconds or more to process. Much more secure for sure, but far too slow for busy people on the go, I guess.

Computers don’t get imaginative when it comes to overcoming security. They follow the rules. When something happens that violates a rule or triggers a policy to deny an action that policy rule is executed. No exceptions. When an incoming connection is denied at a firewall, that connection is dropped. When the rule says to allow it then it is allowed. Computers are very binary like that (yes, pun intended).

Bring The Mountain To Them

We’ve spent a huge amount of time and effort making security unobtrusive. Think of Apple’s Touch ID. It created a novel and secure way to encourage users to put passcode locks on phones. People can now just unlock their phone with a thumbprint instead of a long passcode. Yet even Touch ID was slow at first. It took some acclimation. And when it was sped up to the point where it caused issues for the way people checked their phones for notifications and such. Apple has even gone to greater lengths in iOS 10 to introduce features to get around the fast Touch ID authentication times caused by new sensors.

Technology will always be one or more steps ahead of where people want it to be. It will always work faster than people think and cause headaches when it behaves in a contrary way. The key to solving security issues related to people is not to try and outsmart them with a computer. People are far too inventive to lose that battle. Even the most computer illiterate person can find a way to bypass a lockout or write a domain administrator password on a sticky note.

Tom’s Take

We need to teach people to think about security from a perspective of need. Why do we have complex passwords? Why do we need to rotate them? Why do the doors of a mantrap open separately? People can understand security when it’s explained in a way that makes them understand the purpose. They still may not like it, but at least they won’t be trying to circumvent it any longer. We hope.

Ten Years of Cisco Live – Community Matters Most of All

Posted on by
4
CLUS2016SignPic

Hey! I made the sign pic this year!

I’ve had a week to get over my Cisco Live hangover this year. I’ve been going to Cisco Live for ten years and been involved in the social community for five of them. And I couldn’t be prouder of what I’ve seen. As the picture above shows, the community is growing by leaps and bounds.

People Are What Matter

TomsCornerSelfie

I was asked many, many times about Tom’s Corner. What was it? Why was it important? Did you really start it? The real answer is that I’m a bit curious. I want to meet people. I want to talk to them and learn their stories. I want to understand what drives people to learn about networking or wireless or fax machines. Talking to a person is one of the best parts of my job, whether it be my Bruce Wayne day job or my Batman night job.

Social media helps us all stay in touch when we aren’t face-to-face, but meeting people in real life is as important too. You know who likes to hug. You find out who tells good stories. Little things matter like finding out how tall someone is in real life. You don’t get that unless you find a way to meet them in person.

FishHug

Hugging Denise Fishburne

Technology changes every day. We change from hardware to software and back again. Routers give way to switches. Fabrics rise. Analytics tell all. But all this technology still has people behind it. Those people make the difference. People learn and grow and change. They figure out how to make SDN work today after learning ISDN and Frame Relay yesterday. They have the power to expand beyond their station and be truly amazing.

Conferences Are Still King

Cisco Live is huge. Almost 30,000 attendees this year. The Mandalay Bay Convention Center was packed to the gills. The World of Solutions took up two entire halls this year. The number of folks coming to the event keeps going up every year. The networking world has turned this show into the biggest thing going on. Just like VMworld, it’s become synonymous with the industry.

People have a desire to learn. They want to know things. They want high quality introductions to content and deep dives into things they want to know inside and out. So long as those sessions are offered at conferences like Cisco Live and Interop people will continue to flock to them. For the shows that assemble content from the community this is an easy proposition. People are going to want to talk where others are willing to listen. For single sourced talks like Cisco Live, it’s very important to identify great speakers like Denise Fishburne (@DeniseFishburne) and Peter Jones (@PeterGJones) and find ways to get them involved. It’s also crucial to listen to feedback from attendees about what did work and what they want to see more of in the coming years.

Keeping The Community Growing

CLUS2016Tweetup

One thing that I’m most proud of is seeing the community grow and grow. I love seeing new faces come in and join the group. This year had people from many different social circles taking part in the Cisco Live community. Reddit’s /r/networking group was there. Kilted Monday happened. Engineering Deathmatches happened. Everywhere you looked, communities were doing great things.

As great as it was to see so many people coming together, it’s just as important to understand that we have to keep the momentum going. Networking doesn’t keep rolling along without new ideas and new people expressing them. Four years ago I could never have guessed the impact that Matt Oswalt (@Mierdin) and Jason Edelman (@JEdelman8) could have had on the networking community. They didn’t start out on top of the world. They fought their way up with new ideas and perspectives. The community adopted what they had to say and ran with it.

We need to keep that going. Not just at Cisco Live either. We need to identify the people doing great things and shining a spotlight on them. Thankfully, my day job affords me an opportunity to do just that. But the whole community needs to be doing it as well. If you can just find one person to tell the world about it’s a win for all of us. Convince a friend to write a blog post. Make a co-worker join Twitter. In the end every new voice is a chance for us all to learn something.

Tom’s Take

As Dennis Leary said in Demolition Man,

I’m no leader. I do what I have to do. Sometimes people come with me.

That’s what Cisco Live is to me. It’s not about a corner or a table or a suite at an event. It’s about people coming together to do things. People talking about work and having a good time. The last five years of Cisco Live have been some of the happiest of my life. More than any other event, I look forward to seeing the community and catching up with old friends. I am thankful to have a job that allows me to go to the event. I’m grateful for a community full of wonderful people that are some of the best and brightest at what they do. For me, Cisco Live is about each of you. The learning and access to Cisco is a huge benefit. But I would go for the people time and time and time again. Thanks for making the fifth year of this community something special to me.

Fixing The CCIE Written – A Follow Up

Posted on by
22

955951_28854808

I stirred up quite the hornet’s nest last week, didn’t I? I posted about how I thought the CCIE Routing and Switching Written Exam needed to be fixed. I got 75 favorites on Twitter and 40 retweets of my post, not to mention the countless people that shared it on a variety of forums and other sites. Since I was at Cisco Live, I had a lot of people coming up to me saying that they agreed with my views. I also had quite a few people that weren’t thrilled with my perspective. Thankfully, I had the chance to sit down with Yusuf Bhaiji, head of the CCIE program, and chat about things. I wanted to share some thoughts here.

Clarity Of Purpose

One of the biggest complaints that I’ve heard is that I was being “malicious” in my post with regards to the CCIE. I was also told that it was a case of “sour grapes” and even that the exam was as hard as it was on purpose because the CCIE is supposed to be hard. Mostly, I felt upset that people were under the impression that my post was designed to destroy, harm, or otherwise defame the CCIE in the eyes of the community. Let me state for the record what my position is:

I still believe the CCIE is the premier certification in networking. I’m happy to be a CCIE and love the program.

Why did I write the post? Not because I couldn’t pass the written. Not because I wanted people to tell me that I was wrong and being mean to them. I wrote the post because I saw a problem and wanted to address it. I felt that the comments being made by so many people that had recently taken the test needed to be collected and discussed. Sure, making light of these kinds of issues in a public forum won’t make people happy. But, as I said to the CCIE team, would you rather know about it or let it fester quietly?

Yusuf assured me that the CCIE program holds itself to the highest standards. All questions are evaluated by three subject matter experts (SMEs) for relevance and correctness before being included in the exam. If those three experts don’t sign off, the question doesn’t go in. There are also quite a few metrics built into the testing software that give the CCIE team feedback on questions and answer choices. Those programs can index all manner of statistics to figure out if questions are creating problems for candidates. Any given test can produce pages worth of valuable information for the people creating the test and trying to keep it relevant.

Another point that was brought up was the comment section on the exam. If you have any problem with a question, you need to fill out the comment form. Yes, I know that taking time out of the test to provide feedback can cause issues. It also interrupts your flow of answering questions. But if you even think for an instant that the question is unfair or misleading or incorrect, you have to leave a detailed comment to make sure the question is flagged properly for review. Which of the following comments means more to you?

  • Trivia question

or

  • This question tests on an obscure command and isn’t valid for a CCIE-level test.

I can promise I know which one is going to be evaluated more closely. And yes, every comment that has purpose is reviewed. The exam creators can print off every comment ever left on a question. The more detailed the comment, the more likely to trigger a review. So please make sure to leave a comment if you think there is a problem with the question.

Clarity Of Vision

Some of the conversations that I had during Cisco Live revolved around the relevance of the questions on the test to a CCIE candidate. Most of the people that I talked to were CCIEs already and using the test for recertification. A few came to me to talk about the relevance of the test questions to candidates that are qualifying for the lab.

While I’m not able to discuss any of the specific plans for the future of the program, I will say that there are ideas in place that could make this distinction matter less. Yusuf told me that the team will be releasing more details as soon as they are confirmed.

The most important point is that the issues that I have with the CCIE Written exam are fixable. I also believe that criticism without a suggestion solution is little more than whining. So I decided to put my money where my mouth is with regard to the CCIE written exam.

I volunteered to fix it.

I stepped up and offered my time as an SME to review the questions on the written exam for relevance, correctness, and grammar. That’s not a light undertaking. There are a ton of questions in the pool that need to be examined. So for every person that agreed with my post or told me that they thought the exam needed to be fixed, I’m putting you all on the spot as well.

It’s time for us as a community of CCIEs to do our part for the exam. Yusuf told me the easiest way to take part in the program is to visit the following URL:

http://www.cisco.com/go/certsme

Sign up for the SME program. Tell them that you want to help fix the CCIE. Maybe you only have to look at 5-10 questions. If the hundred or so people that agreed with me volunteered today, the entire test question pool could be analyzed in a matter of weeks. We could do our part to ensure that people taking the exam have the best possible test in front of them.

But I also challenge you to do more. Don’t just correct grammar or tell them they spelled “electricity” wrong in the question. Challenge them. Ask yourself if this is a question a CCIE candidate should know the answer to. There’s a chance that you could make a difference there. But you can’t do that unless you step up the plate.

Tom’s Take

I had at least ten people tell me that they would do whatever it took to fix the CCIE test last week after I talked to the CCIE cert team. They were excited and hopeful that the issues they saw with the test could be sorted out. I’ll admit that I stepped out on a pretty big limb here by doing this in public as opposed to over email or through official channels. And I do admit that I didn’t clarify my intent to build the program up as opposed to casting the whole exam team and process in a bad light.

Mea culpa.

But, my motivation succeed in getting people to talk about the CCIE written. There are many of you that are ready to do your part to help. Please, go sign up at the link above to join the SME program. Maybe you’ll never look at a single question, Maybe you’ll look at fifty. The point is that you step up and tell Cisco that you’re willing. If even fifteen people come forward and agree to help then that message will sound loud and clear that each and every one of us is proud of being a CCIE and want the program to continue long past the time when we’re retired and telling our grandchildren about the good old days of hard but fair tests.

If you have any questions about participating in the program or you want to reach out to me with your thoughts, don’t hesitate to contact me. Let’s put the power of community behind this!

The CCIE Routing And Switching Written Exam Needs To Be Fixed

Posted on by
29
CCIELogo

The former logo listed in this post was removed by request of Cisco

I’m having a great time at Cisco Live this year talking to networking professionals about the state of things. Most are optimistic about where their jobs are going to fit in with networking and software and the new way of doing things. But there is an undercurrent of dissatisfaction with one of the most fundamental pieces of network training in the world. The discontent is palpable. From what I’ve heard around Las Vegas this week, it’s time to fix the CCIE Written Exam.

Whadda Ya Know?!?

The CCIE written is the bellwether of network training. It’s a chance for network engineers that use Cisco gear to prove they have what it takes to complete a difficult regimen of training to connect networks of impressive size. It’s also a rite of passage to show others that you know how to study, prep, and complete a difficult practical examination without losing your cool. But all that hard work starts with a written test.

The CCIE written has always been a tough test. It’s the only barrier to entry to the CCIE lab. Because the CCIE has never had prerequisites and likely never will due to long standing tradition, the only thing standing in the way of you ability to sit the grueling lab test is a 100 question multiple choice exam that gauges your ability to understand networking at a deep technical level.

But within the last year or so, the latest version of the CCIE written exam has begun to get very bad reviews from all takers of the test. There are quite a few people that have talked about how bad the test is for candidates. Unlike a lot of “sour grapes” cases of people railing against a test they failed, the feedback for the CCIE written is entirely different. It tends to fall into a couple of categories:

The Test Is Poorly Written

The most resounding critique of the exam is that it is a poorly constructed and executed test. The question quality is subpar. There are spelling mistakes throughout and test questions that have poor answer selections. Having spent a large amount of time helping construct the CCNA exam years ago, I can tell you that you will spend the bulk of your time creating wrong answers as distractors to the right ones. Guidelines say that a candidate should have no better than a 25% chance to guess the correct answer from all the choices. If you’ve ever taken a math test that has four multiple choice answers with three being correct for various mistakes in working the problem, you know just how insidious proper distractors can be (and math teachers too).

The CCIE written is riddled with bad distractors according to reports. It also has questions that don’t have a true proper answer or a set of answers that are all technically correct with no way to select them all. That frustrates test takers and makes it very difficult to study for the exam. The editing and test mechanics errors must be rectified quickly in order to restore confidence to the people taking the test.

The Test Doesn’t Cover The Material

Once people stop telling me how bad the test is constructed, they start telling me that the questions are bad on a conceptual level as well. No NDAs are violated during these discussions to protect everyone involved, but the general opinion is that the test has skewed in the wrong direction. Cisco seems to be creating a test that focuses more on the Cisco and less on the Internetworking part of the CCIE.

The test has never been confused for being a vendor-neutral exam. Any look at the blueprint will tell you that there a plenty of proprietary protocols and implementation methods there. But the older versions of the exam did do a good job of teaching you how to build a network that could behave itself with other non-Cisco sections. Redistributing EIGRP and OSPF is a prime example. But the focus of the new exam seems to be skewed toward very specific Cisco proprietary protocols and the minutia around how they operate. I’ve always thought that knowing the hello and dead timers of OSPF NBMA areas is a huge time sink and really only justified for test takers, but I also see why knowing that would be important in multi-vendor operations. But knowing the same thing for an EIGRP DMVPN seems a bit pointless.

The other problem is that, by the admission of most test takers, the current CCIE Written Exam study guide doesn’t cover the areas of the blueprint that are potentially on the test. I feel very sorry for my friend Narbik Kocharians here. He worked very hard to create a study guide that would help test takers pass the exam with the knowledge necessary to do well on the lab. And having a test over a completely different area than his guide makes him look bad in the eyes of testers without good cause. It’s like a college class when the professor tells you to study the book but gives you a test over his or her lectures. It’s not fair because you studied what you were told and failed because they tested something else.

CCIEs Feel There Are Better Recert Options

This is the most damaging problem in my mind. About half the test takers for the CCIE written are candidates looking to qualify for the lab. That requires them to take the written exam for their specific track. But the other half of the test takers are CCIEs that have passed the lab and are looking to recertify. For these professionals, any CCIE written exam is valid for recertification.

Many CCIE candidates look to broaden their horizons by moving to different track to keep their CCIE current while they study for service provider, data center, or even collaboration as a topic area of study. For them, the CCIE is a stepping stone to keep the learning process going. But many CCIEs I’ve spoken to in the past few months are starting to take other exams not because they want to learn new things, but because the CCIE Routing and Switch written exam is such a terrible test.

Quite a few CCIEs are using the CCDE written to recertify. They feel it is a better overall test even though it doesn’t test the material to the level that the CCIE R&S written exam does. They would even be willing to take the chance of getting a question on an area of technology that they know nothing about to avoid having to deal with poor questions in their areas of study. Still more CCIEs are choosing to become Emeritus and “retire” so as to avoid the pain of the written exam. While this has implications for partner status and a host of other challenges for practicing engineers, you have to wonder how bad things must be to make retirement of your CCIE number look like a better option.

Tom’s Take

I took the CCIE R&S written last year at Cisco Live. I was so disgusted with the exam that I immediately switched to the CCDE written and recertified my number while simultaneously vowing never to take the R&S written again. From what I’ve heard this year, the test quality is still slipping with no relief in sight. It’s a sad state of affairs when you realize that the flagship test for Cisco engineers is so horribly broken that those same engineers believe it can’t be fixed. They feel that all the comments and feedback in the world are ignored and their expertise in taking exams is pushed aside for higher cut scores and a more exclusive number of candidates. The dark side of it all is the hope that there isn’t an agenda to push official training materials or other kinds of shortcuts that would help candidates while charging them more and/or locking out third party training providers that work hard to help people study for the lab.

Cisco needs to fix this problem now. They need to listen to feedback and get their written problems under control. If they don’t, they may soon find the only people taking the R&S written test are the same kinds of dumpers and cheaters they think they are trying to keep out with a poorly constructed test.

NOTE: I have published an update to this post here: Fixing The CCIE Written – A Follow Up

The Complexity Conundrum

Posted on by
3

NailPuzzle

Complexity is the enemy of understanding. Think about how much time you spend in your day trying to simplify things. Complexity is the reason why things like Reddit’s Explain Like I’m Five exist. We strive in our daily lives to find ways to simplify the way things are done. Well, except in networking.

Building On Shifting Sands

Networking hasn’t always been a super complex thing. Back when bridges tied together two sections of Ethernet, networking was fairly simple. We’ve spent years trying to make the network do bigger and better things faster with less input. Routing protocols have become more complicated. Network topologies grow and become harder to understand. Protocols do magical things with very little documentation beyond “Pure Freaking Magic”.

Part of this comes from applications. I’ve made my feelings on application development clear. Ivan Pepelnjak had some great comments on this post as well from Steve Chalmers and Derick Winkworth (@CloudToad). I especially like this one:

https://twitter.com/cloudtoad/status/750830744902127616

Derick is right. The application developers have forced us to make networking do more and more faster with less requirement for humans to do the work to meet crazy continuous improvement and continuous development goalposts. Networking, when built properly, is a static object like the electrical grid or a plumbing system. Application developers want it to move and change and breathe with their needs when they need to spin up 10,000 containers for three minutes to run a test or increase bandwidth 100x to support a rollout of a video streaming app or a sticker-based IM program designed to run during a sports championship.

We’ve risen to meet this challenge with what we’ve had to work with. In part, it’s because we don’t like being the scapegoat for every problem in the data center. We tire of sitting next to the storage admins and complaining about the breakneck pace of IT changes. We have embraced software enhancements and tried to find ways to automate, orchestrate, and accelerate. Which is great in theory. But in reality, we’re just covering over the problem.

Abstract Complexity

The solution to our software networking issues seems simple on the surface. Want to automate? Add a layer to abstract away the complexity. Want to build an orchestration system on top of that? Easy to do with another layer of abstraction to tie automation systems together. Want to make it all go faster? Abstract away!

“All problems in computer science can be solved with another layer of indirection.”

This is a quote from Butler Lampson often attributed to David Wheeler. It’s absolutely true. Developers, engineers, and systems builders keep adding layers of abstraction and indirection on top of complex system and proclaiming that everything is now easier because it looks simple. But what happens why the abstraction breaks down?

Automobiles are perfect example of this. Not too many years ago, automobiles were relatively simple things. Sure, internal combustion engines aren’t toys. But most mechanics could disassemble the engine and fix most issues with a wrench and some knowledge. Today’s cars have computers, diagnostics systems, and require lots of lots of dedicated tools to even diagnose the problem, let alone fix it. We’ve traded simplicity and ease of repairability the appearance of “simple” which conceals a huge amount of complexity under the surface.

To refer back to the Lampson/Wheeler quote, the completion of it is, “Except, of course, for the problem of too many indirections.” Even forty years ago it was understood that too many layers of abstraction would eventually lead to problems. We are quickly reaching this point in networking today. With all the reliance on complex tools providing an overwhelming amount of data about every point of the network, we find ourselves forced to use dashboards and data lakes to keep up with the rapid pace of changes dictated to the network by systems integrations being driven by developer desires and not sound network systems thinking.

Networking professionals can’t keep up. Just as other systems now must be maintained by algorithms to keep pace, so too does the network find itself being run by software instead of augmented by it. Even if people wanted to make a change they would be unable to do so because validating those changes manually would cause issues or interactions that could create havoc later on.

Simple Solutions

So how do we fix the issues? Can we just scrap it all and start over? Sadly, the answer here is a resounding “no”. We have to keep moving the network forward to match pace with the rest of IT. But we can do our part to cut down on the amount of complexity and abstraction being created in the process. Documentation is as critical as ever. Engineers and architects need to make sure to write down all the changes they make as well as their proposed designs for adding services and creating new features. Developers writing for the network need to document their APIs and their programs liberally so that troubleshooting and extension are easily accomplished instead of just guessing about what something is or isn’t supposed to be doing.

When the time comes to build something new, instead of trying to plaster over it with an abstraction, we need to break things down into their basic components and understand what we’re trying to accomplish. We need to augment existing systems instead of building new ones on top of the old to make things look easy. When we can extend existing ideas or augment them in such as way as to coexist then we can worry less about hiding problems and more about solving them.

Tom’s Take

Abstraction has a place, just like NAT. It’s when things spiral out of control and hide the very problems we’re trying to fix that it becomes an abomination. Rather than piling things on the top of the issue and trying to hide it away until the inevitable day when everything comes crashing down, we should instead do the opposite. Don’t hide it, expose it instead. Understand the complexity and solve the problem with simplicity. Yes, the solution itself may require some hard thinking and some pretty elegant programming. But in the end that means that you will really understand things and solve the complexity conundrum.

Will Dell Networking Wither Away?

Posted on by
2

chopping-block-Dell-EMC

The behemoth merger of Dell and EMC is nearing conclusion. The first week of August is the target date for the final wrap up of all the financial and legal parts of the acquisition. After that is done, the long task of analyzing product lines and finding a way to reduce complexity and product sprawl begins. We’ve already seen the spin out of Quest and Sonicwall into a separate entity to raise cash for the final stretch of the acquisition. No doubt other storage and compute products are going to face a go/no go decision in the future. But one product line which is in real danger of disappearing is networking.

Whither Whitebox?

The first indicator of the problems with Dell and networking comes from whitebox switching. Dell released OS 10 earlier this year as a way to capitalize on the growing market of free operating systems running on commodity hardware. Right now, OS 10 can run on Dell equipment. In the future, they are hoping to spread it out to whitebox devices. That assumes that soon you’ll see Dell branded OSes running on switches purchased from non-Dell sources booting with ONIE.

Once OS 10 pushes forward, what does that mean for Dell’s hardware business? Dell would naturally want to keep selling devices to customers. Whitebox switches would undercut their ability to offer cheap ports to customers in data center deployments. Rather than give up that opportunity, Dell is positioning themselves to run some form of Dell software on top of that hardware for management purposes, which has always been a strong point for Dell. Losing the hardware means little to Dell if they have to lose profit margin to keep it there in the first place.

The second indicator of networking issues comes from comments from Michael Dell at EMCworld this year. Check out this short video featuring him with outgoing EMC CEO Joe Tucci:

Some of the telling comments in here involve Michael Dell’s praise for the NSX business model and how it is being adopted by a large number of other vendors in the industry. Also telling is their reaffirmation that Cisco is an important partnership in VCE and won’t be going away any time soon. While these two things don’t seem to be related on the surface, they both point to a truth Dell is trying hard to accept.

In the future, with overlay network virtualization models gaining traction in the data center, the underlying hardware will matter little. In almost every case, the hardware choice will come down to one of two options:

  1. Which switch is the cheapest?
  2. Which switch is on the Approved List?

That’s it. That’s the whole decision tree. No one will care what sticker is on the box. They will only care that it didn’t cost a fortune and that they won’t get fired for buying it. That’s bad for companies that aren’t making white boxes or named Cisco. Other network vendors are going to try and add value in some way, but the overlay sitting on top of those bells and whistles will make it next to impossible to differentiate in anything but software. Whether that’s superior management capabilities, open plug-in model, or some other thing we haven’t thought of will make no difference in the end. Software will still be king and the hardware will be an inexpensive pawn or a costly piece that has been pre-approved.

Whither Wireless?

The other big inflection point that makes me worry about the Dell networking story is the lack of movement in the wireless space. Dell has historically been a company to partner first and acquire second. But with HPE’s acquisition of Aruba Networks last year, the dominos in the wireless space are still waiting to fall. Brocade raced out to buy Ruckus. Meru offered itself on a platter to anyone that would buy them. Now Aerohive stands as the last independent wireless vendor without a dance partner. Yes, they’ve announced that they are partnering with Dell, but have you been to the Dell Wireless Networking page? Can you guess what the Dell W-series is? Here’s a hint: it rhymes with “Peruba”.

Every time Dell leads with a W-series deployment, they are effectively paying their biggest competitor. They are opening the door to allowing HPE/Aruba to come in and not only start talking about wireless but servers, storage, and other networking as well. Dell would do well at this point to start deemphasizing the W-series and start highlighting the “new generation” of Aerohive APs and how they are going to the be the focus moving forward.

The real solution would be for Dell to buy a wireless company and take all the wireless expertise they are selling in-house. That would show they are serious about both the campus network of the future and the data center network needed to support their other server and storage infrastructure. Sadly, with Dell being leveraged due to the privatization of his company just two years ago and mounting debt for this mega merger, Dell is looking to make cash with spin offs instead of spending it on yet another company to ingest and subsume. Which means a real non-partner wireless solution is still many years away.

Tom’s Take

Dell’s networking strategy is in maintenance mode. Make switches to support faster speeds for now, probably with Tomahawk support soon, and hope that this whole networking thing goes software sooner rather than later. Otherwise, the need to shore up the campus wireless areas along with the coming decision about showing support fully behind NSX and partnerships is going to be a bitter pill to swallow. Perhaps Dell Networking will exist as an option for companies wanting a 100% Dell solution? Or maybe they are waiting for a new offering from Dell/EMC in the data center to drive profits to research and development to keep pace with Cisco and Arista? One can only hope that their networking flower doesn’t wither on the vine.

DockerCon Thoughts – Secure, Sufficient Applications

Posted on by
1

containerssuspended

I got to spend a couple of days this week at DockerCon and learn a bit more about software containers. I’d always assumed that containers were a slightly different form of virtualization, but thankfully I’ve learned my lesson there. What I did find out about containers gives me a bit of hope about the future of applications and security.

Minimum Viable App

One of the things that made me excited about Docker is that the process isolation idea behind building a container to do one thing has fascinating ramifications for application developers. In the past, we’ve spent out time building servers to do things. We build hardware, boot it with an operating system, and then we install the applications or the components thereof. When we started to virtualize hardware into VMs, the natural progression was to take the hardware resource and turn it into a VM. Thanks to tools that would migrate a physical resource to a virtual one in a single step, most of the first generation VMs were just physical copies of servers. Right down to phantom drivers in the Windows Device Manager.

As we started building infrastructure around the idea of virtualization, we stopped migrating physical boxes and started building completely virtual systems from the ground up. That meant using things like deployment templates, linked clones, and other constructs that couldn’t be done in hardware alone. As time has rolled on, we have a method of quickly deploying virtual resources that we could never do on purely physical devices. We finally figured out how to use virtual platforms efficiently.

Containers are now at the crossroads we saw early on in virtualization. As explained by Mike Coleman (@MikeGColeman), many application developers are starting their container journey by taking an existing app and importing it directly into a container. It’s a bit more involved than the preferred method, but Mike mentioned that even running the entire resource pool in a container does have some advantages. I’m sure the Docker people see container adoption as the first step toward increased market share. Even if it’s a bit clumsy at the start.

The idea then moves toward the breakdown of containers into the necessary pieces, much as it did with virtual machines years ago. Instead of being forced to think about software as a monolithic construct that has to live on a minimum of one operating system, developers can break the system down into application pieces that can execute one program or thread at a time on a container. Applications can be built using the minimum amount of software constructs needed for an individual process. That means that those processes can be spread out and scaled up or down as needed to accomplish goals.

If your database query function is running as a containerized process instead of running on a query platform in a VM then scaling that query to thousands or tens of thousands of instances only requires spinning up new containers instead of new VMs. Likewise, scaling a web-based app to accommodate new users can be accomplished with an explosion of new containers to meet the need. And when the demand dies back down again, the containers can be destroyed and resources returned to the available pool or turned off to save costs.

Segment Isolation

The other exciting thing I saw with containers was the opportunity for security. The new buzzword of the day in security and networking is microsegmentation. VMware is selling it heavily with NSX. Cisco has countered with a similar function in ACI. At the heart of things microsegmentation is simply ensuring that processes that shouldn’t be talking to each other won’t be talking to each other. This prevents exposure by having your app database server visible on the public Internet, for instance.

Microsegmentation is great in overlay and network virtualization systems where we have to take steps to prevent systems from talking to each other. That means policies and safeguards in place to prevent communications. It’s a great way to work on existing networks where the default mode is to let everything on the same subnet talk to everything else. But what if the default was something different.

With containers, there is a sandbox environment for each container to talk to other containers in the same area. If you create a named container network and attach a container to it, that container gains a network interface on that particular named network. It won’t be able to talk to other containers on different networks without creating an explicit connection between the two networks. That means that the default mode of communications for the containers is restricted out of the box.

Imagine how nice it will be to create a network that isn’t insecure by default. Rather than having to unconnected all the things that shouldn’t speak, you can spend your time building connections between the things that should be speaking. That means a little mistake or forgotten connection will prevent communications instead of opening it up. That means much less likelihood that you’re going to cause an incident.

There are still some issues with scaling the networking aspect of Docker right now. The key/value store doesn’t provide a lot of visibility and definitely won’t scale up to tens or hundreds of thousands of connections. My hope is that down the road Docker will implement a more visible solution that can perform drag-and-drop connectivity between containers and leave an audit trail so networking pros can figure out who connected what and how that exposed everything. It also makes it much easier when the connection between the devices has to be explicit to prove intent or malice. But those features are likely to come down the road as Docker builds a bigger, better management platform.

Tom’s Take

I think Docker is doing things right. By making developers look at the core pieces they need to build apps and justify why things are being done the way they’ve always been done, containers are allowing for flexibility and new choices to be made. At the same time, those choices are inherently more secure because resources are only shared when necessary. It’s the natural outgrowth of sandboxing and Jails in the OS from so many years ago. Docker has a chance to make application developers better without making them carry the baggage of years of thinking along with them to a new technology.