Tag Archives: Gartner

Magical Mechanics

Posted on by
1

If you’re a fan of this blog, you’ve probably read my last post about the new SD-WAN magic quadrant that’s been making the rounds and generating discussion. Some people are smiling that this report places Cisco in an area other than leadership in the SD-WAN space. Others are decrying the report as being unfair and contradictory. I wanted to take another look at it given some new information and some additional thoughts on the results.

Fair and Square

The first thing I wanted to do is make sure that I was completely transparent with the way the Gartner Magic Quadrant (MQ) works. I have a very good idea thanks to a conversation with Andrew Lerner (@Fast_Lerner), who is the Research VP of Networking at Gartner. Andrew was nice enough to clarify my understanding of the MQ and accompanying documentation. I’ll quote him here to make sure I don’t get anything wrong:

In an MQ, we assess the overall vendors’ behavior and offering in the market. Product, service/support sales, marketing, innovation, etc. if a vendor has multiple products in a market and sells them regularly to the enterprise, they are part of the MQ assessment. Viable products are not “excluded”.

As you can see from Andrew’s explanation, the MQ takes into account all the aspects of a company. It’s not just a product. It’s the sales, marketing, and other aspects of the company that give the overall score for a company. So how does Gartner figure out how products and services? That’s where their Critical Capabilities documents come into play. They are focused exclusively on products and services. They don’t take marketing or sales or anything else into account.

According to Andrew, when Gartner did their Critical Capabilities document on Cisco, they looked at Meraki MX and IOS-XE only. Viptela vEdge was not examined. So, the CC documents give us the Gartner overview of technology behind the MQ analysis. While the CC documents are focused solely on the Meraki MX and IOS-XD SD-WAN technology, they are components of the overall analysis in the MQ that was published.

What does that all mean in the long run?

Axis and Allies

In order to break this down a bit further, let’s ignore the actual quadrants in the MQ for the moment. Instead, let’s think about this picture as a graph. One axis of the graph is “Ability to Execute,” or in other words can the company do what they say they’re going to do? The other axis is “Completeness of Vision.” This is a question of how the company has rounded out their understanding of the market forces and direction. I want to use this sample MQ with some labels thrown in to help my readers understand how each axis can affect the ranking a company gets:

So, let’s look at where Cisco was situated on those graphs. They are not in the upper right part of the graph, which is the “good” part according to what most people will tell you when they glance at it. Cisco was ranked almost directly in the middle of the graph. Why would that be?

Let’s look at the Execution axis. Why would Cisco have some issues with execution on SD-WAN? Well, the biggest one is probably the shift to IOS-XE and the issues with code quality. Almost everyone involved deploying IOS-XE has told me that Cisco had significant issues in the early releases. Daniel Dib (@DanielDibSwe) had a great conversation with me about the breakdown between the router code and the controller code a week ago. Here’s the first tweet in the chain:

So, there were issues that have been addressed. But is the code completely stable right now? I can’t say, since I haven’t deployed it. But ask around and see what the common wisdom is. I would be genuinely interested to hear how much better things have gotten in the last six months. But, those code quality issues from months ago are going to be a concern in the report. And you can’t guarantee that every box is going to be running on the latest code. Having issues with stable code is going to impact your ability to execute on your vision.

Now, let’s look at the Completeness of Vision axis. If you reference the above picture you’ll see that the Challengers square represents companies that don’t yet understand the market direction. Given that this was the location that Cisco was placed in (barely), let’s examine why that might be. Let’s start by asking “which Cisco product is best for my SD-WAN solution?” Do you know for sure? Which one are you going to be offered?

In my last post, I said that Cisco had been deemphasizing vEdge deployments in favor of IOS-XE. But I completely forgot about Meraki as an additional offering for SMBs and smaller deployments. Where does Meraki make the most sense? And how big does your network need to be before it outgrows a Meraki deployment? Are you chasing a feature? Or maybe you need a service that isn’t offered natively on the Meraki platform? All of these questions need to be answered when you look at what you’re going to do with SD-WAN.

The other companies that Cisco will tell you are their biggest competitors are probably VMware and Silver Peak. How many SD-WAN platforms do they sell? How about companies ranked closer to Cisco in the MQ like Citrix, CloudGenix, or Versa Networks? How many SD-WAN solutions do they offer? How about HPE, Juniper and Aryaka?

In almost every case, the answer is “one”. Each of these companies have settled on a single solution for SD-WAN or SD-Branch. They don’t split those deployments across different product lines. They may have different sized boxes for things, but they all run the same common software. Can you integrate an IOS-XE SD-WAN appliance into a Meraki deployment? Can you take a Meraki MX and make it work with vEdge?

You may be starting to see that the completeness of Cisco’s vision isn’t lacking in SD-WAN but instead how they’re going to accomplish it. Rather than having one solution that can be scaled to fit all needs, Cisco is choosing to offer two different solutions for SMBs and enterprises. And if you count vEdge as a separate product from IOS-XE, as Cisco has suggested in some of their internal reports, then you have three products! I’m not saying that Cisco doesn’t have a vision. But it really looks like that vision is hazier than it should be.

If Cisco had a unified vision with stable code that was integrated up and down the stack I have no doubts they would have been rated higher. When Cisco was deploying Viptela vEdge as the sole SD-WAN offering they had it was much easier to figure out how everything was going to integrate together. But, just like all transitions, the devil is in the details here as Cisco tries to move to IOS-XE. Code quality is going to be a potential source of problems no matter what. But if you are staking your reputation on moving everyone to a single code base from a different more stable one you had better get it right quickly. Otherwise you’re going to get it counted against you.

Tom’s Take

I really appreciate Andrew Lerner for reaching out regarding my analysis of the MQ. I will admit I wasn’t exactly spot on with the differences between the MQ and the Critical Capabilities documents. But the results are close. The CC analyzes Cisco’s big platforms and tells people how they work. The MQ takes everything as a whole and gives Cisco a ranking of where they stand in the market. Sales numbers aside, do you think Cisco should be a leader in the SD-WAN space? Do you feel that where they are today with IOS-XE and Meraki MX is a complete vision? If you do then you likely won’t care about the MQ either way. But for those that have questions about execution and vision, maybe it’s time to figure out what might have caused Cisco to be right in the middle this time around.

SD-WAN Squares and Perplexing Planes

Posted on by
4

The latest arcane polygon is out in the SD-WAN space. Normally, my fortune telling skills don’t involve geometry. I like to talk to real people about their concerns and their successes. Yes, I know that the gardening people do that too. It’s just that no one really bothers to read their reports and instead makes all their decisions based on boring wall art.

Speaking of which, I’m going to summarize that particular piece of art here. Note this isn’t the same for copyright reasons but close enough for you to get the point:

4D8DB810-3618-44EA-8AA2-99EB7EAA3E45

So, if you can’t tell by the colors here, the big news is that Cisco has slipped out of the top Good part of the polygon and is now in the bottom Bad part (denoted by the red) and is in danger of going out of business and being the laughing stock of the networking community. Well, no, not so much that last part. But their implementation has slipped into the lower part of the quadrant where first-stage startups and cash-strapped companies live and wish they could build something.

Cisco released a report rebutting those claims and it talks about how Viptela is a huge part of their deployments and that they have mindshare and marketshare. But after talking with some colleagues in the networking industry and looking a bit deeper into the issue, I think Cisco has the same problem that Boeing had this year. Their assurances are inconsistent with reality.

A WANderful World

When you deploy Cisco SD-WAN, what exactly are you deploying? In the past, the answer was IWAN. IWAN was very much an initial attempt to create SD-WAN. It wasn’t perfect and it was soon surpassed by other technologies, many of which were built by IWAN team members that left to found their own startups. But Cisco quickly realized they needed to jump in the fray and get some help. That’s when the bought Viptela.

Here’s where things start getting murky. The integration of Viptela SD-WAN has been problematic. The initial sales model after the acquisition was to continue to sell Viptela vEdge to the customer. The plan was then to integrate the software on a Cisco ISR, then to integrate the Viptela software into IOS-XE. They’ve been selling vEdge boxes for a long while and are supporting them still. But the transition plan to IOS-XE is in full effect. The handwriting on the wall is that soon Cisco will only offer IOS-XE SD-WAN for sale, not vEdge.

Flash forward to 2019. A report is released about the impact and forward outlook for SD-WAN. It has a big analyst name attached to it. In that report, they leave out the references to vEdge and instead grade Cisco on their IOS-XE offering which isn’t as mature as vEdge. Or as deployed as vEdge. Or, as stated by even Cisco, as stable as vEdge, at least at first. That means that Cisco is getting graded on their newest offering. Which, for Cisco, means they can’t talk about how widely deployed and stable vEdge is. Why would this particular analyst firm do that?

Same is Same

The common wisdom is that Gartner graded Cisco on the curve that the sales message is that IOS-XE is the way and the future of SD-WAN at Cisco. Why talk about what came before when the new hot thing is going to be IOS-XE? Don’t buy this old vEdge when this new ISR is the best thing ever.

Don’t believe me? Go call your Cisco account team and try to buy a vEdge. I bet you get “upsold” to an ISR. The path forward is to let vEdge fade away. So it only makes sense that you should be grading Cisco on what their plans are, not on what they’ve already sold. I’ll admit that I don’t put together analyst reports with graphics very often, so maybe my thinking is flawed. But if I call your company and they won’t sell me the product that you want me to grade you on, I’m going to guess I shouldn’t grade you on it.

So Cisco is mad that Gartner isn’t grading them on their old solution and is instead looking at their new solution in a vacuum. And since they can’t draw on the sales numbers or the stability of their existing solution that you aren’t going to be able to buy without a fight, they slipped down in the square to a place that doesn’t show them as the 800-lb gorilla. Where have a heard something like this before?

Plane to See

The situation that immediately sprung to mind was the issue with Boeing’s 737-MAX airliner. In a nutshell, Boeing introduced a new airliner with a different engine and configuration that changed its flight profile. Rather than try to get the airframe recertified, which could take months or years, they claimed it was the same as the old one and just updated training manuals. They also didn’t disclose there was a new software program that tried to override the new flight characteristics and that particular “flaw” caused the tragic crashes of two flights full of people.

I’m not trying to compare analyst reports to tragedies in any way. But I do find it curious that companies want their new stuff to be treated just like their old stuff with regards to approvals and regulation and analysis. They know that the new product is going to have issues and concerns but they don’t want you to count those because remember how awesome our old thing was?

Likewise, they don’t want you to count the old problems against them. Like the Pinto or the Edsel from Ford, they don’t want you to think their old issues should matter. Just look at how far we’ve come! But that’s not how these things work. We can’t take the old product into account when trying to figure out how the new one works. We can’t assume the new solution is the same as the old one without testing, no matter how much people would like us to do that. It’s like claiming the Space Shuttle was as good as the Saturn V rocket because it went into space and came from NASA.

If your platform has bugs in the first version, those count against you too. You have to work it all out and realize people are going to remember that instability when they grade your product going forward. Remember that common wisdom says not to install an operating system update until the first service patch. You have to consider that reputation every time you release a patch or an update.

Tom’s Take

The SD-WAN MQ tells me that Cisco isn’t getting any more favors from the analysts. The marketing message not lining up with the install base is the heart of a transition away from hardware and software that Cisco doesn’t own. The hope that they could just smile and shrug their shoulders and hope that no one caught on has been dashed. Instead, Cisco now has to realize their going to have to earn that spot back through good code releases and consistent support and licensing. No one is going to give them any slack with SD-WAN like they have with switches and unified communications. If Cisco thinks that they’re just going to be able to bluff their way through this product transition, that idea just won’t fly.

I Can Fix Gartner

Posted on by
3

MQFix

I’ve made light of my issues with Gartner before. From mild twitching when the name is mentioned to outright physical acts of dismissal. Aneel Lakhani did a great job on an episode of the Packet Pushers dispelling a lot of the bad blood that most people have for Gartner. I listened and my attitude toward them softened somewhat. It wasn’t until recently that I I finally realized that my problem isn’t necessarily with Gartner. It’s with those that use Gartner as a blunt instrument against me. Simply put, Gartner has a perception problem.

Because They Said So

Gartner produces a lot of data about companies in a number technology related spaces. Switches, firewalls, and wireless devices are all subject to ranking and data mining by Gartner analysts. Gartner takes all that data and uses it to give form to a formless part of the industry. They take inquiries from interested companies and produce a simple ranking for them to use as a yardstick for measuring how one cloud hosting provider ranks against another. That’s a good and noble cause. It’s what happens afterwards that shows what data in the wrong hands can do.

Gartner makes their reports available to interested parties for a price. The price covers the cost of the analysts and the research they produce. It’s no different that the work that you or I do. Because this revenue from the reports is such a large percentage of Gartner’s income, the only folks that can afford it are large enterprise customers or vendors. Enterprise customers are unlikely to share that information with anyone outside their organization. Vendors, on the other hand, are more than willing to share that information with interested parties. Provided that those parties offer up their information as a lead generation exercise and the Gartner report is favorable to the company. Vendors that aren’t seen as a leader in their particular slice of the industry aren’t particularly keen on doing any kind of advertising for their competitors. Leaders, on the other hand, are more than willing to let Gartner do their dirty work for them. Often, that conversation goes like this:

Vendor: You should buy our product. We’re the best.
Customer: Why are you the best? Are you the fastest or the lowest cost? Why should I buy your product?
Vendor: We’re the best because Gartner says so.

The only way that users outside the large enterprises see these reports is when a vendor publishes them as the aforementioned lead generation activity. This skews things considerably for a lot of potential buyers. This disparity becomes even more insulting when the club in question is a polygon.

Troubling Trigonometry

Gartner reports typically include a lot of data points. Those data points tell a story about performance, cost, and value. People don’t like reading data point. They like graphs and charts. In order to simplify the data into something visual, Gartner created their Magic Quadrant (MQ). The MQ distills the entire report into four squares of ranking. The MQ is the real issue here. It’s the worst kind of graph. It doesn’t have any labels on either axis. There’s no way to rank the data points without referring to the accompanying report. However, so many readers rarely read the report that the MQ becomes the *only* basis for comparison.

How much better is Company A at service provider routing than Company B? An inch? Half an inch? $2 billion in revenue? $2,000 gross margin? This is the key data that allows the MQ to be built. Would you know where to find it in the report if you had to? Most readers don’t. They take the MQ as the gospel truth and the only source of data. And the vendors love to point out that they are further to the top and right of the quadrant than their competitors. Sometimes, the ranking seems arbitrary. What makes a company be in the middle of the leaders quadrant versus toward the middle of the graph? Are all companies in the leaders quadrant ranked and placed against each other only? Or against all companies outside the quadrant? Details matter.

Assisting the Analysis

Gartner can fix their perception problems. It’s not going to be easy though. They have the same issue as the Consumer’s Union, producer of Consumer Reports. Where the CU publishes a magazine that has no advertising, they use donations and subscription revenues to offset operating costs. You don’t see television or print ads with Consumer Reports reviews pasted all over them. That’s because the Consumer’s Union specifically forbids their inclusion for commercial purposes.

Gartner needs to take a similar approach if they want to fix the issues with how they’re seen by others. Sell all the reports you want to end users that want to know the best firewall to buy. You can even sell those reports to the firewall vendors themselves. But the vendors should be forbidden from using those reports to resell their products. The integrity you gain from that stance may not offset the loss of vendor revenue right away. But it will gain you customers in the long run that will respect your stance refusing the misuse of Gartner reports as 3rd party advertising copy.

Put a small disclaimer at the bottom of every report: “Gartner provides analysis for interested parties only. Any use of this information as a sales tool or advertising instrument is unintended and prohibited.” That shows what the purpose of the report is about as well as discouraging use simply to sell another hundred widgets.

Another idea that might work to dispel advertising usage of the MQ is releasing last year’s report for little to no cost after 12 months.  That way, the small-to-medium enterprises gain access to the information without sacrificing their independence from a particular vendor.  I don’t think there will be any loss of revenue from these reports, as those that typically buy them will do so within 6-8 months of the release.  That will give the vendors very little room to leverage information that should be in the public domain anyway.  If you feel bad for giving that info away, charge a nominal printing fee of $5 or something like that.  Either way, you’ll blunt the advertising advantage quickly and still accomplish your goal of being seen as the leader in information gathering.

Tom’s Take

I don’t have to whinny like a horse every time someone says Gartner. It’s become a bit of legend by now. What I do take umbrage with is vendors using data points intended for customers to rank purchases and declare that the non-labeled graph of those data points is the sole arbiter of winners and losers in the industry. What if your company doesn’t fit neatly into a Magic Quadrant category? It’s hard to call a company like Palo Alto a laggard in traditional firewalls when they have something that is entirely non-traditional. Reader discretion is key. Use the data in the report as your guide, not the pretty pictures with dots all over them. Take that data and fold it into your own analysis. Don’t take anyone’s word for granted. Make your own decisions. Then, give feedback. Tell people what you found and how accurate those Gartner reports were in making your decision. Don’t give your email address to a vendor that wants to harvest it simply to gain access to the latest report that (surprisingly) show them to be the best. When the advertising angle dries up, vendors will stop using Garter to sell their wares. When that day comes, Gartner will have a real opportunity to transcend their current image and become something more. And that’s a fix worth implementing.