Network Consumer Reports

I’m a huge fan of Consumer Reports magazine.  They do a great job of reviewing all manner of products from household appliances to SUVs.  They provide unbiased reviews for all products because they do not accept any outside advertising from companies nor do they accept any free samples from manufactures, instead choosing to purchase all of the items they review outright.  This gives them a substantial amount of credibility in the industry and their opinion has been known to influence the direction of many manufacturers, especially in the automotive arena.

Why is it that reviews in the networking space don’t have the same reputation?  Networking manufacturers are quick to refer to Gartner numbers or Tolly reports that back their equipment as being superior to their competitors.  For the most part, mention of either of these two names around network rock stars brings groans and cat calls.  The general consensus that I get from people I’ve talked to is that many of these reports are simply bought and paid for.  Joe Onisick has a great blog post about talking with the founder of Tolly about this very subject.  Many reports that are sponsored by a company are (suprisingly) critical of the sponsor’s competitors and give favorable reviews to said sponsors.  Not all that shocking when you think about it.  Even discounting the idea that the report could be massaged in favor of the sponsoring company, the odds are good that an unfavorable review would just be buried and never see the light of day.

This pattern of sponsored reports tends to leave the average network rock star jaded and distrustful of any testing that they haven’t done themselves.  Alas, when moving into a new field or testing equipment outside of the comfort zone it becomes quite easy to get lost and being making mistakes or missing key features or options.  Why can’t we do something about that?  Maybe we can…

I’d like to see a Consumer Reports type of service for networking.  It would have to adhere to the same rules that the Consumer’s Union uses for Consumer Reports.  No advertising, which also means that the reports can’t be used by the vendors for the purposes of selling their product.  That means no touting of the latest scores of the newest switches from Network Consumer Reports (NCR).  Also, all the equipment would need to be purchased outright from the vendors or through distributors or value added resellers (VARs).  This would also introduce some difficulties, as many vendors require complex designs before equipment will be sold or require the interaction of a VAR in order ensure the equipment will be installed correctly.  In order to ensure they fairness and impartiality of the tests, these people must be excluded from the configuration process and only be around for purchasing and delivery.  Only members of NCR would be allowed to install and configure the equipment.  Naturally, it’s going to take some skilled people to do that.

When the equipment for a given test or scenario arrives, it will be configured based on best practice guidelines for the vendor/manufacturer.  These practices should be found on the vendor’s website and be easily available.  No shortcuts or undocumented configurations would be allowed at first.  This is to ensure fairness as well as making the vendors responsible for the documentation that is provided to customers.  For a given test, traffic generators would be used to simulate all kinds of traffic patterns in a real world environment.  That would be similar to things that the real Consumer Reports does, like measuring fuel economy themselves rather than relying on the manufacturer’s EPA fuel economy numbers.  I’d rather see numbers I can believe with strict definitions of traffic rather than seeing tests that provide advantages, such as using different packet sizes for throughput versus latency.  Numbers you can trust are very important.

Once the tests are run and the reports have been generated, each vendor will be contacted with the reports and offered a small window of time to “tweak” things.  You have to offer this chance because invariably vendors begin grousing about not having a chance to fix the broken things.  Let’s say they are given 24 hours to modify the base configurations to increase throughput or reduce latency with the same traffic types used in the first test.  After the 24 hours, the tests will be readministered and the results recorded. However, any changes from the best practices will be documented.  If the new, “tweaked” configuration provides additional advantages, the report should then ask why those tweaks are not included in the best practices.  Each vendor will only be able to work on their own equipment and will not be informed of the results of any other vendor’s test.  In fact, they won’t even be informed which other vendors are being tested.  This is to ensure that no one has the opportunity to spread fear, uncertainty, or doubt (FUD) about a different competing solution.  Facts only here, folks.

After all of this, the reports will be published for all to see.  Perhaps there would be some kind of subscription service to reduce the astronomical cost associated with the acquisition and setup of the equipment.  This would only be necessary to avoid the need to rely on angel investors or the independently wealthy to capitalize such a large project.  Once the reports are published, the subscribers can trust in the content and use it however they see fit to begin to plan new projects or purchase equipment.

Tom’s Take

Why is it so hard to find a voice to trust when it comes to network reviews?  Why do I have to constantly ask myself “Who is behind this report?” I never worry about that when I read Consumer Reports.  I can trust the information they provide because I know it isn’t bought and paid for.  It would be wonderful to have something like that in the networking/storage/server space.  I’m sure the people out there right now do decent jobs of reviewing equipment, but none of them are the go-to type of publication like Consumer Reports.  Of course, bringing that kind of reporting to the IT world has a lot of huge challenges. Between getting capitalized and trying to find a way to buy large amounts of gear without raising any fuss from vendors, it would be a large undertaking.  However, if you can provide credibility with your reports and aid people in making good decisions for their businesses, I think you could make a go of it.  Let’s hope that this isn’t a pipe dream sometime down the road.

*Note: Consumer Reports is a trademark of Consumer’s Union and my use of their publications for examples in this post should not infer any kind of endorsement.

8 thoughts on “Network Consumer Reports

  1. So Tom, are you proposing starting a venture. I think one person who would be interesting to put something like this on is the Packet Pushers team. If we networkers can find a way to make sure the Greg and others are paid for the efforts outside vendor funding, I’d be glad to send them to few shekels so they can continue to make a living.

  2. Tom,

    Great thought, I’d love to see a completely unbiased test/analysis firm. You’ll never get worthwhile results from the incumbents, as you state they are funded by the vendors they test. Even with ‘unfunded’ research reports from the likes of Gartner and Forrester they submit pre-release copies to the vendors named and allow for feedback. The vendor feedback isn’t taken lightly in the next draft because all other funding from that vendor is at stake.

  3. Some related examples exist today such as AnandTech and Tom’s Hardware, but I believe they are given their demo units. Outstanding analysis, however. How long did Consumer’s Union exist before they started covering cars? I’d bet they spent quite some time on toasters and shavers before working out the $50k SUV reviews.
    Did you hear that all of their demo units are made available to employees after the testing for very steep discount? I also found it telling that most of the senior staff drives Hondas.

  4. I love the idea, and have bounced it around the confines of my own skull a bit in the past. My issues are the same challenges that you mention: financial. At a glance, getting gear to test, obtaining traffic generators, and having a facility with enough juice to power the big stuff all seems a bit much to bother with. I think it would be hard to monetize without an army of folks ready to pay for the test results. But I still think the idea is a great one. I haven’t been able to shake it.

  5. Pingback: “Smoke and Mirrors” – Questions and RFCs That Help You Interpret Vendor Performance Claims

  6. Pingback: Spirent – Network Field Day 3 | The Networking Nerd

  7. Pingback: Reality As A Service | The Networking Nerd

  8. Pingback: We Are Number Two! | The Networking Nerd

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s