Facebook Wedge 100 – The Future of the Data Center?

 

FBLike

Facebook is back in the news again. This time, it’s because of the release of their new Wedge 100 switch into the Open Compute Project (OCP). Wedge was already making headlines when Facebook announced it two years ago. A fast, open sourced 40Gig Top-of-Rack (ToR) switch was huge. Now, Facebook is letting everyone in on the fun of a faster Wedge that has been deployed into production at Facebook data centers as well as being offered for sale through Edgecore Networks, which is itself a division of Accton. Accton has been leading the way in the whitebox switching market and Wedge 100 may be one of the ways it climbs to the top.

Holy Hardware!

Wedge 100 is pretty impressive from the spec sheet. They paid special attention to making sure the modules were expandable, especially for faster CPUs and special purpose devices down the road. That’s possible because Wedge is a highly specialized micro server already. Rather than rearchitecting the guts of the whole thing, Facebook kept the CPU and the monitoring stack and just put newer, faster modules on it to ramp to 32x100Gig connectivity.

12809187_1676340369272065_1831349201_n

As suspected in the above image, Facebook is using Broadcom Tomahawk as the base connectivity in their switch, which isn’t surprising. Tomahawk is the roadmap for all vendors to get to 100Gig. It also means that the downlink connectivity for these switches could conceivably work in 25/50Gig increments. However, given the enormous amount of east/west traffic that Facebook must generate, Facebook has created a server platform they call Yosemite that has 100Gig links as well. Given the probably backplane there, you can imagine the data that’s getting thrown around the data centers.

That’s not all. Omar Baldonado has said that they are looking at going to 400Gig connectivity soon. That’s the kind of mind blowing speed that you see in places like Google and Facebook. Remember that this hardware is built for a specific purpose. They don’t just have elephant flows. They have flows the size of an elephant herd. That’s why they fret about the operating temperature of optics or the rack design they want to use (standard versus Open Racks). Because every little change matters a thousand fold at that scale.

Software For The People

The other exciting announcement from Facebook was on the software front. Of course, FBOSS has been updated to work with Wedge 100. I found it very interesting in the press release that much of the programming in FBOSS went into interoperability with Wedge 40 and with fixing the hardware side of things. This makes some sense when you realize that Facebook didn’t need to spend a lot of time making Wedge 40 interoperate with anything, since it was a wholesale replacement. But Wedge 100 would need to coexist with Wedge 40 as the rollout happens, so making everything play nice is a huge point on the checklist.

The other software announcement that got the community talking was support for third-party operating systems running on Wedge 100. The first one up was Open Network Linux from Big Switch Networks. ONL ran on the original Wedge 40 and now runs on the Wedge 100. This means that if you’re familiar with running BSN OSes on your devices, you can drop in a Wedge 100 in your spine or fabric and be ready to go.

The second exciting announcement about software comes from a new company, Apstra. Apstra announced their entry into OCP and their intent to get their Apstra Operating System (AOS) running on Wedge 100 by next year. That has a big potential impact for Apstra customers that want to deploy these switches down the road. I hope to hear more about this from Apstra during their presentation at Networking Field Day 13 next month.


Tom’s Take

Facebook is blazing a trail for fast ToR switches. They’ve got the technical chops to build what they need and release the designs to the rest of the world to be used for a variety of ideas. Granted, your data center looks nothing like Facebook. But the ideas they are pioneering are having an impact down the line. If Open Rack catches on you may see different ideas in data center standardization. If the Six Pack catches on as a new chassis concept, it’s going to change spines as well.

If you want to get your hands dirty with Wedge, build a new 100Gig pod and buy one from Edgecore. The downlinks can break out into 10Gig and 25Gig links for servers and knowing it can run ONL or Apstra AOS (eventually) gives you some familiar ground to start from. If it runs as fast as they say it does, it may be a better investment right now than waiting for Tomahawk II to come to your favorite vendor.

 

 

Advertisements

Vendor Whitebox Switches – Better Together?

ChocoPeanut

Whitebox switching has moved past the realm of original device manufacturers and has been taken up by traditional networking vendors. Andre Kindness (@AndreKindness) of Forrester recently posted that he fields several calls from his customers every day asking about a particular vendor’s approach to whitebox switching. But what do these vendor offerings look like? And can we predict how a given vendor will address the whitebox market?

Chocolate In My Peanut Butter

Dell was one of the first traditional networking vendors to announce a whitebox switch offering that decoupled the operating system from the switching hardware. Dell offered packages from Cumulus Linux and Big Switch Networks alongside their PowerConnect lineup. This makes sense when you consider that the operating system on the switch has never been the strong suit of Dell. The PowerConnect OS is not very popular with network engineers, being very dissimilar from more popular CLIs such as Cisco IOS and its look-alikes.  Their attempts to capitalize on the popularity of Force Ten OS (FTOS) and adapt it or use on PowerConnect switches has been difficult at best, due to the divide been hardware architecture of the two platforms.

What Dell is very good at is offering hardware at a greatly reduced cost. By utilizing this strength, they can enter the whitebox market successfully by partnering with OS vendors to provide customer options. This also gives them time to adapt FTOS to more switches and attempt to drive acquisition posts down once the port of FTOS to PowerConnect is complete.

Peanut Butter In My Chocolate

What happens when a vendor sees software as their strength? You get an announcement like the one last week from Juniper Networks. Juniper has put a significant amount of time and effort into Junos. The FreeBSD base of the system gives it the adaptability that Cumulus enjoys. Since Juniper sees Junos as a huge advantage, their oath to whitebox switching was to offer hardware that reduces the acquisition cost. Porting Junos to run on the OCP-based OCX1100 allows Juniper to use silicon that is more in line with merchant offering price points. The value to the customer comes from existing experience with Junos allowing for reduced learning time on the new platform.

So how will the rest of the market adopt whitebox switching offerings? HP will likely go the same route as Dell, as their software picture is murky with products split evenly between HP Procurve OS and 3Com/H3C Comware. HP has existing silicon manufacturing facilities that allow for economy of scale to reduce acquisition costs to the customer. Conversely, Brocade will likely leverage existing Vyatta development and investment in projects like OpenDaylight to standardize their whitebox offerings on software while offering OCP-style hardware platforms.

The 800-pound Whitebox Gorilla

And what of Cisco? Cisco had invested significant time and effort into both hardware and software. IOS is being renovated with API access and being ported into containers to broaden the platforms on which it can operate. The Cisco investment in custom silicon development is significant as well, with only the Nexus 3000 and 9000 series using merchant offerings from Broadcom. Their eventual whitebox offering could take any form.

Cisco feels very strongly about keeping IOS and its variants exclusive to Cisco hardware. Given that they sued Arista Networks late last week for patent infringement in EOS, it should be apparent how strongly they feel about IOS. That will be the impetus that pushes them to offering some limited custom silicon that is capable of running third-party operating systems. This allows Cisco to partner closely with one of those developers to ensure peak performance and tight integrations with whatever hardware Cisco includes.  They would likely offer this platform with a bundle of SmartNET support services, recouping the costs of producing the switch with some very high margin services.

The possibility of porting IOS to an OCP-like reference platform is remote at best. A whitebox IOS offering would still carry a high price tag to reflect Cisco R&D and would be priced too high above what customers would be willing to pay for total acquisition cost.  It would also open the door for someone to “port” that version of IOS to run on platforms that it shouldn’t be running on.  At the very least, it will expose Cisco in the market as having too high a price tag on their intellectual property in IOS and give competitors like Juniper and Big Switch ammunition to fight back.


Tom’s Take

When evaluating vendor whitebox offerings, be sure your assessment of the strengths matches theirs. Wide adoption of a given strategy will solidify that approach in the future. Be sure to give feedback to your local account teams and tell them the critical features you need to be supported. That will ensure the vendor has you in mind when the time comes to produce a whitebox offering.  And remember that you always have the option of going your own way.  Nothing says that you have to buy a solution with bundled services from traditional networking vendors.  If you’re willing to fly without a safety net for a while, you can find some great deals on ODM switches and OSes to run on them.

Should Microsoft Buy Big Switch?

MSBSN

Network virtualization is getting more press than ever.  The current trend seems to be pitting the traditional networking companies, like Cisco and Juniper, against the upstarts in the server virtualization companies, like VMware and OpenStack.  To hear the press and analysts talk about it makes one think that these companies represent all there is in the industry.

Whither Microsoft?

One company that seems to have been left out of the conversation is Microsoft.  The stalwarts of Redmond have been turning heads with their rapid pace of innovation to reach parity with VMware’s offerings.  However, when the conversation turns to networking Microsoft is usually left out in the cold.  That’s because their efforts at networking in the past have been…problematic.  They are very service oriented and care little for the world outside their comfortable servers.  That won’t last forever.  VMware will be able to easily shift the conversation away from feature parity with Hyper-V and concentrate on all the networking expertise that it has now that is missing in the competitor.

Microsoft can fix that problem with a small investment.  If you can innovate by building it, you need to buy it.  Microsoft has the cash to buy several startups, even after sinking a load of it into Nokia.  But which SDN-focused company makes the most sense for Microsoft?  I spent a lot of time thinking about this very question and the answer became clear for me:  Microsoft needs to buy Big Switch Networks.

A Window On The Future

Microsoft needs SDN expertise.  They have no current networking experience outside of creating DHCP and DNS services on their platforms.  I mean, did anyone ever use their Network Access Protocol solution as a NAC option?  Microsoft has traditionally created bare bones network constructs to please their server customers.  They think networking is a resource outside their domain, which coincidentally is just how their competitors used to look at it as well.  At least until Martin Casado changed their minds.

Big Switch is a perfect fit for Microsoft.  They have the chops to talk OpenFlow.  Their recent shift away from overlays to software on bare metal would play well as a marketing point against VMware and their “overlays are the best way” message.  They could also help Microsoft do more development on NV-GRE, the also ran to VxLAN.  Ivan Pepelnjak (@IOSHints) was pretty impressed with NV-GRE last December, but it’s dropped of the radar in the wake of VMware embracing VxLAN in NSX.  I think having a bit more development work from the minds at Big Switch would put it back into the minds of some smaller network virtualization companies looking to support something other than the de facto standard.  I know that Big Switch has moved away from the overlay model, but if NV-GRE can easily be adapted to the work Big Switch was doing a few months ago, it would be a great additional offering to the idea of running everything in an SDN-enabled switch OS.

Microsoft will also benefit from the pile of SDN applications that Big Switch has rumored to be sitting around and festering for lack of attention.  Applications like network taps sell Big Switch products now.  With NSX introducing the ideas of integrated load balancers and firewalls into the base product, Big Switch is going to be hard pressed to charge extra for them.  Instead, they’re going to have to go out on a limb and finish developing them past the alpha stage and hope that they are enough to sell more product and recoup the development costs.  With the deep pockets in Redmond, finishing those applications would be a drop in the bucket if it means that the new product can compete directly on an even field with VMware.

Building A Bigger Switch

Big Switch gains in this partnership also.  They get to take some pressure of their overworked development team.  It can’t be easy switching horses in mid-stream, especially when it involves changing your entire outlook on how SDN should be done.  Adding a few dozen more people to the project will allow you to branch out and investigate how integrating software into your ideas could be done.  Big Switch has already done a great job developing Project Floodlight.  Why not let some big brains chew on other ideas in the same vein for a while.

Big Switch could also use the stability of working for an established company.  They have a pretty big target on their backs now that everyone is developing an SDN strategy.  Writing an OS for bare metal switches is going to bring them into contention with Cumulus Networks.  Why not let an OS vendor do some of the heavy lifting?  It would also allow Microsoft’s well established partner program to offer incentives to partners that want to sell white label switches with software from Big Switch to get into networking much more cheaply than before.  Think about federal or educational discounts that Microsoft already gives to customers.  Do you think they’d be excited to see the same kind of consideration when it comes to networking hardware?

Tom’s Take

Little fish either get eaten by bigger ones or they have to be agile enough to avoid being snapped up.  The smartest little fish in the ocean may be the remora.  It survives by attaching itself to a bigger fish and providing a benefit for them both.  The remora gets the protection of not being eaten while also not taking too much from the host.  Microsoft would do well to setup some kind of similar arrangement with Big Switch.  They could fund future development into NV-GRE compatible options, or they just by the company outright.  Both parties get something out of the deal: Microsoft gets the SDN component they need.  Big Switch gets a backer with so much industry clout that they can no longer be dismissed.