Facebook Wedge 100 – The Future of the Data Center?



Facebook is back in the news again. This time, it’s because of the release of their new Wedge 100 switch into the Open Compute Project (OCP). Wedge was already making headlines when Facebook announced it two years ago. A fast, open sourced 40Gig Top-of-Rack (ToR) switch was huge. Now, Facebook is letting everyone in on the fun of a faster Wedge that has been deployed into production at Facebook data centers as well as being offered for sale through Edgecore Networks, which is itself a division of Accton. Accton has been leading the way in the whitebox switching market and Wedge 100 may be one of the ways it climbs to the top.

Holy Hardware!

Wedge 100 is pretty impressive from the spec sheet. They paid special attention to making sure the modules were expandable, especially for faster CPUs and special purpose devices down the road. That’s possible because Wedge is a highly specialized micro server already. Rather than rearchitecting the guts of the whole thing, Facebook kept the CPU and the monitoring stack and just put newer, faster modules on it to ramp to 32x100Gig connectivity.


As suspected in the above image, Facebook is using Broadcom Tomahawk as the base connectivity in their switch, which isn’t surprising. Tomahawk is the roadmap for all vendors to get to 100Gig. It also means that the downlink connectivity for these switches could conceivably work in 25/50Gig increments. However, given the enormous amount of east/west traffic that Facebook must generate, Facebook has created a server platform they call Yosemite that has 100Gig links as well. Given the probably backplane there, you can imagine the data that’s getting thrown around the data centers.

That’s not all. Omar Baldonado has said that they are looking at going to 400Gig connectivity soon. That’s the kind of mind blowing speed that you see in places like Google and Facebook. Remember that this hardware is built for a specific purpose. They don’t just have elephant flows. They have flows the size of an elephant herd. That’s why they fret about the operating temperature of optics or the rack design they want to use (standard versus Open Racks). Because every little change matters a thousand fold at that scale.

Software For The People

The other exciting announcement from Facebook was on the software front. Of course, FBOSS has been updated to work with Wedge 100. I found it very interesting in the press release that much of the programming in FBOSS went into interoperability with Wedge 40 and with fixing the hardware side of things. This makes some sense when you realize that Facebook didn’t need to spend a lot of time making Wedge 40 interoperate with anything, since it was a wholesale replacement. But Wedge 100 would need to coexist with Wedge 40 as the rollout happens, so making everything play nice is a huge point on the checklist.

The other software announcement that got the community talking was support for third-party operating systems running on Wedge 100. The first one up was Open Network Linux from Big Switch Networks. ONL ran on the original Wedge 40 and now runs on the Wedge 100. This means that if you’re familiar with running BSN OSes on your devices, you can drop in a Wedge 100 in your spine or fabric and be ready to go.

The second exciting announcement about software comes from a new company, Apstra. Apstra announced their entry into OCP and their intent to get their Apstra Operating System (AOS) running on Wedge 100 by next year. That has a big potential impact for Apstra customers that want to deploy these switches down the road. I hope to hear more about this from Apstra during their presentation at Networking Field Day 13 next month.

Tom’s Take

Facebook is blazing a trail for fast ToR switches. They’ve got the technical chops to build what they need and release the designs to the rest of the world to be used for a variety of ideas. Granted, your data center looks nothing like Facebook. But the ideas they are pioneering are having an impact down the line. If Open Rack catches on you may see different ideas in data center standardization. If the Six Pack catches on as a new chassis concept, it’s going to change spines as well.

If you want to get your hands dirty with Wedge, build a new 100Gig pod and buy one from Edgecore. The downlinks can break out into 10Gig and 25Gig links for servers and knowing it can run ONL or Apstra AOS (eventually) gives you some familiar ground to start from. If it runs as fast as they say it does, it may be a better investment right now than waiting for Tomahawk II to come to your favorite vendor.




Is The Blog Dead?

I couldn’t help but notice an article that kept getting tweeted about and linked all over the place last week.  It was a piece by Jason Kottke titled “R.I.P. The Blog, 1997-2013“.  It’s actually a bit of commentary on a longer piece he wrote for the Nieman Journalism Lab called “The Blog Is Dead, Long Live The Blog“.  Kottke talks about how people today are more likely to turn to the various social media channels to spread their message rather than the tried-and-true arena of the blog.

Kottke admits in both pieces that blogging isn’t going away.  He even admits that blogging is going to be his go-to written form for a long time to come.  But the fact that the article spread around like wildfire got me to thinking about why blogging is so important to me.  I didn’t start out as a blogger.  My foray into the greater online world first came through Facebook.  Later, as I decided to make it more professional I turned to Twitter to interact with people.  Blogging wasn’t even the first thing on my mind.  As I started writing though, I realized how important it is to the greater community.  The reason?  Blogging is thought without restriction.

Automatic Filtration

Social media is wonderful for interaction.  It allows you to talk to friends and followers around the world.  I’m still amazed when I have conversations in real time with Aussies and Belgians.  However, social media facilitates these conversations through an immense filtering system.  Sometimes, we aren’t aware of the filters and restrictions placed on our communications.

twitter02_color_128x128Twitter forces users to think in 140-ish characters.  Ideas must be small enough to digest and easily recirculate.  I’ve even caught myself cutting down on thoughts in order to hit the smaller target of being about to put “RT: @networkingnerd” at the begging for tweet attribution.  Part of the reason I started a blog was because I had thoughts that were more than 140 characters long.  The words just flow for some ideas.  There’s no way I could really express myself if I had to make ten or more tweets to express what I was thinking on a subject.  Not to mention that most people on Twitter are conditioned to unfollow prolific tweeters when they start firing off tweet after tweet in rapid succession.

facebook_color02_128x128Facebook is better for longer discussion, but they are worse from the filtering department. The changes to their news feed algorithm this year weren’t the first time that Facebook has tweaked the way that users view their firehose of updates.  They believe in curating a given users feed to display what they think is relevant.  At best this smacks of arrogance.  Why does Facebook think they know what’s more important to me that I do?  Why must my Facebook app always default to Most Important rather than my preferred Most Recent?  Facebook has been searching for a way to monetize their product even before their rocky IPO.  By offering advertisers a prime spot in a user’s news feed, they can guarantee that the ad will be viewed thanks to the heavy handed way that they curate the feed.  As much reach as Facebook has, I can’t trust them to put my posts and articles where they belong for people that want to read what I have to say.

Other social platforms suffer from artificial restriction.  Pinterest is great for those that post with picture and captions or comments.  It’s not the best for me to write long pieces, especially when they aren’t about a craft or a wish list for gifts.  Tumblr is more suited for blogging, but the comment system is geared toward sharing and not constructive discussion.  Add in the fact that Tumblr is blocked in many enterprise networks due to questionable content and you can see how limiting the reach of a single person can be when it comes to corporate policy.  I had to fight this battle in my old job more than once in order to read some very smart people that blogged on Tumblr.

Blogging for me is about unrestricted freedom to pour out my thoughts.  I don’t want to worry about who will see it or how it will be read.  I want people to digest my thoughts and words and have a reaction.  Whether they choose to share it via numerous social media channels or leave a comment makes no difference to me.  I like seeing people share what I’ve committed to virtual paper.  A blog gives me an avenue to write and write without worry.  Sometimes that means it’s just a few paragraphs about something humorous.  Other times it’s an activist rant about something I find abhorrent.  The key is that those thoughts can co-exist without fear of being pigeonholed or categorized by an algorithm or other artificial filter.

Tom’s Take

Sometimes, people make sensationalist posts to call attention to things.  I’ve done it before and will likely do it again in the future.  The key is to read what’s offered and make your own conclusion.  For some, that will be via retweeting or liking.  For others, it will be adding a +1 or a heart.  For me, it’s about collecting my thoughts and pouring them out via a well-worn keyboard on WordPress.  It’s about sharing everything rattling around in my head and offering up analysis and opinion for all to see.  That part isn’t going away any time soon, despite what others might say about blogging in general.  So long as we continue to express ourselves without restriction, the blog will never really die no matter how we choose to share it.

Why Facebook’s Open Compute Switches Don’t Matter. To You.

Facebook_iconFacebook announced at Interop that they are soliciting ideas for building their own top-of-rack (ToR) switch via the Open Compute Project.  This sent the tech media into a frenzy.  People are talking about the end of the Cisco monopoly on switches.  Others claimed that the world would be a much different place now that switches are going to be built by non-vendors and open sourced to everyone.  I yawned and went back to my lunch.  Why?

BYO Networking Gear

As you browse the article that you’re reading about how Facebook is going to destroy the networking industry, do me a favor and take note of what kind of computer you’re using.  Is is a home-built desktop?  Is it something ordered from a vendor?  Is it a laptop or mobile device that you built? Or bought?

The idea that Facebook is building switches isn’t far fetched to me.  They’ve been doing their own servers for a while.  That’s because their environment looks wholly different than any other enterprise on the planet, with the exception of maybe Google (who also builds their own stuff).  Facebook has some very specialized needs when it comes to servers and to networking.  As they mention at conferences, the amount of data rushing into their network on an hourly, let alone daily, basis is mind boggling.  Shaving milliseconds off query times or reducing traffic by a few KB per flow translates into massive savings when you consider the scale they are operating at.

To that end, anything they can do to optimize their equipment to meet their needs is going to be a big deal.   They’ve got a significant motivation to ensure that the devices doing the heavy lifting for them are doing the best job they can.  That means they can invest a significant amount of capital into building their own network devices and still get a good return on the investment.  Much like the last time I built my own home desktop.  I didn’t find a single machine that met all of my needs and desires.  So I decided to cannibalize some parts out of an old machine and just build the rest myself.  Sure, it took me about a month to buy all the parts, ship them to my house, and then assemble the whole package together.  But in the end I was very happy with the design.  In fact, I still use it at home today.

That’s not to say that my design is the best for everyone, or anyone for that matter.  The decisions I made in building my own computer were one’s that suited me.  In much the same way, Facebook’s ToR switches probably serve very different needs than existing data centers.  Are your ToR switches optimized for east-west traffic flow?  I don’t see a lot of data at Facebook directed to other internal devices.  I think Facebook is really pushing their systems for north-south flow.  Data requests coming in from users and going back out to them are more in line with what they’re doing.  If that’s the case, Facebook will have a switch optimized for really fast data flows.  Only they’ll be flowing in the wrong direction for what most people are using data flows for today.  It’s like having a Bugatti Veyron and living in a city with dirt roads.

Facebook admitted that there are things about networking vendors they don’t like.  They don’t want to be locked into a proprietary OS like IOS, EOS, or Junos.  They want a whitebox solution that will run any OS on the planet efficiently.  I think that’s because they don’t want to get locked into a specific hardware supplier either.  They want to buy what’s cheapest at the time and build large portions of their network rapidly as needed to embrace new technology and data flows.  You can’t get married to a single supplier in that case.  If you do, a hiccup in the production line or a delay could cost you thousands, if not millions.  Just look at how Apple ensures diversity in the iPhone supply chain to get an idea of what Facebook is trying to do.  If Apple were to lose a single part supplier there would be chaos in the supply chain.  In order to ensure that everything works like a well-oiled machine, they have multiple companies supplying each outsourced part.  I think Facebook is driving for something simliar in their switch design.

One Throat To Choke

The other thing that gives me pause here is support.  I’ve long held that one of the reasons why people still buy computers from vendors or run Windows and OS X on machines is because they don’t want the headache of fixing things.  A warranty or support contract is a very reassuring thing.  Knowing that you can pick up the phone and call someone to get a new power supply or tell you why you’re getting a MAC flap error lets you go to sleep at night.  When you roll your own devices, the buck stops with you when you need to support something.  Can’t figure out how to get your web server running on Ubuntu?  Better head to the support forums.  Wondering why your BYOSwitch is dropping frames under load?  Hope you’re a Wireshark wizard.  Most enterprises don’t care that a support contract costs them money.  They want the assurance that things are going to get fixed when they break.  When you develop everything yourself, you are putting a tremendous amount of faith into those developers to ensure that bugs are worked out and hardware failures are taken care of.  Again, when you consider the scale of what Facebook is doing, the idea of having purpose-build devices makes sense.  It also makes sense that having people on staff that can fix those specialized devices is cost effective for you.

Face it.  The idea that Facebook is going to destroy the switching market is ludicrous.  You’re never going to buy a switch from Facebook.  Maybe you want to tinker around with Intel’s DPDK with a lab switch so you can install OpenFlow or something similar.  But when it comes time to forklift the data center or populate a new campus building with switches, I can almost guarantee that you’re going to pick up the phone and call Cisco, Arista, Juniper, Brocade, or HP.  Why?  Because they can build those switches faster than you can.  Because even though they are a big captial expenditure (capex), it’s still cheaper in the long run if you don’t have the time to dedicate to building your own stuff.  And when something blows up (and something always blows up), you’re going to want a TAC engineer on the phone sharing the heat with you when the CxOs come headhunting in the data center when everything goes down.

Facebook will go on doing their thing their way with their own servers and switches.  They’ll do amazing things with data that you never dreamed possible.  But just like buying a Sherman tank for city driving, their solution isn’t going to work for most people.  Because it’s built by them for them.  Just like Google’s server farms and search appliances.  Facebook may end up contributing a lot to the Open Compute Project and advancing the overall knowledge and independence of networking hardware.  But to think they’re starting a revolution in networking is about as far fetched as thinking that Myspace was going to be the top social network forever.