One of the highlights of my trip to Cisco Live 2011 was an invitation from Aneel Lakhani and Brian Gracely of Cisco to take an impromptu tour of the SuperNAP facility in Las Vegas, Nevada.  Now, I must admit that I was a bit ignorant of what SuperNAP was when I accepted the invitation.  That changed quickly, though.

We left the Mandalay Bay hotel on the south end of The Strip and drove over to the facility.  The drive would have been less than fifteen minutes except for some road construction traffic.  Once we turned the corner into the facility, we were greeted with this visage:

This is the kind of facility that just screams “stay away”.  The only entrance is the door in the middle right of the picture.  You must have an appointment to enter the facility, and the voice on the loudspeaker isn’t a friendly one.  Once our tour group gained access, we parked in one of the three parking spots and were immediately greeted by a man in a tactical vest with an assortment of hardware, like a walkie-talkie, flashlight, and sidearm.  He informed us that we needed to have our identification ready and there were ABSOLUTELY no pictures to be taken inside the facility.  While he didn’t confiscate our cell phones, he did question one of our group about an insulin pump on his hip.

Once inside the entrance, I immediately noticed the hardware located behind the security office.  It was more impressive than what I had seen in the FBI buildings I’ve visited.  Behind several inches of bullet resistant glass were M4 assault rifles and what appeared to be shotguns as well.  Coupled with the the posture of the security guards, I was pretty sure this was going to be a rather interesting tour.  Once I surrendered by driver’s license for a visitor pass, we were escorted through the steel mantrap.  The security guard had to buzz each visitor in individually with his access badge and his thumbprint, a very nice combo of two-factor authorization for a security nerd like me.  We were escorted to a conference room filled with industrial-looking tables and comfortable chairs.  The guard asked if we needed to use the restroom, and after no one accepted his offer he asked that we stay in this room until our appointment arrived.  When I heard the door click behind him, I realized that was more of a statement rather than a request.

Once the rest of our group arrived, Missy Young came in to start our sightseeing show.  She started out giving us an overview of the facility itself, focusing on the layout and the air handling system.  In most datacenters, keeping the massive amount of equipment cool is one of the more difficult tasks.  When Switch first asked about cooling SuperNAP with the traditional air cooling units, the cooling vendor’s response to their request would have filled half the datacenter floor with A/C units.  So Switch built their own:

Each of those units contains four different types of cooling systems to ensure the most efficient method is used to keep the data center at the appropriate temperature.  There’s even a software program, Living Datacenter, running at all times to monitor the air temperature outside that keeps the air handlers running at peak efficiency and the data center from becoming a greenhouse.  Altogether, there are 44,000 tons of cooling available for the 407,000 square foot facility.  Inside, it was nice and chilly, just like the servers like it.  Thanks to all that cooling power, SuperNAP customers are not required to space their equipment out to allow proper airflow.  The building can deliver the right amount of cooling to every square inch of the floor, therefore allowing Switch to make the most efficient use of the space.

SuperNAP houses lots of equipment from various different companies like Sony and Ebay.  They also host a variety of servers from government agencies as well, many of which they aren’t allowed to talk about.  Because of this, the facility can never be down.  They promise 100% uptime to their customers, and they have the backup systems to deliver.  The facility has 100 MW of power delivered to it for running systems and has almost 150 MW of generator capacity.  Each of those generators is powered by 7,000 gallons of diesel fuel.  In the event of a power outage, SuperNAP has contracts with several fuel companies to start delivering diesel within two hours of the outage report and every eight hours thereafter until power is restored.  In the event that the fuel resupply fails, the security forces are authorized to begin commandeering fuel from the civilian population of Las Vegas.  However, I doubt it would come to that anytime soon.  SuperNAP taps the two national power mains that deliver electricity to Las Vegas upstream of the city.  Even if Sin City starts having brownouts, SuperNAP will stay online.  Due to their level of importance in keeping the lights on, the only facility in Nevada that would get power preference before them is the Hoover Dam.

After the overview, we all signed our waivers and walked out a door to begin the tour of the facility.  Since we couldn’t take our own pictures, I’m going to post some of theirs.  Trust me, they’re real.

It’s really impressive in person.  Even though there were only nine of us on the tour, we were followed by an armed guard at all times.  He radioed in every time we walked into a different section of the facility.  It was a little eerie, but I can see how they might want to keep tabs on a shifty fellow like me.

We walked into one of the air handling units and got to see each of the sections bringing air into the facility.  There is a pressure differential inside each unit, so in order to show us the amount of air being pushed by the fans, they had to crack an outside door to equalize things inside.  Somehow, I got stuck on the end nearest the fan door, and when the rather large guard opened it up, I got blasted with hurricane-force cold air for a few seconds.  I felt like Jim Cantore for a bit.  That much air flowing into the facility is a large reason behind its success at keeping the whole thing cool and stable, and they’ve got 30 of these things around the perimeter.  Combined with Living Datacenter, the units are even smart enough to shut off the outside air supply in the event of a dust storm or other unwanted conditions and recirculate the hot exhaust coming back from the server areas.

In between areas runs the Power Spine, the location of the large PDUs that distribute the go-go juice to all those hungry servers.  It looked like something out of a sci-fi movie:

Each rack can be provided 26 Kw of power from each of those color-coded units, and there are two running to each rack to provide redundancy, with a third available just in case.  There’s even enough floor space to double up and provide 200 Mw of power to the whole facility.  The floor itself uses 4,500 psi concrete to be able to support all that weight.  And since there isn’t any raised floor space in the whole facility (all the infrastructure is overhead), it allows customers to pack in some seriously heavy computing power.

One thing I will note here.  You notice that everything in these pictures looks very polished and theatrical.  That’s a bit on purpose.  From the armed guards to the LED lighting to the enamel paint everywhere, this whole facility screams theater from the inside.  Some disbelieve their ability to deliver what they promise.  Missy’s response was “just ask our customers.”  I agree that there is a bit of a show being put on at the facility.  I surmise that it’s most likely due to an extension of will from those that built the facility in the first place.  After all, it’s very impressive to visitors to see all this hardware powering rows and rows of computing power.  Why not take the extra time and effort to make it pretty?  Besides, if someone leaves a mess, you’ve always got the security guards around to shoot the offenders.

Tom’s Take

If there’s ever a zombie apocalypse, I’m getting in the car and heading for SuperNAP.  I’m going to call ahead and make sure that Missy knows I’m coming though, because that could get interesting when the guards don’t know whether or not I’m zombified.  The SuperNAP facility delivers a very impressive profile for a datacenter, both in size and operation.  It was like walking through Toys R Us as a kid, only the toys here are multi-million dollar server equipment and the sales clerks carry assault rifles.  SuperNAP delivers on their promises of 100% uptime and their customer list is rather impressive.  I think that everyone interested in the hows and whys of data center design should take a peek inside to see what it looks like when it’s done right.  Just call ahead first.

Thanks again to Aneel Lakhani and Brian Gracely of Cisco for the invite and thanks to the rest of the group for allowing me to tag along and not get us shot at.