Yearly Archives: 2013

FaceTime Audio: The Beginning or The End?

Posted on by
4

BlackApple

The world of mobile devices is a curious one. Handset manufacturers are always raising the bar for features in both hardware and software in order to convince customers to use their device. Yet, no matter how much innovation goes into the handset the vendors are still very reliant upon the whims of the carriers. Apple knows this perhaps better than anyone

In Your FaceTime

FaceTime was the first protocol to feel the wrath of the carriers. Apple developed it as a way to facilitate video communication between parties. The idea was that face-to-face video communications could be simplified to create a seamless experience. And it did, for the most part. Except that AT&T decided that using FaceTime over 3G would put too much strain on their network. At first, they forced Apple to limit FaceTime to only work with wireless connections. That severely inhibited the utility of the protocol. If the only place that a you can video call someone is at home or in a coffee shop (or on crappy hotel wireless) that makes the video call much less useful.

Apple finally allowed FaceTime to operate over cellular networks in iOS 6, yet AT&T (and other carriers) restricted the use of the protocol to those customers on the most current data plans. This eliminated those on older, unlimited data plans from utilizing the service. The carriers eventually gave in to customer pressure and started rolling out the capability to all subscribers. By then, it was too late. Apple had decided to take a different track – replace the need for a carrier.

Message For You

The first shot in this replacement battle came with iMessage. Apple created a messaging protocol like the iChat system for Mac, only it ran on iPhones and iPads (and later Macs). It was enabled by default, which was genius. The first time you sent an Short Message Service (SMS) text to a friend, the system detected you were messaging another iPhone user on a compatible version of software. The system then flipped the messaging over to use iMessage instead of SMS and the chat bubbles turned blue instead of green. Now, you could send pictures of any size as well as texts on any length with no restrictions. 160-character limits were no longer a concern. Neither was paying your carrier for an SMS plan. So long as the people you spoke with were all iDevice users the service was completely free.

iMessage was Apple’s first attempt to sideline the carriers. It removed a huge portion of their profitability. According to an article published at the launch of iMessage, carriers were making $.20 per message outside of an SMS plan for data that would cost about $.0125 on a data plan. Worse yet, that message traversed a control channel that was always present for the user. There was no additional cost to the carrier beyond flipping a switch to enable message delivery to the phone. It was a pure-profit enterprise. Apple seized on the opportunity to erode that profitability.

Today, you can barely find a cellular plan that *doesn’t* include unlimited text messaging. The carriers can no longer reap the rewards of a high profit, low cost service like SMS because of Apple and iMessage. Carriers are instead including it as a quality of life feature that they make nothing from. Cupertino has eliminated one of the sources of carrier entanglement. And they’re poised to do it again in iOS 7.

You Can Hear Me Now

FaceTime Audio was one of the features of iOS 7 that got swept under the rug in favor of talking about flat design or parallax wallpaper. FaceTime Audio uses the same audio codec from FaceTime, AAC-ELD, to initiate a phone call between two iDevice users. Only it doesn’t use the 3G/LTE radio to make the call. It’s all done via the data connection.

I tested FaceTime Audio for the first time after my wife upgraded her phone to iOS 7. The results were beyond astonishing. The audio quality of the call was as crisp and clear as any I’d every heard. In fact, I would compare it to the use of Cisco’s Wideband G.722 codec on an enterprise voice system. My wife, a non-technical person even noticed the difference by remarking, “It’s like you’re right next to me in the same room!” I specifically tried it over 3G/LTE to make sure it wasn’t blocked like FaceTime video. Amazingly, it wasn’t.

The Mean Opinion Score (MOS) rating that telephony network use to rate call clarity runs from 1 to 5. A 1 means you can’t hear them at all. A 5 means there is no difference between talking on the phone and talking in the same room. Most of the “best” calls get a MOS rating in the 4.1-4.3 range. I would rate FaceTime audio at a 4.5 or higher. Not only could I hear my wife clearly on the calls we made, but I also heard background noise clearly when she turned her head to speak to someone. The clarity was so amazing that I even tweeted about it.

FaceTime Audio calling could be poised to do the same thing to voice minutes that iMessage did to SMS. I’ve already changed the favorite for my wife’s number to dial her via FaceTime Audio instead of her mobile phone number. The clarity makes that much of a difference. It also helps that I’m not using any of my plan minutes to call her. Yes, I realize that many carriers make mobile-to-mobile calls free already. However, I was also able to call my wife via FaceTime Audio from my iPad as a test that worked perfectly. Now, I not only don’t use voice minutes but have the flexibility to call from a device that previously had no capability to do so.

Who Needs A Phone?

Think about the iPod Touch. It is a device that is very similar to the iPhone. In fact, with the exception of the cellular radio one might say they’re identical. With iMessage, I can get texts on an iPod touch using my Apple ID. So long as I’m around a wireless connection (or have a 3G MiFi device) I’m connected to the world. With FaceTime audio, the same Apple ID now allows me to take phone calls. The only thing the carriers now have to provide is a data connection. You still can’t text or call non-Apple devices with iMessage and FaceTime. However, you can reduce the amount of money you are paying for their services due to a reduction in the amount of minutes and/or texts you are sending. That should have the mobile carriers running scared.

Tom’s Take

I once said I would never own a cellular phone because sometimes I didn’t want to be found. Today, I get nervous if mine isn’t with me at all times. I also didn’t get SMS messaging at first. Now I spend more time doing that than anything else. Mobile technology has changed our lives. We’ve spent far too much time chained to the carriers, however. They have dictated what when can do with our phones. They have enforced how much data we use and how much we can talk. With protocols like FaceTime Audio, the handset manufacturers are going to start deciding how best to use their own devices. No carrier will be able to institute limits on minutes or texts. I think that if FaceTime Audio takes off in the same way as iMessage, you’ll see mobile carriers offering unlimited talk plans alongside the unlimited text plans within the next two years. If 50% of your userbase is making calls on their data plans, they need for all those “rollover” minutes becomes spurious. People will start reducing their plans down to the minimum necessary to get good data coverage. And if a carrier decides to start gouging for data service? Just take your device to another carrier. Or drop you contact in favor of a MiFi or similar data-only connection. FaceTime Audio is the beginning of easy Voice over IP (VoIP) calling. It’s the end of the road for carrier dominance.

Related articles

SpectraLogic: Who Wants To Save Forever?

Posted on by
Reply

spectra-logic-logo

Data retention is a huge deal for many companies.  When you say “tape backup”, the first thing that leaps to people’s minds is backup operations.  Servers with Digital Audio Tape (DAT) drives or newer Linear-Tape Open (LTO) units.  Judiciously saving those bits for the future when you might just need to dig up one or two in order to recover emails or databases.  After visiting with SpectraLogic at their 2013 Spectra Summit, I’m starting to see that tape isn’t just for saving the day.  It’s for saving everything.

Let’s Go To The Tape

Tape is cheap.  As outlined in this Computer World article, for small applications of less than 6 tape drives, tape is 1/6th the cost of disk backup.  It also lasts virtually forever.  I’ve still got VHS tapes from the 80s that I can watch if I so desire.  And that’s consumer grade magnetic media.  Imagine how well enterprise grade stuff would work?  It’s also portable.  You can eject a tape and take it home on the weekends as a form of disaster recovery.  If you have at least one tape offsite in the grandfather-father-son rotation, you can be assured of getting at least some of your data back in the event of a disaster.

Tape has drawbacks.  It’s slow.  Really slow.  The sequential access of tape drives makes them inefficient as a storage medium.  You can batch writes to a cluster of drives, but good luck if you ever want to get that data back in a reasonable time frame.  I once heard someone refer to tape as “Write Once, Read Never”.  It also has trouble scaling very large.  In the end, you need to cluster several tape units together in order to achieve the kind of scale that you need to capture data from the the virtual firehose today.

Go Deeper

T-Finity. Photo by Stephen Foskett

T-Finity. Photo by Stephen Foskett

SpectraLogic launched a product called DeepStorage.  That is in no way affiliated with Howard Marks (@DeepStorageNet).  DeepStorage is the idea that you can save files forever.  It uses a product called BlackPearl to eliminate one of the biggest issues with tape: speed.  BlackPearl comes with SSD drives to use as a write cache for data being sent to the tape archive.  BlackPearl uses a SpectraLogic protocol called DS3, which stands for DeepS3, to hold the data until it can be written to the tape archive in the most efficient manner.  DS3 looks a lot like Amazon S3.  That’s on purpose.  With the industry as a whole moving toward RESTful APIs and more web interfaces, making a RESTful API for tape storage seems like a great fit for SpectraLogic.

It’s goes a little deeper than that, though (pardon the pun).  One other thing that made me pause was LTFS – the Linear Tape File System.  LTFS allows for a more open environment to write data.  In the past, any data that you backed up to tape left you at the mercy of the software you used to write that data.  CommVault couldn’t read Veritas volumes.  ARCServe didn’t play nicely with Symantec.  With LTFS, you can not only read data from multiple different backup vendors, but you can also stop treating tape drives like Write Once, Read Never devices.  LTFS allows a cluster of tape units to look and act just like a storage array.  A slow array to be sure, but still an array.

SpectraLogic took the ideas behind LTFS and coupled them with DeepStorage to create an idea – “buckets”.  Buckets function just like the buckets you find in Amazon S3.  These are user-defined constructs that hold data.  The BlackPearl caches these buckets and optimizes the writes to your tape array.  Where the bucket metaphor works well is the portability of the bucket.  Let’s say you wanted to transfer long-term data like phone records or legal documents between law firms that are both using DeepStorage.  All you need to do is identify the bucket in question, eject the tape (or tapes) needed to recreate that bucket, and then send the tapes to the destination.  Once there, the storage admin just needs to import the bucket from the tapes in question and all the data in that bucket can be read.  No software version mismatches.  No late night panicked calls because nothing will mount.  Data exchange without hassles.

The Tape Library of Congress

The ideas here boggle the mind.  While at the Spectra Summit, we heard from companies like NASCAR and Yahoo.  They are using BlackPearl and DS3 as a way to store large media files virtually forever.  There’s no reason you can’t do something similar.  I had to babysit a legal server migration one night because it had 480,000 WordPerfect documents that represented their entire case log for the last twenty years.  Why couldn’t that be moved to long-term storage?  For law offices that still have paper records of everything and don’t want to scan it all in for fear of an OCR mistake, why not just make an image of every file and store it on an LTFS volume fronted by DS3?

The flexibility of a RESTful API means that you can created a customized interface virtually on the fly.  Afraid the auditors aren’t going to be able to find data from five years ago?  Make a simple searching interface that is customized to their needs.  Want to do batch processing across multiple units with parallel writes for fault tolerance?  You can program that as well.  With REST calls, anything is possible.

DS3 is going to enable you to keep data forever.  No more worrying about throwing things out.  No need to rent storage lockers for cardboard boxes full of files.  No need to worry about the weather or insects.  Just keeping the data center online is enough to keep your data in a readable format from now until forever.

For more information on SpectraLogic and their solutions, you can find them at http://www.spectralogic.com.  You can also follow them on Twitter as @SpectraLogic.

Disclaimer

I was a guest of SpectraLogic for their 2013 Spectra Summit.  They paid for my flight and lodging during the event.  They also provided a t-shirt, a jacket, and a 2 GB USB drive containing marketing collateral.  They did not ask for any consideration in the writing of this review, nor were they promised any.  The conclusions reach herein are mine and mine alone.  In addition, any errors or omissions are mine as well.

Avaya and the Magic of SPB

Posted on by
7

Avaya_logo-wpcf_200x57

I was very interested to hear from Avaya at Interop New York.  They were the company I knew the least about.  I knew the most about them from the VoIP side of the house, but they’ve been coming on strong with networking as well.  They are one of the biggest champions of 802.1aq, more commonly known as Shortest Path Bridging (SPB).  You may remember that I wrote a bit about SPB in the past and referred to it as the Betamax of networking fabric technologies.  After this presentation, I may be forced to eat my words to a degree.

Paul Unbehagen really did a great job with this presentation.  There were no slides, but he kept the attention of the crowd.  The whiteboard supported his message.  While informal, there was a lot of learning.  Paul knows SPB.  It’s always great to learn from someone that knows the protocol.

Multicast Magic

One of the things I keyed on during the presentation was the way that SPB deals with multicast.  Multicast is a huge factor in Ethernet today.  So much so that even the cheapest SOHO Ethernet switch has a ton of multicast optimization.  But multicast as implemented in enterprises is painful.  If you want to make an engineer’s blood run cold, walk up and whisper “PIM“.  If you want to watch a nervous breakdown happen in real time, follow that up with “RPF“.

RPF checks in multicast PIM routing are nightmarish.  It would be wonderful to get rid of RPF checks to eliminate any loops in the multicast routing table.  SPB accomplishes that by using a Dijkstra algorithm.  The same algorithm that OSPF and IS-IS use to compute paths.  Considering the heavily roots of IS-IS in SPB, that’s not surprising.  The use of Dijkstra means that additional receivers on a multicast tree don’t negatively effect the performance of path calculation.

I’ve Got My IS-IS On You

In fact, one of the optimized networks that Paul talked about involved surveillance equipment.  Video surveillance units that send via multicast have numerous endpoints and only a couple of receivers on the network.  In other words, the exact opposite problem multicast was designed to solve.  Yet, with SPB you can create multicast distribution networks that allow additional end nodes to attach to a common point rather than talking back to a rendezvous point (RP) and getting the correct tree structure from there.  That means fast convergence and simple node addition.

SPB has other benefits as well.  It supports 16.7 million ISIDs, which are much like VLANs or MPLS tags.  This means that networks can grow past the 4,096 VLAN limitation.  It looks a lot like VxLAN to me.  Except for the reliance on multicast and lack of a working implementation.  SPB allows you to use a locally significant VLAN for a service and then defined an ISID that will transport across the network to be decapsulated on the other side in a totally different VLAN that is attached to the ISID.  That kind of flexibility is key for deployments in existing, non-green field environments.

If you’d like to learn more about Avaya and their SPB technology, you can check them out at http://www.avaya.com.  You can also follow them on Twitter as @Avaya.

Tom’s Take

Paul said that 95% of all SPB implementations are in the enterprise.  That shocked me a bit, as I always thought of SPB as a service provider protocol.  I think the key comes down to something Paul said in the video.  When we are faced with applications or additional complexity today, we tend to just throw more headers at the problem.  We figured that wrapping the whole mess in a new tag or a new tunnel will take care of everything.  At least until it all collapses into a puddle.  Avaya’s approach with SPB was to go back down to the lower layers and change the architecture of things to optimize everything and make it work the right way on all kinds of existing hardware.  To quote Paul, “In the IEEE, we don’t build things for the fun it.”  That means SPB has their feet grounded in the right place.  Considering how difficult things can be in data center networking, that’s magical indeed.

Tech Field Day Disclaimer

Avaya was a presenter at the Tech Field Day Interop Roundtable.  They did not ask for any consideration in the writing of this review nor were they promised any.  The conclusions and analysis contained in this post are mine and mine alone.

HP Networking and the Software Defined Store

Posted on by
Reply

HP

HP has had a pretty good track record with SDN.  Even if it’s not very well-known.  HP has embraced OpenFlow on a good number of its Procurve switches.  Given the age of these devices, there’s a good chance you can find them laying around in labs or in retired network closets to test with.  But where is that going to lead in the long run?

HP Networking was kind enough to come to Interop New York and participate in a Tech Field Day roundtable.  It had been a while since I talked to their team.  I wanted to see how they were handling the battle being waged between OpenFlow proponents like NEC and Brocade, Cisco and their hardware focus, and VMware with NSX.  Jacob Rapp and Chris Young (@NetManChris) stepped up to the plate to talk about SDN and the vision on HP.

They cover a lot of ground in here.  Probably the most important piece to me is the SDN app store.

The press picked up on this quickly.  HP has an interesting idea here.  I should know.  I mentioned it in passing in an article I wrote a month ago.  The more I think about the app store model, the more I realize that many vendors are going to go down the road.  Just not in the way HP is thinking.

HP wants to curate content for enterprises.  They want to ensure that software works with their controller to be sure that there aren’t any hiccups in implementation.  Given their apparent distaste for open source efforts, it’s safe to say that their efforts will only benefit HP customers.  That’s not to say that those same programs won’t work on other controllers.  So long as they operate according to the guidelines laid down by the Open Networking Foundation, all should be good.

Show Me The Money

Where’s the value then?  That’s in positioning the apps in the store.  Yes, you’re going to have some developers come to HP and want to simple apps to put in the store.  Odds are better that you’re going to see more recognizable vendors coming to the HP SDN store.  People are more likely to buy software from a name they recognize, like TippingPoint or F5.  That means that those companies are going to want to have a prime spot in the store.  HP is going to make something from hosting those folks.

The real revenue doesn’t come from an SMB buying a load balancer once.  It comes from a company offering it as a service with a recurring fee.  The vendor gets a revenue stream. HP would be wise to work out a recurring fee as well.  It won’t be the juicy 30% cut that Apple enjoys from their walled garden, but anything would be great for the bottom line.  Vendors win from additional sales.  Customers win from having curated apps that work every time that are easy to purchase, install, and configure.  HP wins because everyone comes to them.

Fragmentation As A Service

Now that HP has jumped on the idea of an enterprise-focused SDN app store, I wonder which company will be the next to offer one?  I also worry that having multiple app stores won’t end up being cumbersome in the long run.  Small developers won’t like submitting their app to four or five different vendor-affiliated stores.  More likely they’ll resort to releasing code on their own rather than jump through hoops.  That will eventually lead to support fragmentation.  Fragmentation helps no one.

Tom’s Take

HP Networking did a great job showcasing what they’ve been doing in SDN.  It was also nice to hear about their announcements the day before they broke wide to the press.  I think HP is going to do well with OpenFlow on their devices.  Integrating OpenFlow visibility into their management tools is also going to do wonders for people worried about keeping up with all the confusing things that SDN can do to a traditional network.  The app store is a very intriguing concept that bears watching.  We can only hope that it ends up being a well-respect entry in a long line of easing customers into the greater SDN world.

Tech Field Day Disclaimer

HP was a presenter at the Tech Field Day Interop Roundtable.  In addition, they also provided the delegates a 1TB USB3 hard disk drive.  They did not ask for any consideration in the writing of this review nor were they promised any.  The conclusions and analysis contained in this post are mine and mine alone.

SDN 101 at ONUG Academy

Posted on by
3

300x250_TFD10_V2

Software defined networking is king of the hill these days in the greater networking world.  Vendors are contemplating strategies.  Users are demanding functionality.  And engineers are trying to figure out what it all means.  What’s needed is a way for vendor-neutral parties to get together and talk about what SDN represents and how best to implement it.  Most of the talk so far has been at vendor-specific conferences like Cisco Live or at other conferences like Interop.  I think a third option has just presented itself.

Nick Lippis (@NickLippis) has put together a group of SDN-focused people to address concerns about implementation and usage.  The Open Networking User Group (ONUG) was assembled to allow large companies using SDN to have a semi-annual meeting to discuss strategy and results.  It allows Facebook to talk to JP Morgan about what they are doing to simplify networking through use of things like OpenFlow.

This year, ONUG is taking it a step further by putting on the ONUG Academy, a day-long look at SDN through the eyes of those that implement it.  They have assembled a group of amazing people, including the founder of Cumulus Networks and Tech Field Day’s own Brent Salisbury (@NetworkStatic).  There will be classes about optimizing networks for SDN as well as writing SDN applications for the most popular controllers on the market.  Nick shares more details about the ONUG academy here:

If you’re interested in attending ONUG either for the academy or for the customer-focused meetings, you need to register today.  As a special bonus, if you use the code TFD10 when you sign up, you can take 10% of the cost of registration.  Use that extra cash to go out and buy a cannoli or two.

I’ll be at ONUG with Tech Field Day interviewing customers and attendees about their SDN strategies as well as where they think the state of the industry is headed.  If you’re there, stop by and say hello.  And be sure to bring me one of those cannolis.

Cisco CMX – Marketing Magic? Or Big Brother?

Posted on by
8

Cisco Logo

The first roundtable presenter at Interop New York was Cisco. Their Enterprise group always brings interesting technology to the table. This time, the one that caught my eye was the Connected Mobile Experience (CMX). CMX is a wireless mobility technology that allows a company to do some advanced marketing wizardry.

CMX uses your Cisco wireless network to monitor devices coming into the air space. They don’t necessarily have to connect to your wireless network for CMX to work. They just have to be beaconing for a network, which all devices do. CMX can then push a message to the device. This message can be a simple “thank you” for coming or something more advanced like a coupon or notification to download a store specific app. CMX can then store the information about that device, such as whether or not they joined the network, where they went, and how long they were there. This gives the company to pull some interesting statistics about their customer base. Even if they never hop on the wireless network.

I have to be honest here. This kind of technology gives me the bit of the creeps. I understand that user tracking is the hot new thing in retail. Stores want to know where you went, how long you stayed there, and whether or not you saw an advertisement or a featured item. They want to know your habits so as to better sell to you. The accumulation of that data over time allows for some patterns to emerge that can drive a retail operation’s decision making process.

A Thought Exercise

Think about an average person. We’ll call him Mike. Mike walks four blocks from his office to the subway station every day after work. He stops at the corner about halfway between to cross a street. On that street just happens to be a coffee shop using something like CMX. Mike has a brand new phone that uses wifi and bluetooth and Mike keeps them on all the time. CMX can detect when the device comes into range. It knows that Mike stays there for about 2 minutes but never joins the network. It then moves out of the WLAN area. The data cruncher for the store wants to drive new customers to the store. They analyze the data and find that lots of people stay in the area for a couple of minutes. They equate this to people stopping to decide if they want to have a cup of coffee from the shop. They decide to create a CMX coupon push notification that pops up after one minute on devices that have been seen in the database for the last month. Mike will see a coupon for $1 off a cup of coffee the next time he waits for the light in front of the coffee shop.

That kind of reach is crazy. I keep thinking back to the scenes in Minority Report where the eye scanners would detect you looking at an advertisement and then target a specific ad based on your retina scan. You may say that’s science fiction. But with products like CMX, I can build a pretty complete profile of your behavior even if I don’t have a retina scan. Correlating information provides a clear picture of who you are without any real identity information. Knowing that someone likes to spend their time in the supermarket in the snack aisles and frozen food aisles and less time in the infants section says a lot. Knowing the route a given device takes through the store can help designers place high volume items in the back and force shoppers to take longer routes past featured items.

Tom’s Take

I’m not saying that CMX is a bad product. It’s providing functionality that can be of great use to retail companies. But, just like VHS recorders and Bittorrent, good ideas can often be used to facilitate things that aren’t as noble. I suggested to the CMX developers that they could implement some kind of “opt out” message that popped up if I hadn’t joined the wireless network in a certain period of time. I look at that as a way of saying to shoppers “We know you aren’t going to join. Press the button and we’ll wipe our your device info.” It puts people at ease to know they aren’t being tracked. Even just showing them what you’re collecting is a good start. With the future of advertising and marketing focusing on instant delivery and data gathering for better targeting, I think the products like CMX will be powerful additions. But, great power requires even greater responsibility.

Tech Field Day Disclaimer

Cisco was a presenter at the Tech Field Day Interop Roundtable.  They did not ask for any consideration in the writing of this review nor were they promised any.  The conclusions and analysis contained in this post are mine and mine alone.

Keep It In The Family

Posted on by
3

kids-fighting

I have a brother.  We act like brothers.  We argue and fight with each other fairly often.  We get along in our own way.  One thing we do *not* tolerate is anyone else picking on us. We’ve been known to have a disagreement, but when someone comes up and starts something, we will put aside our disagreement and band together to fight against whoever thought it was a good idea to mess with either one of us.  That’s what brothers do.  Right or wrong, you back your brother.

The Work Family

Managers aren’t all that different from family.  We spend a lot of our lives working in proximity with managers.  The middle of the road types are like family members we tolerate.  Those that aren’t so good tend to be like family members we don’t really get along with at all.  There are some that end up being just like a close family member, like a brother or sister.  It doesn’t mean that the relationship isn’t still managerial.  What makes it key is what happens when someone comes down on you.

My last manager was a great person.  He was calm and thoughtful.  He saw every side of a problem and did what was right.  Those qualities make him great to work for.  But to transcend above that, you have to do something to set yourself apart.  For me, it was the way he corrected behavior.

He tended to give what I like to call “Do Better” talks.  He never yelled.  He never got upset.  In fact, all he usually said was, “I’m disappointed.”  For guys like me, disappointment is ten times worse than getting yelled at.  After chats like that, we agreed to not do whatever it was again and move on.  That’s the essence of a Do Better talk.  No blame, just do better next time.

Backing Your Family

Where he shined was what happened when the other managers came hunting for heads.  You’ve seen this all the time.  Someone screws up and the issue is dealt with.  However, some higher manager feels the need to exert control do they find the person responsible for the issue and give them a dressing down for it.  That doesn’t really serve to correct the behavior.  It’s more about dominance.

Weak willed middle managers will sit back and let the higher manager have their way with employees.  That doesn’t foster a supportive atmosphere.  If you’re never sure who is going to come head hunting, it doesn’t make you want to admit failure.  Which means the issues never get corrected, just covered up.

My old manager was different.  Whenever I screwed up and someone at the top of the food chain came looking for me I never had to worry.  The same man that would call me on the carpet for making mistakes would turn right around and defend me to those that came looking to chew me out again.  In his eyes, he’d dealt with the problem.  Picking on me served no purpose.  Just like my brother, he’d take up arms with me to defend our “family” against others.  It didn’t take long for everyone to learn that my manager dealt with things his way.  And there was no point to trying to deal with his employees against him.

Managers that are willing to defend you right after chewing you out are a special breed.  Those are the kind of folks that employees will walk across broken glass to work for.  It’s not for everyone, though.  Standing up to the heat to defend someone that did something wrong is never easy.  Especially if the higher manager is upset and emotional.  The key is to trust in your instincts as a manager and believe that dealing with the situation your way is the right way.  Having someone come in and undermine you management skills makes you look ineffective.  It’s better to weather the storm of yelling from one person than to lose the respect of everyone in your department.

Treating your employees like family doesn’t mean you can’t be their manager.  You can still  be in charge and have a good relationship with them.  You can win their respect time and time again by backing them even after you’d had to correct them.  When they see you standing up for them against all comers, they’ll have someone they can believe in.  And they’ll be as close to you as any family member.

If you’d like to see more thoughts about management and some great career advice, be sure to check out The Tech Interview.  It’s a great site with great articles and run by awesome people.  Take it to heart and you’ll go far in this world.

Why Do We Tolerate Bad Wireless?

Posted on by
3

HotelSpeedConnection

If there is one black eye on the hospitality industry, it has to be wireless.  I don’t think I’ve ever talked to anyone that is truly happy with the wireless connectivity they found in a hotel.  The above picture from an unnamed hotel in Silicon Valley just serves to underscore that point.  When I was on a recent speaking trip in New England, I even commented about the best hotel wireless I’d ever seen:

Granted, that was due to a secluded hotel on MIT‘s university network, but the fact remains that this shouldn’t be the exception.  This should be the rule.

Thanks to advances in mobile technology like LTE, we have a new benchmark for what a mobile device is capable of producing.  My LTE tablet and phone outrun my home cable connection.  That’s fine for browsing on a picture frame.  However, when it’s time to get real work done I still need to fire up my laptop.  And since there isn’t an integrated LTE/4G hotspot in my MacBook, I have to rely on wireless.

Wireless access has gone from being a kitschy offering at specialized places to being an everpresent part of our daily lives.  When I find myself in need of working outside the office, I can think of at least five different local establishments that offer me free wireless access.  Signing up for mobile hotspot services easily doubles that number.  There are very few places that I go any more that don’t give me the ability to use WiFI.  However, there is a difference between having availability and having “good” availability.

Good Enough Wireless

I would never upload video at a coffee shop or an airport.  The sheer number of folks using the network causes massive latency and throughput issues.  Connections are spotty and it’s not uncommon to see folks throw their hands up in the air because something just randomly stopped working.  However, the most telling statistic is how often we will go back to that same location to use the free wifi again.

Hotels have a captive audience.  You’re there to attend a conference during the day or spend the night.  You are geographically isolated.  You get what you get when it comes to connectivity.  Newer hotel chains that focus on business travelers understand the need for wireless connectivity.  They usually offer it for free with your room.  That’s because they usually have the infrastructure to support wireless coverage from large numbers of guests.  Older hotels that aren’t quite up to snuff or don’t understand why travelers need Internet access usually charge exorbitant fees or bundle the wireless into a “resort” package that gives you a whole bunch of high-margin useless services to get what you want.  Sometimes they use those fees to upgrade the infrastructure.  Or they just pocket the money and go on with their day.

Internet In My Pocket

As much as we complain about terrible wireless at hotels, it’s not like we have an alternative.  Wireless hotspot devices, commonly called “MiFis” after the Verizon branding, are popular with real road warriors.  Why hunt for a coffee shop when you can fire up a wireless network in your pocket?  Most current mobile devices even come with hotspot functionality built in.  But the carriers haven’t gotten the message yet.  For every one that allows hotspot usage (Verizon), you have those that don’t believe in hotspot and want to gouge you with higher fees or data plan changes to revamp bad mobile data decisions in the past.  Yes, I’m looking right at you AT&T.

Mobile hotspots can fix wireless problems in isolated cases, but loading a hotel full of people on MiFis will inevitably end in disaster.  Each of them uses a portion of the LTE/4G spectrum.  Think about a large gathering where everyone’s mobile phones cause spotty reception.  Not because they are all in use, but because they just happen to be occupying the same space.  Towers get overloaded, backhaul networks slow down, and service suffers for everyone.  If you don’t believe me, try making a phone call at Cisco Live some time.  It’s not pretty.

As long as there are no options for solving the problem, hospitality will go right on offering the same terrible coverage they do now.  As far as they are concerned, wireless is best effort.  Best effort should never be acceptable.  You can fix this problem by going to the front desk and telling them all about it.  No, don’t yell at the desk attendant.  They have zero control over what’s going on.  There’s a better way.

Satisfaction Not Guaranteed

Ask for a satisfaction survey.  Fill it out and be brutally honest when you get to the “Are You Pleased” section.  Those surveys go right up the chain into the chain satisfaction ratings.  If they start getting disgruntled comments about bad wireless coverage, I can promise that some Quality Champion somewhere is going to look into things.  Hotels hate black eyes on their satisfaction ratings.  Bad reviews keep people from staying at a hotel.  If you want to get the wireless fixed, tell them how important it is.  Tell them you’ll stay somewhere else next time because you can accomplish anything.  Voting with your wallet is a sure fire way to make an impact.

Tom’s Take

I remember the old Cingular/AT&T Wireless commercials with the cell phones cutting out during calls.  I laughed and thought about all the times it had happened to me.  It because such a sticking point that every carrier worked to upgrade their network and provide better call quality.  No one would stand for spotty service any more as they began to rely on their mobile phones as their primary communications devices.

Wireless is the same now as cell phones were then.  We need a concerted effort to upgrade the experience for everyone to make it usable for things like Hotspot 2.0, which will offload traffic from LTE to WiFI seemlessly.  We can’t let terrible wireless rule us like spotty cell phone coverage did years ago.  Do everything you can to make wireless useful for everyone.

Your Data Center Isn’t Facebook And That’s Just Fine

Posted on by
8

FBLike

While at the Software Defined Data Center Symposium, I had the good fortune to moderate a panel focused on application focused networking in the data center. There were some really smart engineers on that panel. One of the most impressive people was Najam Ahmad from Facebook. He is their Director of Technical Operations. He told me some things about Facebook that made me look at what they are doing a in a new light.

Najam said when I asked him about stakeholder perceptions that he felt a little out of sorts on stage because Ivan Pepelnjak (@IOSHints) and David Cheperdak (@DavidCheperdak) had spent the last fifteen minutes talking about virtual networking. Najam said that he didn’t really know what a hypervisor or a vSwitch were because they don’t run them at Facebook. All of their operating systems and servers run directly on bare metal. That shocked me a bit. Najam said that inserting anything in between the server and what its function was added unnecessary overhead. That’s a pretty unique take on things when you look at how many data centers are driving toward full virtualization.

Old Tools, New Uses

Facebook also runs BGP to the top-of-rack (ToR) switches in their environment. That means that they are doing layer 3 all the way to their access layer. What’s funny is that while BGP in the ToR switches provides for scalability and resiliency, they don’t use BGP as their primary protocol when exchanging routes with providers.  For Facebook, BGP at the edge of doesn’t provide enough control over network egress. They take the information that BGP is providing and they crunch it a bit further before adding that all into a controller-based solution that applies business logic and policies to determine the best solution for a given network scenario.

Najam also said that they had used NetFlow for a while to collect data from their servers in order to build a picture of what was going on inside the network. What they found is that the collectors were becoming overwhelmed by the amount of data that they were being hammered with. So instead of installing bigger, faster collectors the Facebook engineers broke the problem apart by putting a small shim program on every server to collect the data and then forward to a system designed to collect data inputs, not just NetFlow inputs. Najam lovingly called their system “FBFlow”.

I thought about this for a while before having a conversation with Colin McNamara (@ColinMcNamara). He told me that this design was a lot more common than I previously thought and that he had implemented it a few times already. At service providers. That’s when things really hit home for me.

Providing Services

Facebook is doing the same things that you do in your data center today. They’re just doing it at a scale that’s one or two orders of magnitude bigger. The basics are all still there: Facebook pushes packets around a network to feed servers and provide applications for consumption by users. What is so different is that the scale at which Facebook does this begins to look less and less like a traditional data center and more and more like a service provider. After all, they *are* providing a service to their users.

I’ve talked before about how Facebook’s Open Compute Project (OCP) switch wasn’t going to be the death knell for traditional networking. Now you see some of that validated in my opinion. Facebook is building hardware to meet their needs because they are a strange hybrid of data center and service provider. Things that we would do successfully in a 500 VM system don’t scale at all for them. Crazy ideas like running exterior gateway routing protocols on ToR switches work just fine for them because of the scale at which they are operating.

Which brings me to the title of the post. People are always holding Facebook and Google in such high regard for what they are doing in their massive data centers. Those same people want to try to emulate that in their own data centers and often find that it just plain doesn’t work.  It’s the same set of protocols.  Why won’t this work for me?

Facebook is solving problems just like a service provider would.  They are building not for continuous uptime, but instead for detectable failures that are quickly recoverable.  If I told you that your data center was going to be down for ten minutes next month you’d probably be worried.  If I told you that those outages were all going to be one minute long and occur ten times, you’d probably be much less worried.  Service providers try to move around failure instead of pouring money into preventing it in the first place.  That’s the whole reasoning behind Facebook’s “Fail Harder” mentality.

Failing Harder means making big mistakes and catching them before they become real problems.  Little issues tend to get glossed over and forgotten about.  Thing about something like Weighted Random Early Detection (WRED).  WRED works because you can drop a few packets from a TCP session and it will keep chugging and request the missing bits.  If you kill the entire connection or blow up a default gateway then you’ve got a real issue.  WRED fixes a problem, global TCP synchronization, by failing quietly once in a while.  And it works.

Tom’s Take

Instead of comparing your data center to Facebook or Google you should be taking a hard look at what you are actually trying to do.  If you are doing Hadoop your data center is going to look radically different than a web services company.  There are lessons you can learn from what the big boys are doing.  Failing harder and using old tools in novel new ways are a good start your own data center analysis and planning.  Just remember that those big data centers aren’t alien environments.  They just have different needs to meet.

Here’s the entire SDDC Symposium Panel with Najam if you’d like to watch it.  He’s got a lot of interesting insights into things besides what I wrote about above.

The Vision Of A ThousandEyes

Posted on by
1

ThousandEyes_Logo

Scott Adams wrote a blog post once about career advice and whether is was better to be excellent at one thing or good at several things. Basically, being the best at something is fairly hard. There’s always going to be someone smarter or faster than you doing it just a bit better. Many times it’s just as good to be very good at what you do. The magic comes when you take two or three things that are very good and combine them in a way that no one has seen before to make something amazing. The kind of thing that makes people gaze in wonder then immediately start figuring out how to use your thing to be great.

During Networking Field Day 6, ThousandEyes showed the delegates something very similar to what Scott Adams was talking about. ThousandEyes uses tools like Traceroute, Ping, and BGP data aggregation to collect data. These tools aren’t overly special in and of themselves. Ping and Traceroute are built into almost every networking stack. BGP looking glass servers and data analysis have been available publicly for a while and can be leveraged in a tool like BGPMon. All very good tools. What ThousandEyes did was combine them in a way to make them better.

ThousandEyes can show data all along the path of a packet. I can see response times and hop-by-hop trajectory. I can see my data leave one autonomous system (AS) and land in another. Want to know what upstream providers your ISP is using? ThousandEyes can tell you that. All that data can be collected in a cloud dashboard. You can keep tabs on it to know if you service level agreements (SLAs) are being met. Or, you could think outside the box and do something that I found very impressive.

Let’s say you are a popular website that angered someone. Maybe you published an unflattering article. Maybe you cut off a user doing something they should have. Maybe someone out there just has a grudge. With the nuclear options available to most “hackers” today, the distributed denial of service (DDoS) attack seems to be a popular choice. So popular that DDoS mitigation services have sprung up to shoulder the load. The basic idea is that when you determine that you’re being slammed with gigabits of traffic, you just swing the DNS for your website to a service that starts scrubbing away attack traffic and steering legitimate traffic to your site. In theory it should prevent the attackers from taking you offline. But how can you prove it’s working?

ThousandEyes can do just that. In the above video, they show what happened when Bank of America (BoA) was recently knocked offline by a huge DDoS attack. The information showed two of the three DDoS mitigation services were engaged. The third changeover didn’t happen. All that traffic was still being dumped on BoA’s servers. Those BoA boxes couldn’t keep up with what they were seeing, so even the legitimate traffic that was being forwarded on by the mitigation scrubbers got lost in the noise. Now, if ThousandEyes can tell you which mitigation provider failed to engage then that’s a powerful tool to have on your side when you go back to them and tell them to get their act together. And that’s just one example.

I hate calling ISPs to fix circuits because it never seems to be their fault. No matter what I do or who I talk to it never seems to be anything inside the provider network. Instead, it’s up to me to fiddle with knobs and buttons to find the right combination of settings to make my problem go away, especially if it’s packet loss. Now, imagine if you had something like ThousandEyes on your side. Not only could you see the path that your packets are taking through your ISP, you can check latency and see routing loops and suboptimal paths. And, you can take a screenshot of it to forward to the escalation tech during those uncomfortable phone arguments about where the problem lies. No fuss, no muss. Just the information you need to make your case and get the problem fixed.

If you’d like to learn more about ThousandEyes and their monitoring solutions, check out their website at http://www.thousandeyes.com. You can also follow them on Twitter as @ThousandEyes.

Tom’s Take

Vision is a funny thing. Some have it. Some don’t. Having vision can mean many things. It can be someone who assembles tools in a novel way to solve a problem. It can be the ability to collect data and “see” what’s going on in a network path. It can also mean being able to take that approach and use it in a non-obvious way to provide a critical service to application providers that they’ve never had before. Or, as we later found out at Networking Field Day 6 during a presentation with Solarwinds, it can mean having the sense to realize when someone is doing something right, as Joel Dolisy said when asked about ThousandEyes, “Oh, we’ve got our eye on them.” That’s a lot of vision. A ThousandEyes worth.

Special thanks to Ivan Pepelnjak (@IOSHints) for giving me some ideas on this review.

Networking Field Day Disclaimer

While I was not an official delegate at Networking Field Day 6, I did participate in the presentations and discussions. ThousandEyes was a sponsor of Networking Field Day 6. In addition to hosting a presentation in their offices, they provided snacks and drink for the delegates. They also provided a gift bag with a vacuum water bottle, luggage tag, T-shirt, and stickers (which I somehow managed to misplace). At no time did they ask for any consideration in the writing of this review, nor were they offered any. Independence means no restrictions.  The analysis and conclusions contained in this post are mine and mine alone.