The Vision Of A ThousandEyes


Scott Adams wrote a blog post once about career advice and whether is was better to be excellent at one thing or good at several things. Basically, being the best at something is fairly hard. There’s always going to be someone smarter or faster than you doing it just a bit better. Many times it’s just as good to be very good at what you do. The magic comes when you take two or three things that are very good and combine them in a way that no one has seen before to make something amazing. The kind of thing that makes people gaze in wonder then immediately start figuring out how to use your thing to be great.

During Networking Field Day 6, ThousandEyes showed the delegates something very similar to what Scott Adams was talking about. ThousandEyes uses tools like Traceroute, Ping, and BGP data aggregation to collect data. These tools aren’t overly special in and of themselves. Ping and Traceroute are built into almost every networking stack. BGP looking glass servers and data analysis have been available publicly for a while and can be leveraged in a tool like BGPMon. All very good tools. What ThousandEyes did was combine them in a way to make them better.

ThousandEyes can show data all along the path of a packet. I can see response times and hop-by-hop trajectory. I can see my data leave one autonomous system (AS) and land in another. Want to know what upstream providers your ISP is using? ThousandEyes can tell you that. All that data can be collected in a cloud dashboard. You can keep tabs on it to know if you service level agreements (SLAs) are being met. Or, you could think outside the box and do something that I found very impressive.

Let’s say you are a popular website that angered someone. Maybe you published an unflattering article. Maybe you cut off a user doing something they should have. Maybe someone out there just has a grudge. With the nuclear options available to most “hackers” today, the distributed denial of service (DDoS) attack seems to be a popular choice. So popular that DDoS mitigation services have sprung up to shoulder the load. The basic idea is that when you determine that you’re being slammed with gigabits of traffic, you just swing the DNS for your website to a service that starts scrubbing away attack traffic and steering legitimate traffic to your site. In theory it should prevent the attackers from taking you offline. But how can you prove it’s working?

ThousandEyes can do just that. In the above video, they show what happened when Bank of America (BoA) was recently knocked offline by a huge DDoS attack. The information showed two of the three DDoS mitigation services were engaged. The third changeover didn’t happen. All that traffic was still being dumped on BoA’s servers. Those BoA boxes couldn’t keep up with what they were seeing, so even the legitimate traffic that was being forwarded on by the mitigation scrubbers got lost in the noise. Now, if ThousandEyes can tell you which mitigation provider failed to engage then that’s a powerful tool to have on your side when you go back to them and tell them to get their act together. And that’s just one example.

I hate calling ISPs to fix circuits because it never seems to be their fault. No matter what I do or who I talk to it never seems to be anything inside the provider network. Instead, it’s up to me to fiddle with knobs and buttons to find the right combination of settings to make my problem go away, especially if it’s packet loss. Now, imagine if you had something like ThousandEyes on your side. Not only could you see the path that your packets are taking through your ISP, you can check latency and see routing loops and suboptimal paths. And, you can take a screenshot of it to forward to the escalation tech during those uncomfortable phone arguments about where the problem lies. No fuss, no muss. Just the information you need to make your case and get the problem fixed.

If you’d like to learn more about ThousandEyes and their monitoring solutions, check out their website at You can also follow them on Twitter as @ThousandEyes.

Tom’s Take

Vision is a funny thing. Some have it. Some don’t. Having vision can mean many things. It can be someone who assembles tools in a novel way to solve a problem. It can be the ability to collect data and “see” what’s going on in a network path. It can also mean being able to take that approach and use it in a non-obvious way to provide a critical service to application providers that they’ve never had before. Or, as we later found out at Networking Field Day 6 during a presentation with Solarwinds, it can mean having the sense to realize when someone is doing something right, as Joel Dolisy said when asked about ThousandEyes, “Oh, we’ve got our eye on them.” That’s a lot of vision. A ThousandEyes worth.

Special thanks to Ivan Pepelnjak (@IOSHints) for giving me some ideas on this review.

Networking Field Day Disclaimer

While I was not an official delegate at Networking Field Day 6, I did participate in the presentations and discussions. ThousandEyes was a sponsor of Networking Field Day 6. In addition to hosting a presentation in their offices, they provided snacks and drink for the delegates. They also provided a gift bag with a vacuum water bottle, luggage tag, T-shirt, and stickers (which I somehow managed to misplace). At no time did they ask for any consideration in the writing of this review, nor were they offered any. Independence means no restrictions.  The analysis and conclusions contained in this post are mine and mine alone.

Death of Conversation and the Nuclear Option

If you’ve spent any time online since the founding of the Internet, you know how quickly things can escalate when it comes to arguments.

"Borrowed" from Tony Bourke (@tbourke)

“Borrowed” from Tony Bourke (@tbourke)

This is no more apparent to me than the recent discourse surrounding “Donglegate.”  The short, short version:

Male Pycon attendees make inappropriate comments.  Female attendee gets upset and publicly tweets about it.  Stuff happens.  People get fired.

I’m not going to go into anything about the situation, as that’s not my place or my area of expertise to comment.  What I did find upsetting was that after the first attendee that made the inappropriate comments was fired from his job, there was a massive Distributed Denial of Service (DDoS) attack launched at the employer of Adria Richards (the female attendee).  Their website was knocked offline for a couple of days.  The news that Ms. Richards had been let go from her job had to be posted on Facebook initially, as there was no other official communication method available.  It wasn’t until the news broke of her removal that the DDoS finally died down to the point where normal operations could continue.

This is a disturbing trend that I’m starting to see in many online disagreements.  As an increasingly online society, we have started to forego polite discourse and jump straight to the “nuclear strike” option of retaliation.  Don’t like a blogger’s post?  Nuke his site with a DoS tool like LOIC.  Think a vendor employee did something wrong?  Shame them in public and release their private information (also known as “doxing”).  Even noted security researcher Brian Krebs had the local SWAT team called to his house after he wrote about a service used to knock websites offline.

How did we end up at the point where we’ve skipped past “I disagree with what you say and would like to debate this topic!” all the way to “You suck and I’m going to burn down your house and the hospital you were born in!!!“?  Rather than have a meaningful and rational discussion, it appears to be in vogue to nuke anything and everything associated with the person that has made you upset.

Look at what’s happened to Spamhaus recently.  Yes, the articles posting about a massive 300Gbps “Internet breaking” DDoS were a bit overblown.  Yet someone has decided that the best way to make Spamhaus “pay” for their crimes is to launch an attack that relies on using DNS exploits to amplify the traffic to the point where even DDoS frontends like CloudFlare are having a hard time keeping up.  Spamhaus does have a reputation for taking things to the extreme as well when it comes to blacklisting IP ranges suspected of providing havens for spammers.  What you end up with is a standoff where neither side is willing to budge from their viewpoint.  Only they fight their war with packet generators and black hole ACLs that cause problems for users and make ISP technicians pull their hair out.

I’m no stranger to the “nuclear option” myself.  I’ve made some comments on my blog that are a bit…pointed, to put it mildly.  While I do get a bit of satisfaction sometimes from verbally sparring with someone that has called me names or done other such unsavory things, that’s where it ends for me.  I have no desire to do any further harm besides jousting with clever phrases.  I’ve never considered erasing their phone or clogging their Internet connection or releasing their Social Security number online.  Ruining someone for the sake of making a point is the height of pettiness.

Here’s a thought: At the height of the arguments leading up to the American Civil War, where American representatives were calling for state secession opening in Congress, decorum never faltered.  Even when referring to a senator that was despised for their politics, the opponent always called them “the distinguished gentleman from <state>.”  Hard to believe that a conflict that saw families torn apart and Americans shooting at each other by the thousands could still have some polite discussion in a government on the verge of being ripped asunder.  Those rules served to keep a bit of decorum in a place where it was required for conversation and useful argument to take place.

Maybe the problem is anonymity.  It’s far to easy to fire off anonymous comments or be a small cog in a larger DDoS and have a huge impact while staying mostly safe behind a curtain of obscurity.  People who might never utter an ill word to another human being suddenly turn into biased uncompromising trolls.  Rather than discuss rational points, they turn to the most extreme option available to either silence their critics or prove a point in a “scorched earth” victory at any costs.  Consider this XKCD comic:

I laughed when I first read this.  Slowly, I realized that the author is right.  Sometimes reading back the comment to people proves the point better than anything.  I frequently use a commenter’s words in my replies to point out what was said and how it was construed (at least by me).

Tom’s Take

In the end, to me it comes down to a matter of manners.  I’ve always made it a rule here to never say anything about a topic or person that I wouldn’t say to them in person.  I also do my best to look at both sides of an argument with a critical eye.  I don’t call people names or threaten them.  Even when people call me biased, narcissistic, or even just plain stupid I just try and debate the facts of the discussion.  Sure, I may rant and rave and shout out loud to myself sometimes.  However, name-calling never accomplishes anything.  Moving beyond that to the nuclear option is equally appalling to me as well.  I have no desire to knock out anyone’s blog or line of business for the sake of proving a point.  If my arguments don’t suffice to change someone’s mind or get a policy I care for overturned, then that’s my fault and not the fault of others.  I just agree to disagree and move on.  Maybe therein lies the spark that will reignite polite conversation and discussion instead of leaping straight to the last resort.  After all, an attentive ear can win more battles than the sharpest spear.