Are We Seeing SD-WAN Washing?

You may have seen a tweet from me last week referencing a news story that Fortinet was now in the SD-WAN market:

It came as a shock to me because Fortinet wasn’t even on my radar as an SD-WAN vendor. I knew they were doing brisk business in the firewall and security space, but SD-WAN? What does it really mean?

SD Boxes

Fortinet’s claim to be a player in the SD-WAN space brings the number of vendors doing SD-WAN to well over 50. That’s a lot of players. But how did the come out of left field to land a deal rumored to be over a million dollars for a space that they weren’t even really playing in six months ago?

Fortinet makes edge firewalls. They make decent edge firewalls. When I used to work for a VAR we used them quite a bit. We even used their smaller units as remote appliances to allow us to connect to remote networks and do managed maintenance services. At no time during that whole engagement did I ever consider them to be anything other than a firewall.

Fast forward to 2018. Fortinet is still selling firewalls. Their website still focuses on security as the primary driver for their lines of business. They do talk about SD-WAN and have a section for it with links to whitepapers going all the way back to May. They even have a contributed article for SDxCentral back and February. However, going back that far the article reads more like a security company that is saying their secure endpoints could be considered SD-WAN.

This reminds me of stories of Oracle counting database licenses as cloud licenses so they could claim to be the fourth largest cloud provider. Or if a company suddenly decided that every box they sold counted as an IPS because it had a function that could be enabled for a fee. The numbers look great when you start counting them creatively but they’re almost always a bit of a fib.

Part Time Job

Imagine if Cisco suddenly decided to start counting ASA firewalls as container engines because of a software update that allowed you to run Kubernetes on the box. People would lose their minds. Because no one buys an ASA to run containers. So for a company like Cisco to count them as part of a container deployment would be absurd.

The same can be said for any company that has a line of business that is focused on one specific area and then suddenly decides that the same line of business can be double-counted for a new emerging market. It may very well be the case that Fortinet has a huge deployment of SD-WAN devices that customers are very happy with. But if those edge devices were originally sold as firewalls or UTM devices that just so happened to be able to run SD-WAN software, it shouldn’t really count should it? If a customer thought they were buying a firewall they wouldn’t really believe it was actually an SD-WAN router.

The problem with this math is that everything gets inflated. Maybe those SD-WAN edge devices are dedicated. But, if they run Fortinet’s security suite are also being counting in the UTM numbers? Is Cisco going to start counting every ISR sold in the last five years as a Viptela deployment after the news this week that Viptela software can run on all of them? Where exactly are we going to draw the line? Is it fair to say that every x86 chip sold in the last 10 years should count for a VMware license because you could conceivably run a hypervisor on them? It sounds ridiculous when you put it like that, but only because of the timelines involved. Some crazier ideas have been put forward in the past.

The only way that this whole thing really works is if the devices are dedicated to their function and are only counted for the purpose they were installed and configured for. You shouldn’t get to add a UTM firewall to both the security side and the SD-WAN side. Cisco routers should only count as traditional layer 3 or SD-WAN, not both. If you try to push the envelope to put up big numbers designed to wow potential customers and get a seat at the big table, you need to be ready to defend your reporting of those numbers when people ask tough questions about the math behind those numbers.

Tom’s Take

If you had told me last year that Fortinet would sell a million dollars worth of SD-WAN in one deal, I’d ask you who they bought to get that expertise. Today, it appears they are content with saying their UTM boxes with a central controller count as SD-WAN. I’d love to put them up against Viptela or VeloCloud or even CloudGenix and see what kind of advanced feature sets they produce. If it’s merely a WAN aggregation box with some central control and a security suite I don’t think it’s fair to call it true SD-WAN. Just a rinse and repeat of some washed up marketing ideas.


Cisco and the Two-Factor Two-Step

In case you missed the news, Cisco announced yesterday that they are buying Duo Security. This is a great move on Cisco’s part. They need to beef up their security portfolio to compete against not only Palo Alto Networks but also against all the up-and-coming startups that are trying to solve problems that are largely being ignored by large enterprise security vendors. But how does an authentication vendor help Cisco?

Who Are You?

The world relies on passwords to run. Banks, email, and even your mobile device has some kind of passcode. We memorize them, write them down, or sometimes just use a password manager (like 1Password) to keep them safe. But passwords can be guessed. Trivial passwords are especially vulnerable. And when you factor in things like rainbow tables, it gets even scarier.

The most secure systems require you to have some additional form of authentication. You may have heard this termed as Two Factor Authentication (2FA). 2FA makes sure that no one is just going to be able to guess your password. The most commonly accepted forms of multi-factor authentication are:

  • Something You Know – Password, PIN, etc
  • Something You Have – Credit Card, Auth token, etc
  • Something You Are – Biometrics

You need at least two of these in order to successfully log into a system. Not having an additional form means you’re locked out. And that also means that the individual components of the scheme are useless in isolation. Knowing someone’s password without having their security token means little. Stealing a token without having their fingerprint is worthless.

But, people are starting to get more and more sophisticated with their attacks. One of the most popular forms of 2FA is the SMS authentication. It combines What You Know, in this case you password for your account, with Something You Have, which is a phone capable of receiving an SMS text message. When you log in, the authentication system sends an SMS to the authorized number and you have to type in the short-lived code to get into the system.

Ask Reddit how that worked out for them recently. A hacker (or group) was able to intercept the 2FA SMS codes for certain accounts and use both factors to log in and gather account data. It’s actually not as trivial as one might think to intercept SMS codes. It’s much, much harder to crack the algorithm of something like a security token. You’d need access to the source code and months to download everything. Like exactly what happened in 2011 to RSA.

In order for 2FA to work effectively, it needs to be something like an app on your mobile device that can be updated and changed when necessary to validate new algorithms and expire old credentials. It needs to be modern. It needs to be something that people don’t think twice about. That’s what Duo Security is all about. And, from their customer base and the fact that Cisco payed about $2.3 billion for them, they must do it well.

Won’t Get Fooled Again

How does Duo help Cisco? Well, first and foremost I hope that Duo puts an end to telnet access to routers forever. Telnet is the lazy way we enable remote access to devices. SSH is ten times better and a thousand times more secure. But setting it up properly to authenticate with certificate authentication is a huge pain. People want it to work when they need it to work. And tying it to a specific machine or location isn’t the easiest or more convenient thing.

Duo can give Cisco the ability to introduce real 2FA login security to their devices. IOS could be modified to require Duo Security app login authentication. That means that only users authorized to log into that device would get the login codes. No more guessed remote passwords!

Think about integrating Duo with Cisco ISE. That could be a huge boon for systems that need additional security. You could have groups of system that need 2FA and others that don’t. You could easily manage those lists and move systems in and out as needed. Or, you could start a policy that all systems needs 2FA and phase in the requirements over time to make people understand how important it is and give them time to download the app and get it set up. The ISE possibilities are endless.

One caveat is that Duo is a program that works with a large number of third party programs right now. Including integrations with Juniper Networks. As you can imagine, that list might change once Cisco takes control of the company. Some organizations that use Duo will probably see a price increase and will continue to offer the service to their users. Others, possibly Juniper as an example, may be frozen out as Cisco tries to keep the best parts of the company for their own use. If Cisco is smart, they’ll keep Duo available for any third party that wants to use the platform or integrate. It’s the best solution out there for solving this problem and everyone deserves to have good security.

Tom’s Take

Cisco buying a security company is no shock. They need the horsepower to compete in a world where firewalls are impediments at best and hackers have long since figured out how to get around static defenses. They need to get involved in software too. Security isn’t fought in silicon any more. It’s all in code and beefing up the software side of the equation. Duo gives them a component to compete in the broader authentication market. And the acquisition strategy is straight out of the Chambers playbook.

A plea to Cisco: Don’t lock everyone out of the best parts of Duo because you want to bundle them with recurring Cisco software revenue. Let people integrate. Take a page from the Samsung playbook. Just because you compete with Apple doesn’t mean you can’t make chips for them. Keep your competitors close and make they use your software and you’ll make more money than freezing everyone out and claiming your software is the best and least used of the bunch.

The Privacy Pickle

I recorded a fantastic episode of The Network Collective last night with some great friends from the industry. The topic was privacy. Originally I thought we were just going to discuss how NAT both was and wasn’t a form of privacy and how EUI-64 addressing wasn’t the end of days for people worried about being tracked. But as the show wore on, I realized a few things about privacy.

Booming In Peace

My mom is a Baby Boomer. We learn about them as a generation based on some of their characteristics, most notably their rejection of the values of their parents. One of things they hold most dear is their privacy. They grew up in a world where they could be private people. They weren’t living in a 1 or 2 room house with multiple siblings. They had the right of privacy. They could have a room all to themselves if they so chose.

Baby Boomers, like my mom, are intensely private adults. They marvel at the idea that targeted advertisements can work for them. When Amazon shows them an ad for something they just searched for they feel like it’s a form of dark magic. They also aren’t trusting of “new” things. I can still remember how shocked my mother was that I would actively get into someone else’s car instead of a taxi. When I explained that Uber and Lyft do a similar job of vetting their drivers it still took some convincing to make her realize that it was safe.

Likewise, the Boomer generation’s personal privacy doesn’t mesh well with today’s technology. While there are always exceptions to every rule, the number of people in their mid-50s and older that use Twitter and Snapchat are far, far less than the number that is the target demographic for each service. I used to wonder if it was because older people didn’t understand the technology. But over time I started to realize that it was more based on the fact that older people just don’t like sharing that kind of information about themselves. They’re not open books. Instead, Baby Boomers take a lot of studying to understand.

Zee Newest

On the opposite side of the spectrum is my son’s generation, Generation Z. GenZ is the opposite of the Boomer generation when it comes to privacy. They have grown up in a world that has never known anything but the ever-present connectivity of the Internet. They don’t understand that people can live a life without being watched by cameras and having everything they do uploaded to YouTube. Their idea of celebrity isn’t just TV and movie stars but also extends to video game streamers on Twitch or Instagram models.

Likewise, this generation is more open about their privacy. They understand that the world is built on data collection. They sign away their information. But they tend to be crafty about it. Rather than acting like previous generations that would fill out every detail of a form this generation only fills out the necessary pieces. And they have been known to put in totally incorrect information for no other reason than to throw people off.

GenZ realizes that the privacy genie is out of the bottle. They have to deal with the world they were born into, just like the Baby Boomers and the other generations that came before them. But the way that they choose to deal with it is not through legislation but instead through self-regulation. They choose what information they expose so as not to create a trail or a profile that big data consuming companies can use to fingerprint them. And in most cases, they don’t even realize they’re doing it! My son is twelve and he instinctively knows that you don’t share everything about yourself everywhere. He knows how to navigate his virtual neighborhood just a sure as I knew how to ride my bike around my physical one back when I was his age.

Tom’s Take

Where does that leave me and my generation? Well, we’re a weird mashup on Generation X and Generation Y/Millenials. We aren’t as private as our parents and we aren’t as open as our children. We’re cynical. We’re rebelling against what we see as our parent’s generation and their complete privacy. Likewise, just like our parents, we are almost aghast at the idea that our children could be so open. We’re coming to live in a world where Big Data is learning everything about us. And our children are growing up in that world. Their children, the generation after GenZ, will only know a world where everyone knows everything already. Will it be like Minority Report, where advertising works with retinal patterns? Or will it be a generation where we know everything but really know nothing because no one tells the whole truth about who they are?

A Review of RSA Conference

So, I recently went to my first RSA Conference. It’s something I’ve had on my radar for a while but never had the opportunity to do. However, with Security Field Day coming up later this year I thought it was high time I went to see what everything was about. Here are some ideas that I came up with during my pilgrimage to the big security conference.

  • It’s Huge. Like, really big. I’ve never seen a bigger conference before. I haven’t gone to Oracle OpenWorld or Dreamforce, but the size of the RSA show floor alone dwarfs anything I’ve seen. Three whole areas, including one dedicated to emerging vendors. That’s big. Almost too big in fact.
  • I Still Hate Moscone. It’s official. No conference should ever use this place again. It’s been 4 years since I railed against it and every word still applies. Doubly so this year, as RSA was being held during construction! Seriously. At this point, Moscone must be paying people to hold a convention there. RSA is too big. I don’t care if it’s cheap to ferry people up from Silicon Valley. Stop doing this to yourself and tarnishing your brand. Just go to Vegas if you want to stay close.
  • Everyone Has A Specialization. Maybe you’re developing a cloud plugin for lateral evasion detection. Or an endpoint hardening solution for mobile. Maybe you’re into detection, intrusion, pen testing, or perhaps just training. Everyone had representation of some kind or another. Everyone solved a problem in a unique or different way. But everyone had their niche. More than any other place I’ve been I saw companies doing all manner of things that are so specialized that I worry they’re either going to go out of business before they can develop a customer base or they’re going to get bought to add a feature onto someone’s existing platform. May fortune favor you all.
  • You Can’t Be Secure Enough. Funny enough, when the RSA Mobile app gets hacked you know someone’s having a bad day. The truth is that so many people out there are going to be taking shots at you when you’re visibility is high that you need to be prepared to take them. You can’t just assume you’re obscure enough to avoid detection. If Shodan has taught us anything, it’s that obscurity isn’t good enough any longer. You need to be looking at your posture and your weak areas. You need to be prepared to pare everything down to the point where no one can get in easily. Defense in depth needs to be very deep today.

Tom’s Take

RSA is similar to Cisco Live and VMworld, but it’s also much different. It’s bigger. People are more reserved. There are a TON of expo passes everywhere. I think I counted maybe 50 full conference passes the whole week. It’s too big for Moscone for sure. When the press room is located a block and half away from the convention center proper, you might need to rethink your strategy for a conference. But, RSA is still full of exciting companies and new ideas. And that’s going to keep it rolling along for a while no matter how many times the mobile app gets “hacked”.

It’s Time For Security Apprenticeships

Breaking into an industry isn’t easy. When you look at the amount of material that is necessary to learn IT skills it can be daunting and overwhelming. Don’t let the for-profit trade school ads fool you. You can’t go from ditch digger to computer engineer in just a few months. It takes time and knowledge to get there.

However, there is one concept in non-technical job roles that feels very appropriate to how we do IT training, specifically for security. And that’s the apprenticeship.

Building For The Future

Apprenticeship is a standard for electricians and carpenters. It’s the way that we train new people to do the work of the existing workforce. It requires time and effort and a lot of training. But, it also fixes several problems with the current trend of IT certification:

  1. You Can’t Get a Job Without Experience – Far too often we see people getting rejected for jobs at the entry level because they have no experience. But how are they supposed to get the experience without doing the job? IT roles paradoxically require you to be cheap enough to hire for nothing but expect you to do the job on day one. Apprenticeships fix this by showing that you’re willing to do the job and get trained at the same time. That way you can be properly trained on the job.
  2. You Need To Be Certified – Coupled tightly with the first problem is the certification trend. Yes, electricians and plumbers are certified to do the work. After 5 years of training. You don’t get to take a test after 3 months and earn Journeyman status. You have to earn it through training and work. It also ensures that every journeyman out there has the same basic level of work experience and training. Compare that with people getting a CCNA right out of college with no experience on devices or a desktop administrator position having an MCSE-level requirement to discourage people from applying.
  3. You Can’t Find a Mentor – This is a huge issue for entry-level positions. The people you work with are often so overtasked with work that they don’t have time to speak to you, let alone show you the ropes. By defining the role of the apprentice and the journeyman trainer, you can ensure that the mentoring relationship exists and is reinforced constantly through the training period. You don’t get through an apprenticeship in a vacuum.
  4. You Can’t Get Paid To Learn – This is another huge issue with IT positions. Employers don’t want to pay to train you. They want you to use the vast powers of the Internet to look up hacks on Stack Overflow and paste them into production equipment rather than buying a book for you or letting you take a class. Some employers are better than others about highly skilled workers, but those employers are usually leveraging those skills to make more money, like in a VAR or MSP. Apprenticeships fix this problem by including a classroom component in the program. You spend time in a classroom every other week while you’re doing the work.

Securing the Future

The apprenticeship benefits above go double for security-related professions. Whereas there is a lot of material to train networking and storage admins, security is more of a game of cat-and-mouse. As new protections are developed, we must always figure out of the old protections are still useful. We need to train people how to learn. We need to bring them up to speed on what works and what doesn’t and why googling for an old article about packet filtering firewalls isn’t as relevant in 2018 as it was a decade ago.

More than that, we need to get the security practitioners to a place where they can teach effectively and mentor people. We need to get away from the idea of just rattling off a long list of recommended protections to people as the way to train them about security. We need to make sure we’re teaching fundamentals and proper procedures. And they only way that can happen is with someone that has the time and motivation to learn. Like an apprentice.

This would also have the added luxury of ensure the next generation of security practitioners learns the pitfalls before they make mistakes that compromise the integrity of the data they’re protecting. Making a mistake in security usually means someone finding out something they aren’t supposed to know. Or data getting stolen and used for illicit means. By using apprenticeships as a way for people to grow, you can quickly bring people up to speed in a safe environment without worrying that their decisions will lead to problems with no one to check their work.

Tom’s Take

I got lucky in my IT career. I worked with two mentors that spent the time to train me like an apprentice. They showed me the ropes and made sure I learned things the right way. And I became successful because of their leadership. Today, I don’t see that mentality. I see people covering their own jobs or worried that they’re training a replacement instead of a valued team member. I worry that we’re turning out a generation of security, networking, and storage professionals that won’t have the benefit of knowing what to do beyond Google searches and won’t know how to teach the coming generation how to deal with the problems they face. It’s not too late to start by taking an apprentice under your wing today.

Memcached DDoS – There’s Still Time to Save Your Mind

In case you haven’t heard, there’s a new vector for Distributed Denial of Service (DDoS) attacks out there right now and it’s pretty massive. The first mention I saw this week was from Cloudflare, where they details that they were seeing a huge influx of traffic from UDP port 11211. That’s the port used by memcached, a database caching system.

Surprisingly, or not, there were thousands of companies that had left UDP/11211 open to the entire Internet. And, by design, memcached responds to anyone that queries that port. Also, carefully crafted packets can be amplified to have massive responses. In Cloudflare’s testing they were able to send a 15 byte packet and get a 134KB response. Given that this protocol is UDP and capable of responding to forged packets in such a way as to make life miserable for Cloudflare and, now, Github, which got blasted with the largest DDoS attack on record.

How can you fix this problem in your network? There are many steps you can take, whether you are a system admin or a network admin:

  • Go to Shodan and see if you’re affected. Just plug in your company’s IP address ranges and have it search for UDP 11211. If you pop up, you need to find out why memcached is exposed to the internet.
  • If memcached isn’t supposed to be publicly available, you need to block it at the edge. Don’t let anyone connect to UDP port 11211 on any device inside your network from outside of it. That sounds like a no-brainer, but you’d be surprised how many firewall rules aren’t carefully crafted in that way.
  • If you have to have memcached exposed, make sure you talk to that team and find out what their bandwidth requirements are for the application. If it’s something small-ish, create a policer or QoS policy that rate limits the memcached traffic so there’s no way it can exceed that amount. And if that amount is more than 100Mbit of traffic, you need to have an entirely different discussion with your developers.
  • From Cloudflare’s blog, you can disable UDP on memcached on startup by adding the -U 0 flag. Make sure you check with the team that uses it before you disable it though before you break something.

Tom’s Take

Exposing unnecessary services to the Internet is asking for trouble. Given an infinite amount of time, a thousand monkeys on typewriters will create a Shakespearean play that details how to exploit that service for a massive DDoS attack. The nature of protocols to want to help make things easier doesn’t make our jobs easier. They respond to what they hear and deliver what they’re asked. We have to prevent bad actors from getting away with things in the network and at the system level because application developers rarely ask “may I” before turning on every feature to make users happy.

Make sure you check your memcached settings today and immunize yourself from this problem. If Github got blasted with 1.3Tbps of traffic this week there’s no telling who’s going to get hit next.

Should We Build A Better BGP?

One story that seems to have flown under the radar this week with the Net Neutrality discussion being so dominant was the little hiccup with BGP on Wednesday. According to sources, sources inside AS39523 were able to redirect traffic from some major sites like Facebook, Google, and Microsoft through their network. Since the ISP in question is located inside Russia, there’s been quite a lot of conversation about the purpose of this misconfiguration. Is it simply an accident? Or is it a nefarious plot? Regardless of the intent, the fact that we live in 2017 and can cause massive portions of Internet traffic to be rerouted has many people worried.

Routing by Suggestion

BGP is the foundation of the modern Internet. It’s how routes are exchanged between every autonomous system (AS) and how traffic destined for your favorite cloud service or cat picture hosting provider gets to where it’s supposed to be going. BGP is the glue that makes the Internet work.

But BGP, for all of the greatness that it provides, is still very fallible. It’s prone to misconfiguration. Look no further than the Level 3 outage last month. Or the outage that Google caused in Japan in August. And those are just the top searches from Google. There have been a myriad of problems over the course of the past couple of decades. Some are benign. Some are more malicious. And in almost every case they were preventable.

BGP runs on the idea that people configuring it know what they’re doing. Much like RIP, the suggestion of a better route is enough to make BGP change the way that traffic flows between systems. You don’t have to be a evil mad genius to see this in action. Anyone that’s ever made a typo in their BGP border router configuration will tell you that if you make your system look like an attractive candidate for being a transit network, BGP is more than happy to pump a tidal wave of traffic through your network without regard for the consequences.

But why does it do that? Why does BGP act so stupid sometimes in comparison to OSPF and EIGRP? Well, take a look at the BGP path selection mechanism. CCIEs can probably recite this by heart. Things like Local Preference, Weight, and AS_PATH govern how BGP will install routes and change transit paths. Notice that these are all set by the user. There are not automatic conditions outside of the route’s origin. Unlike OSPF and EIGRP, there is no consideration for bandwidth or link delay. Why?

Well, the old Internet wasn’t incredibly reliable from the WAN side. You couldn’t guarantee that the path to the next AS was the “best” path. It may be an old serial link. It could have a lot of delay in the transit path. It could also be the only method of getting your traffic to the Internet. Rather than letting the routing protocol make arbitrary decisions about link quality the designers of BGP left it up to the person making the configuration. You can configure BGP to do whatever you want. And it will do what you tell it to do. And if you’ve ever taken the CCIE lab you know that you can make BGP do some very interesting things when you’re faced with a challenge.

BGP assumes a minimum level of competency to use correctly. The protocol doesn’t have any built in checks to avoid doing stupid things outside of the basics of not installing incorrect routes in the routing table. If you suddenly start announcing someone else’s AS with better metrics then the global BGP network is going to think you’re the better version of that AS and swing traffic your way. That may not be what you want. Given that most BGP outages or configurations of this type only last a couple of hours until the mistake is discovered, it’s safe to say that fat fingers cause big BGP problems.

Buttoning Down BGP

How do we fix this? Well, aside from making sure that anyone touching BGP knows exactly what they’re doing? Not much. Some Regional Internet Registrars (RIRs) require you to preconfigure new prefixes with them before they can be brought online. As mentioned in this Reddit thread, RIPE is pretty good about that. But some ISPs, especially ones in the US that work with ARIN, are less strict about that. And in some cases, they don’t even bring the pre-loaded prefixes online at the correct time. That can cause headaches when trying to figure out why your networks aren’t being announced even though your config is right.

Another person pointed out the Mutually Agreed Norms for Routing Security (MANRS). These look like some very good common sense things that we need to be doing to ensure that routing protocols are secure from hijacks and other issues. But, MANRS is still a manual setup that relies on the people implementing it to know what they’re doing.

Lastly, another option would be the Resource Public Key Infrastructure (RPKI) service that’s offered by ARIN. This services allows people that own IP Address space to specify which autonomous systems can originate their prefixes. In theory, this is an awesome idea that gives a lot of weight to trusting that only specific ASes are allowed to announce prefixes. In practice, it requires the use of PKI cryptographic infrastructure on your edge routers. And anyone that’s ever configured PKI on even simple devices knows how big of a pain that can be. Mixing PKI and BGP may be enough to drive people back to sniffing glue.

Tom’s Take

BGP works. It’s relatively simple and gets the job done. But it is far too trusting. It assumes that the people running the Internet are nerdy pioneers embarking on a journey of discovery and knowledge sharing. It doesn’t believe for one minute that bad people could be trying to do things to hijack traffic. Or, better still, that some operator fresh from getting his CCNP isn’t going to reroute Facebook traffic through a Cisco 2524 router in Iowa. BGP needs to get better. Or we need to make some changes to ensure that even if BGP still believes that the Internet is a utopia someone is behind it to ensure those rose colored glasses don’t cause it to walk into a bus.