Building Backdoors and Fixing Malfeasance

You might have seen the recent news this week that there is an exploitable backdoor in Zyxel hardware that has been discovered and is being exploited. The backdoor admin account with the clever name ‘zyfwp’ is not something that has been present in the devices forever. The account was put in during firmware version 4.60, which was released in Q4 2020.

Zyxel is rushing to patch the devices and remove the backdoor account. Users are being advised to disable remote administration until the accounts can be deactivated and proven to be removed. However, the bigger question in my mind relates to the addition of the user account in the first place. Why would you knowingly install a backdoor?

Hello, Joshua

Backdoors are nothing new in the computer world. I’d argue the most famous backdoor account in the history of computer hacking belongs to Joshua, the dormant login for the War Operations Programmed Response (WOPR) computer system in the 1983 movie Wargames. Joshua was an old login for the creator to access the system outside of the military chain of command. When the developer was removed from the project the account was forgotten about until a kid discovered it and kicked off the plot of the movie.

Joshua tells us a lot about developers and their desire to have access to the system. I’ll admit I’ve been in the same boat before. I’ve created my own logins to systems with elevated access to get tasks accomplished. I’ve also notified the users and administrators of those systems about my account and let them deal with it as needed. Most were okay with it being there. Some were hesitant and required it to be disabled after my work was done. Either way, I was up front about what was going on.

Joshua and zyfwp are examples of what happens when those systems are installed outside of the knowledge of the operators. What would have happened if the team in the Netherlands hand’t found the account? What if Zyxel devices were getting hacked and networks breached without anyone knowing the vector? I’m sure the account showed up in all the admin dashboards, right?

Easter Egg Hunts

Do you remember the Windows 3.1 Bear? It was a hidden reference in the credits to the development team’s mascot. You had to jump through a hoop to find it by holding down a keystroke combination and clicking a specific square in the Windows logo. People loved finding those little nuggets in the software all the way up to Windows 98.

What changed? Turns out, as part of Microsoft’s Trustworth Computing Initiative in 2002 they removed all undocumented features and code that could cause these kinds of things. It also might have had something to do with the antitrust investigations into Microsoft in the 1990s and how undocumented features in Windows and Office might have given the company a competitive advantage. Whatever the reason, Microsoft has committed to removing undocumented code.

Easter eggs are fun to find but represent the bright side of the dark issue above. What happens when the easter egg in question isn’t a credit roll but an undocumented account? What if the keystroke doesn’t bring up a teddy bear but instead gives the current user account full admin access? You scoff at the possibility but there’s nothing stopping a developer from making that happen.

These issues are part of the reason why all code and features need to be documented. We need to know what’s going on in the program and how it could impact us. This means no backdoors. If there is a way to access the system aside from the controls built in already it needs to be known and be able to be disabled if necessary. If it can’t be disabled then the users need to be aware of that fact and make the choice to not use the software because of security issues.

If you’re following along closely, you should have picked up on the fact that this same logic applies to backdoors that have been mandated by the government too. The current slate of US Senators seem to believe that we need to create keys that allow end-to-end encryption to be weakened and readable by law enforcement. However, as stated by companies like Apple for years, if you create a key for a lock that should only ever be opened under special circumstances you have still created a weakness that can be unlocked. We’ve seen the tools used by intelligence agencies stolen and used to create malware unlike anything we’ve ever seen before. What do you think might happen if they get the backdoor keys to go through encrypted messaging systems?


Tom’s Take

I don’t run Zyxel equipment in my home or anywhere I used to work. But if I did there would be a pile of it in the dumpster after this mess. Having a backdoor is one thing. Purposely making one is another. And having that backdoor discovered and exploited by the Internet is an entirely differently conversation. The only way to be sure that you’ve fixed your backdoor problem is to not have one in the first place. Joshua and zyfwp are what we need to get away from, not what we need to work toward. Malfeasance only stops when you don’t do it in the first place.

Securing Your Work From Home

UnlockedDoor

Wanna make your security team’s blood run cold? Remind them that all that time and effort they put in to securing the enterprise from attackers and data exfiltration is currently sitting unused while we all work from home. You might have even heard them screaming at the sky just now.

Enterprise security isn’t easy, nor should it be. We constantly have to be on the offensive to find new attack vectors and hunt down threats and exploits. We have spent years and careers building defense-in-depth to an artform not unlike making buttery croissants. It’s all great when that apparatus is protecting our enterprise data center and cloud presence like a Scottish castle repelling invaders. Right now we’re in the wilderness with nothing but a tired sentry to protect us from the marauders.

During Security Field Day 4, I led a discussion panel with the delegates about the challenges of working from home securely. Here’s a link to our discussion that I wanted to spend some time elaborating on:

Home Is Where the Exploits Are

BYOD was a huge watershed moment for the enterprise because we realized for the first time that we had to learn to secure other people’s devices. We couldn’t rely on locked-down laptops and company-issued phones to keep us safe. Security exploded when we no longer had control of the devices we were trying to protect. We all learned hard lessons about segmenting networks and stopping lateral attacks from potentially compromised machines. It’s all for naught now because we’re staring at those protections gathering dust in an empty office. With the way that commercial real estate agents are pronouncing a downturn in their market, we may not see them again soon.

Now, we have to figure out how to protect devices we don’t own on networks we don’t control. For all the talk of VPNs for company devices and SD-WAN devices at the edge to set up on-demand protection, we’re still in the dark when it comes to the environment around our corporate assets. Sure, the company Thinkpad is safe and sound and isolated at the CEO’s house. But what about his wife’s laptop? Or the kids and their Android tablets? Or even the smart speakers and home IoT devices around it? How can we be sure those are all safe?

Worse still, how do you convince the executives of a company that their networks aren’t up to par? How can you tell someone that controls your livelihood they need to install more firewalls or segment their network for security? If the PlayStation suddenly needs to authenticate to the wireless and is firewalled away from the TV to play movies over AirPlay, you’re going to get a lot of panicked phone calls.

Security As A Starting Point

If we’re going to make Build Your Own Office (BYOO) security work for our enterprise employees, we need to reset our goals. Are we really trying to keep everyone 100% safe and secure 100% of the time? Are we trying for total control over all assets? Or is there a level of insecurity we are willing to accept to make things work more smoothly?

On-demand VPNs are a good example. It’s fine to require them to access company resources behind a firewall in the enterprise data center. But does it need to be enabled to access things in the public cloud? Should the employee have to have it enabled if they decide to work on the report at 8:00pm when they haven’t ever needed it on before? These challenges are more about policy than technology.

Policy is the heart of how we need to rebuild BYOO security. We need to examine which policies are in place now and determine if they make sense for people that may never come back into the office. Don’t want to use the VPN for connectivity? Fine. However, you will need to enable two-factor authentication (2FA) on your accounts and use a software token on your phone to access our systems. Don’t want to install the apps on your laptop to access cloud resources? We’re going to lock you out until we’ve evaluated everything remotely for security purposes.

Policy has an important role to play. It is the reason for our technology and the driver for our work. Policy is why I have 2FA enabled on all my corporate accounts. Policy is why I don’t have superuser rights to certain devices but instead authenticate changes as needed with suitable logging. Policy is why I can’t log in to a corporate email server from a vacation home in the middle of nowhere because I’m not using a secured connection. It’s all relevant to the way we do business.

Pushing Policy Buttons

You, as a security professional, need to spend the rest of 2020 doing policy audits. You’re going to get crosseyed. You’re going to hate it. So will anyone you contact about it. Trust me, they hate it just like you do. But you have to make it happen., You have to justify why you’re doing things the way you’re doing them. “This is how we’ve always done it” is no longer justification for a policy. We’re still trying to pull through a global pandemic that has costs thousands their jobs and displaced thousands more to a home they never thought was going to support actual work. Now is not the time to get squeamish.

It’s time to scrub your policies down to the baseboards and get to cleaning and building them back up. Figure out what you need and what is required. Implement changes you’ve always needed to make, like software updates or applications that enhance security. If you want to make it stick in this new world of working from home you need to put it in place at the deepest levels now. And it needs to stick for everyone. No executive workarounds. No grace extensions for them to keep their favorite insecure apps or allowing them to not have 2FA enabled on everything. They need to lead by example from the front, not lag in the back being insecure.


Tom’s Take

I loved the talk at Security Field Day about security at home. We weighed a lot of things that people aren’t really thinking about right now because we haven’t had a major breach in “at home” security. Yet. We know it’s coming and if it happens the current state of network segementation isn’t going to be kind to whomever is under the gun. Definitely watch the video above and tell me your thoughts, either on the video comments or here. We can keep things safe and secure no matter where we are. We just need to think about what that looks like at the lowest policy level and build up from there.

Security and Salt

One of the things I picked up during the quarantine is a new-found interest in cooking. I’ve been spending more time researching recipes and trying to understand how my previous efforts to be a four-star chef have fallen flat. Thankfully, practice does indeed make perfect. I’m slowly getting better , which is to say that my family will actually eat my cooking now instead of just deciding that pizza for the fourth night in a row is a good choice.

One of the things I learned as I went on was about salt. Sodium Chloride is a magical substance. Someone once told me that if you taste a dish and you know it needs something but you’re not quite sure what that something is, the answer is probably salt. It does a lot to tie flavors together. But it’s also a fickle substance. It has the power to make or break a dish in very small amounts. It can be the difference between perfection and disaster. As it turns out, it’s a lot like security too.

Too Much is Exactly Enough

Security and salt are alike in the first way because you need the right amount to make things work. You have to have a minimum amount of both to make something viable. If you don’t have enough salt in your dish you won’t be able to taste it. But you also won’t be able to pull the flavors in the dish together with it. So you have to work with a minimum. Whether its a dash or salt or a specific minimum security threshold, you have to have enough to matter otherwise it’s the same as not having it at all.

To The Salt Mines

Likewise, the opposite effect is also detrimental. If you need to have the minimum amount to be effective, the maximum amount of both salt and security is bad. We all know what happens when we put too much salt into a dish. You can’t eat it at all. While there are tricks to getting too much salt out of a dish they change the overall flavor profile of whatever you’re making. Even just a little too much salt is very apparent depending on the dish you’re trying to make. Likewise, too much security is a deterrent to getting actual work done. Restrictive controls get in the way of productivity and ultimately lead to people trying to work out solutions that don’t solve the problem but instead try to bypass the control.

Now you may be saying to yourself, “So, the secret is to add just the right amount of security, right?” And you would be correct. But what is the right amount? Well, it’s not unlike trying to measure salt by sight instead of using a measuring device. Have you ever seen a chef or TV host pour an amount of salt into their hands and say it needs “about that much”? Do you know how they know how much salt to add? It’s not rocket science. Instead, it’s the tried-and-true practice of practice. They know about how much salt a dish needs for a given cooking time or flavor profile. They may have even made the dish a few times in order to understand when it might need more or less salt. They know that starches need more salt and delicate foods need less. Most importantly, they measured how much salt they can hold in their cupped hand. So they know what a teaspoon and tablespoon of salt look like in their palm.

How is this like security? Most Infosec professionals know inherently how to make things more secure. Their experience and their training tell them how much security to add to a system to make it more secure without putting too much in place to impede the operations of the system. They know where to put an IPS to provide maximum coverage without creating too many false positives. And they can do that because they have the experience to know how to do it right without guessing. Because the odds are good they’ve done it wrong at least one time.

The last salty thing to remember is that even when you have the right amounts down to a science you’re still going to need to figure out how to make it perfect. Potato soup is a notoriously hard dish to season properly. As mentioned above, starchy foods tend to soak up salt. You can fix a salty dish by putting a piece of a potato in it to soak up the salt. But is also means that it’s super hard to get it right when everything in your dish soaks up salt. But the best chefs can get it right. Because they know where to start and they know to test the dish before they do more. They know they need to start from a safe setup and push out from there without ruining everything. They know that no exact amount is the same between two dishes and the only way to make sure it’s right is to test until you get it right. Then make notes so you know how to make it better the next time.


Tom’s Take

Salt is one of my downfalls. I tend to like things salty, so I put too much in when I taste things. It’s never too salty for me unless my mouth shrinks up like a desiccated dish. That’s why I also have to rely on my team at home to help me understand when something is just right for them so I don’t burn out their taste buds either. Security is the same. You need a team that understands everything from their own perspective so they can help you get it right all over. You can’t take salt out of a dish without a massive crutch. And you can’t really reduce too much security without causing issues like budge overruns or costly meetings to decide what to remove. It’s better to get your salt and your security right in the first place.

Eventually Secure?

I have a Disney+ account. I have kids and I like Star Wars, so it made sense. I got it all set up the day it came out and started binge watching the Mandalorian. However, in my haste to get things up and running I reused an old password instead of practicing good hygiene. As the titular character might scold me, “This is not the way.” I didn’t think anything about it until I got a notification that someone from New Jersey logged into my account.

I panicked and reset my password like a good security person should have done in the first place. I waited for the usual complaints that people had been logged out of the app and prepared to log everyone in again and figure out how to remove my New Jersey interloper. Imagine my surprise when no one came to ask me to turn Phineas and Ferb back on. Imagine my further surprise when I looked in the app and on the Disney+ website and couldn’t find a way to see which devices were logged in to this account. Nor could I find a way to disconnect a rogue device as I could with Netflix or Hulu.

I later found out that this functionality exists but you have to call the Disney+ support team to make it happen. I also have no doubts that the functionality will eventually come to the app as more and more people are sharing account information so they can binge watch Clone Wars. However, this eventual security planning has me a bit concerned. And that concern extends beyond Mice and Mandalorians.

Minimum Secure Product

If you’re figuring out how to secure your newest application or a new building or even just a new user, you first have to figure out what “secure” looks like. If you have trouble figuring that out, all you need to do is look at your closest competitor. They will usually have a good baseline of the security and accessibility features you should have.

Maybe it’s basic device and user controls like the Disney+ example above. Maybe it’s encryption of your traffic end-to-end, as Zoom learned a couple of weeks ago. Or maybe it’s something as simple as ensuring that you don’t have a hard-coded backdoor password for SSH, like Fortinet remembered earlier this year. The real point is that you can survey the landscape and figure out what you need to do to make your product or app meet a minimum standard.

On the extremely off-chance that you’re developing something new and unique and never-before-seen in the world, you have a different problem. For one, you need to chill on the marketing. Maybe you’re using something in a novel and different way. But unless you’ve developed psychic powers or anti-gravity boosters or maybe teleportation you haven’t come up with anything completely unique. Secondly, you still have some references to draw on. You can look for similar things and use similar security controls.

If your teleport requires a login by a qualified person to operate you should look at login security for other industries that are similar to determine what is appropriate. Maybe it’s like a medical facility where you have two-factor authentication (2FA) with smart cards or tokens as well as passwords or biometrics. Maybe it’s a lockout system with two operators required to engage the mechanism so someone’s arm doesn’t actually get teleported away without the rest of them. Even if your teleport produces massive amounts of logs you should keep them lest someone show up on the other pad with a different color hair than when they left. Those logs may be different from anything ever seen before, but even Airbus knows how to store the flight data from every A380 flight.

Security isn’t a hard problem. It’s a series of challenges that must be overcome. All of them are rooted in common sense and discovery. Sure, you may not know all the problems right now. But you know what they look like in general and you also know what the outcome should look like. Common sense comes into play when you start thinking like a bad actor. If I were able to get into this app, what would I want to do? Maybe I want to sign up for the all-inclusive package and not get a confirmation sent to an account. So put a control in place that makes you confirm that. Sure, it reduces the likelihood that someone is going to sign up for something without realizing what they’ve done. But the side effect is that you also have happier customers because they were stopped from doing something they may not have wanted to do. Your security controls served a double purpose.


Tom’s Take

Ultimately, security should be about preventing bad or unwanted outcomes. Theft, destruction, and impersonation are all undesired outcomes of something. If your platform doesn’t protect against those you are not secure. If your process requires intervention to make those outcomes happen you’re not there yet. Disney+ could have launched with device reports and the ability to force logoff after password change. But the developers were focused on other things. It’s time for developers to learn how to examine what the minimum requirements are to be secure and ensure they’re included in the process along the way. We shouldn’t have to hope that we might one day become eventually secure.

Denial of Services as a Service

Hacking isn’t new. If you follow the 2600 Magazine culture of know the name Mitnick or Draper you know that hacking has been a part of systems as long as their have been systems. What has changed in recent years is the malicious aspect of what’s going on in the acts themselves. The pioneers of hacking culture were focused on short term gains or personal exploitation. It was more about proving you could break into a system and getting the side benefit of free phone calls or an untraceable mobile device. Today’s hacking cultures are driven by massive amounts of theft and exploitation of resources to a degree that would make any traditional hacker blush.

It’s much like the difference between petty street crime and “organized” crime. With a patron and a purpose, the organizers of the individual members can coordinate to accomplish a bigger goal than was ever thought possible by the person on the street. Just like a wolf pack or jackals, you can take down a much bigger target with come coordination. I talked a little bit about how the targets were going to start changing almost seven years ago and how we needed to start figuring out how to protect soft targets like critical infrastructure. What I didn’t count on was how effectively people would create systems that can cripple us without total destruction.

Deny, Deny, Deny

During RSA Conference this year, I had a chance to speak briefly with Tom Kellerman of Carbon Black. He’s a great guy and I loved the chance to chat with him about some of the crazy stuff that Carbon Black has been seeing in the wild. He gave me a peek at their 2020 Cybersecurity Report and some of the new findings they’ve been seeing. A couple of things jumped out at me during our discussion though.

The first is that the bad actors that have started pushing attacks toward critical infrastructure have realized that denying that infrastructure to users is just as good as destroying it. Why should I take the time to craft a beautiful and elegant piece of malware like Stuxnet to deny a key piece of SCADA systems when I can just use a cyptolocker to infect all the control boxes for a tenth of the cost? And, if the target does pay up to get things unlocked, just leave them there in a state of shutdown!

A recent episode of the Risky Business podcast highlights this to great effect. A natural gas processing plant system was infected and needed to be cleaned. However, when gas is flowing through the pipelines you can’t just shut off one site to have it cleaned. You have to do a full system shutdown! That meant knocking the entire facility offline for two days to restore one site’s systems. That’s just the tip of the iceberg.

Imagine if you could manage to shut down a hospital like the accidental spanning tree meltdown at Beth Israel Deaconess Medical Center in 2002. Now, imagine a cyptolocker or a wiper that could shut down all the hospitals in California during a virus outbreak. Or maybe one that could infected and wipe out the control systems for all the dams providing power for the Tennessee Valley Authority. Getting worried yet? Because the kinds of people that are targeting these installations don’t care about $5,000 worth of Bitcoin to unlock stuff. They care about causing damage. They want stuff knocked offline. Or someone that organizes them does. And the end goal is the same: chaos. It doesn’t matter if the system is out because of the malware or down for days or weeks to clean it. The people looking to benefit from the chaos win no matter what.

Money, Money, Money

The biggest key to this kind of attack is the same as it always has been. If you want to know where the problems are coming from, follow the money. In the past, it was following the money to the people that are getting paid to do the attacks. Today, it’s more about following the money to the people that make the money from these kinds of attacks. It’s not enough to get Bitcoin or some other amount of peanuts in an untraceable wallet. If you can do something that manipulates global futures markets or causes fluctuations in commodity prices on the order of hundreds of thousands or even millions of dollars you suddenly don’t care about whether or not some company’s insurance is going to pay out to unlock their HR files.

Think about it in the most simple terms. If I could pay someone to shut down half the refineries in the US for a month to spike oil prices for my own ends would that be worth paying a few thousand dollars to a hacking team to pull off? Better yet, if that same hacking team was now under my “protection” from retaliation from the target, do you think they’d continue to work for me to ensure that they couldn’t be caught in the future? Sure, go ahead and freelance when you want. Just don’t attack my targets and be on-call when I need something big done. It’s not unlike all those crazy spy movies where a government agency keeps a Rolodex full of assassins on tap just in case.


Tom’s Take

The thought of what would happen in a modern unrestricted war scares me. Even 4-5 years from now would be a massive problem. We don’t secure things from the kind of determined attackers that can cause mass chaos. Let’s just shut down all the autonomous cars or connected vehicles in NY and CA. Let’s crash all the hospital MRI machines or shut down all the power plants in the US for a day or four. That’s the kind of coordination that can really upset the balance of power in a conflict. And we’re already starting to see that kind of impact with freelance groups. You don’t need a war to deny access to a service. Sometimes you just need to hire it out to the right people for the right price.

Meraki Is Almost An Enterprise Solution

You may remember a three or so years ago when I famously declared that Meraki is not a good solution for enterprises. I know the folks at Meraki certainly haven’t. The profile for the hardware and services has slowly been rising inside of Cisco. More than just wireless with the requisite networking components, Meraki has now embraced security, SD-WAN, and even security cameras. They’ve moved into a lot of areas that customers have been asking about while also still trying to maintain the simplicity that Meraki is known for.

Having just finished up a Meraki presentation during Tech Field Day Extra at Cisco Live Europe, I thought it would be a good time to take a look at the progress that Meraki has been making toward embracing their enterprise customer base. I’m not entirely convinced that they’ve made it yet, but the progress is starting to look good.

Playing for Scale

The first area where Meraki is starting to really make strides is in the scalability department. This video from Tech Field Day Extra is all about new security features in the platform, specifically with firewalls. Take a quick look:

Toward the end of the video is one of the big things I got excited about. Meraki has introduced rule groups into their firewall platform. Sounds like a strange thing to get excited about, right? Kind of funny how the little things end up mattering in the long run. The real reason I’m getting excited about it has nothing to do with the firewall or even the interface. It has everything to do with being scalable.

One of the reasons why I’ve always seen Meraki as a solution that is more appropriate for small businesses is the lack of ability to scale. Meraki’s roots as a platform built for small deployments means that the interface has always been focused on making it easy to configure. You may remember from my last post that I wasn’t a fan of the way everything felt like it was driven through deployment wizards. Hand holding me through my first firewall deployment is awesome. Doing it for my 65th deployment is annoying. In enterprise solutions I can easily script or configure this via the command line to avoid the interface. But Meraki makes me use the UI to get it done.

Enterprises don’t run on wizards. They don’t work with assistance turned on. Enterprises need scalability. They need to be able to configure things to run across dozens or hundreds of devices quickly and predictably. They need that to happen quickly, too. Sure, it may only take four minutes to configure something via the firewall. Now, multiply that by 400 devices. Just that one little settings going to take over 26 hours to configure. And that’s assuming you don’t need to take time for a break or to sleep. When you’re working at the magnitude of an enterprise, those little shortcuts matter.

You might be saying right now, “But what about policies and groups for devices?” You would be right that groups can definitely speed up the process. But how many groups do you think the average enterprise would have for devices? I doubt all routers or switches or firewalls would conveniently fit into a single group. Or even ten groups. And there’s always the possibility that a policy change among those groups may get implemented correctly nine times out of those ten. The tenth time it gets an error that could still affect hundreds of devices. You see how this could get out of hand.

That’s why I’m excited about the little things like firewall groups. It means that Meraki is starting to see that these things need to be programmatically done. Building a series of policies in software makes it easy to deploy over and over again through scripting or enhanced device updating. Polices are good for rules. They’re not so good for devices. So the progress means that Meraki needs to keep building toward letting us script these deployments and updates across the entire organization.

Hextuple Option

The other thing that’s neatly buried at the end of the video is courtesy of a question from my friend Jody Lemoine (@GhostInTheNet). He points out that there are IPv6 addresses on the dashboard. The Meraki presenters confirm that they are testing IPv6 support natively and not just in bridged mode. Depending on when you read this post in the future, it may even be out already. You know that I’m an IPv6 fan and I’ve been tough on Meraki in the past about their support for it. So I’m happy to see that it’s in the works.

But more importantly I’m pleased that Meraki has jumped into a complex technical solution with both feet. Enterprises don’t need a basic set of services. They don’t want you to just turn on the twenty most common settings. Enterprises need odd things sometimes. They need longer VPN lifetimes or weird routing LSA support. Sometimes they need to do the really odd things because their thousand-odd devices really have to have this feature turned on to make it work.

Now, I’ve rightfully decried the idea that you should just do whatever your customers want, but the truth is that doing something silly for one customer isn’t the same as doing it for a dozen or more that are asking for a feature. Meraki has always felt shy to me about the way they implement features in their software. It’s almost the opposite of Cisco, in a way. Cisco is happy to include corner-case options on software releases on a whim to satisfy million-dollar customers. Meraki, on the other hand, has seemed to wait until well past critical mass to turn something on. It almost feels like you have to break down their door to get something advanced enabled.

To me, IPv6 is the watershed. It’s something that the general public doesn’t think they need or doesn’t know they really should have. Cisco has had IPv6 support in IOS for years. Meraki has been dragging along until they feel the need to implement it. But implementing it in 2020 makes me feel they will finally start implementing features in a way that makes sense for users. Hopefully that also means they’ll be more responsive to their Make A Wish feature requests and start indexing how many customers really want a certain feature or certain option enabled.

Napoleon Complex

The last thing that I’ll say about the transformation of Meraki is about their drive to embrace complexity. I know that Russ White and I don’t always see eye-to-eye about complexity. But I feel that hiding it is ultimately detrimental to IT staff members. Sure, you don’t want the CEO or the janitor in the wireless system deploying new SSIDs on a daily basis or disabling low data rates on APs. But you also need to have access to those features when the time comes. That was one of my big takeaways in my previous Meraki post.

I know that Meraki prides themselves on having a clean, easy-to-use interface. I know that it’s going to take a while before Meraki starts exposing their interface to power users. But, it also took Microsoft a long time to let people start doing massive modifications via PowerShell. Or Apple letting users go wild under the hood. These platforms finally opened a little and let people do some creative things. Sure, Apple IOS is still about as locked down as Meraki is, but every WWDC brings some new features that can be tinkered with here and there. I’m not expecting a fully complexity-embracing model in the next couple of years from Meraki, but I feel that the right people internally are starting to understand that growth comes in the form of enterprise customers. Enterprises don’t shy away from complexity. They don’t want it hidden. They want to see it and understand it and plan for it. And, ultimately, embrace it.


Tom’s Take

I will freely admit that I’m hard on the Meraki team. I do it because I see potential. I remember seeing them for the first time all the way back at Wireless Field Day 2 in their cramped San Francisco townhome office. In the years since the Cisco acquisition they’ve grown immensely with talent and technology. The road to becoming something more than you start out doing isn’t easy. And sometimes you need someone willing to stop you now and then and tell you where directions make more sense. I don’t believe for one moment that my armchair quarterbacking has really had a significant impact on the drive that Meraki has to get into larger enterprises. But I hope that my perspective has shown them how the practitioners of the world think and how they’re slowly transforming to meet those needs and goals. Hopefully in the next couple of years I can write the final blog post in this trilogy when Meraki embraces the enterprise completely.

The Future of Hidden Features

 

948AD2EC-79D1-4828-AF55-C71EA8715771You may have noticed last week that Ubiquiti added a new “feature” to their devices in a firmware updated. According to this YouTube video from @TomLawrenceTech, Ubiquiti built an new service that contacts a URL to “phone home” and check in with their servers. It got some heavy discussion going, especially on Reddit.

The consensus is that Ubiquiti screwed up here by not informing people they were adding the feature up front and also not allowing users to opt-out initially. The support people at Ubiquiti even posted a quick workaround of blocking the URL at a perimeter firewall to prevent the communications until they could patch in the option to opt-out. If this was an isolated incident I could see some manner of outcry about it, but the fact of the matter is that companies are adding these hidden features more and more every day.

The first issue comes from the fact that most release notes for apps any more are nothing aside from platitudes. “Hey, we fixed some bugs and stuff so turn on automatic updates so you get the best version of our stuff!” is somewhat common now when it comes to a list of changes. That has a lot to do with applications developers doing unannounced A/B testing with people. It’s not uncommon to have two identical version numbers of an app running two wildly different code releases or having dissimilar UIs just because some bean counter wants to know how well Feature X polls with a certain demographic.

Slipstreaming Stuff

While that’s all well and good for consumer applications, the trend is starting to seep into enterprise software as well. While we still get an exhaustive list of things that have been fixed in a release, we’re getting more than we bargained for on occasion as well. Doing a scrub of code to ensure that it actually fixes the bugs that you have in your environment is important for reliability purposes. When the quality of the code you’re trying to publish is of less-than-stellar quality, you have have to spend more time ensuring that the stated fixes aren’t going to cause issues during implementation.

But what about when the stated features aren’t the only things included? Could you imagine the nightmare of installing a piece of software to fix an issue only to find out that something that was hidden in the code, like a completely undocumented feature, caused some other issue elsewhere? And when I say “undocumented feature” I’m not using it as a euphemism for another bug. I’m talking about a service that got installed that no one knows about. Like a piece of an app that will be enabled at a later time for instance.

Remember Microsoft back in the early 90s? They were invincible. They had everything they wanted in the palm of their hand. And then their world exploded. They got sued by the US government in an anti-trust case. One of the things that came out of this lawsuit was a focus on undocumented features in Microsoft software. The government said that Microsoft was adding features to their application software that gave them performance advantages over other vendors. And only they knew about them.

Microsoft agreed to get rid of all undocumented features, including some of the Easter eggs that were hidden in their software. That irritated some people that enjoyed finding these fun little gifts. But in 2005, there was a great blog post that discussed why Microsoft has a strict “no Easter eggs” policy now. And reading it almost 15 years later makes it sound so prescient.

Security Sounds Simple, Right?

Undocumented features are a security risk. If there is code in your app that isn’t executing or hasn’t been completely checked against the interactions in your environment you’re adding a huge amount of risk. There can be interactions that someone doesn’t understand or that you have no way of seeing. And you can better believe that if the regulators of your industry find them before you do you’re going to have a lot of explaining to do.

That’s even if you notice it in the first place. How many times have you been told to whitelist a specific URL or IP stack to make something work? I can remember being told to whitelist 17.0.0.0/8 for Apple to make their push notification service work. That’s an awful lot of IP addresses to enable push notifications! And that’s a lot of data that could be corrupted or misused by a smart threat actor.

The solution we need to fix the problem is actually pretty simple. We need to push back against the idea that we can slip undocumented features into updates. Release notes need to list everyting that comes in the software package. We need to tell our vendors and companies that we have to have a full listing of the software features. And regulatory bodies need to be ready to share the blame when someone breaks those rules instead of punishing the people that had no idea there was something in the code that was misbehaving.


Tom’s Take

I’m not a fan of finding things I wasn’t expecting. At one point my wife and I had the latest Facebook app updates and somehow her UI looked radically different than mine. But if it’s on my phone or my home computer I don’t really have much to complain about. Finding undocumented apps and features in my enterprise software is a huge issue though. Security is paramount and undocumented code is an entry point to disaster. We are the only ones that can stem the tide by pushing back against this practice now before it becomes commonplace in the future.

The End of SD-WAN’s Party In China

As I was listening to Network Break Episode 257 from my friends at Packet Pushers, I heard Greg and Drew talking about a new development in China that could be the end of SD-WAN’s big influence there.

China has a new policy in place, according to Axios, that enforces a stricter cybersecurity stance for companies. Companies doing business in China or with offices in China must now allow Chinese officials to get into their networks to check for security issues as well as verifying the supply chain for network security.

In essence, this is saying that Chinese officials can have access to your networks at any time to check for security threats. But the subtext is a little less clear. Do they get to control the CPE as well? What about security constructs like VPNs? This article seems to indicate that as of January 1, 2020, there will be no intra-company VPNs authorized by any companies in China, whether Chinese or foreign businesses in China.

Tunnel Collapse

I talked with a company doing some SD-WAN rollouts globally in China all the way back in 2018. One of the things that was brought up in that interview was that China was an unknown for American companies because of the likelihood of changing that model in the future. MPLS is the current go-to connectivity for branch offices. However, because you can put an SD-WAN head-end unit there and build an encrypted tunnel back to your overseas HQ it wasn’t a huge deal.

SD-WAN is a wonderful way to ensure your branches are secure by default. Since CPE devices “phone home” and automatically build encrypted tunnels back to a central location, such as an HQ, you can be sure that as soon as the device powers on and establishes global connectivity that all traffic will be secure over your VPN until you change that policy.

Now, what happens with China’s new policy? All traffic must transit outside of a VPN. Things like web traffic aren’t as bad but what about email? Or traffic destined for places like AWS or Azure? It was an unmentioned fact that using SD-WAN VPNs to transit through the content filters in place in China was a way around issues that might arise from accessing resources inside of a very well secured country-wide network.

With the policy change and enforcement guidelines set forth to be enacted in 2020, this could be a very big deal for companies hoping to use SD-WAN in China. First and foremost, you can’t use your intra-company VPN functions any longer. That effectively means that your branch office can’t connect to the HQ or the rest of your corporate network. Given some of the questions around intellectual property issues in China that might not be a bad thing. However, it is going to cause issues for your users trying to access the mail and other support services. Especially if they are hosted somewhere that is going to create additional scrutiny.

The other potential issue is whether or not Chinese officials are even going to allow you to use CPE of your own choosing in the future. If the mandate is that officials should have access to your network for security concerns, who is to say they can’t just dictate what CPE you should use in order to facilitate that access. Larger companies can probably negotiate for come kind of on-site server that does network scanning. But smaller branches are likely going to need to have an all-in-one device at the head end doing all the work. The additional benefit for the Chinese is that control of the head end CPE ensures that you can’t build a site-to-site VPN anywhere.

Peering Into The Future

Greg and Drew pontificate a bit on the future on what this means for organizations from foreign countries doing business in China in the future. I tend to agree with them on a few points. I think you’re going to see a push for Chinese offices of major companies treating them like zero-trust endpoints. All communications will be trading minimal information. Networks won’t be directly connected, either by VPN substitute or otherwise.

Looking further down the road makes the plans even more murky. Is there a way that you can certify yourself to have a standard for cybersecurity? We have something similar with regulations here in the US were we can submit compliance reports for various agencies and submit to audits or have audits performed by third parties. But if the government won’t take that as an answer how do you even go about providing the level of detail they want? If the answer is “you can’t”, then the larger discussion becomes whether or not you can comply with their regulations and reduce your business exposure while still making money in this market. And that’s a conversation no technology can solve.


Tom’s Take

SD-WAN gives us a wonderful set of features included in the package. Things like application inspection are wonderful to look at on a dashboard but I’ve always been a bigger fan of the automatic VPN service. I like knowing that as soon as I turn up my devices they become secure endpoints for all my traffic. Alas, all the technology in the world can be defeated by business or government regulation. If the rules say you can’t have a feature, you either have to play by the rules or quit playing the game. It’s up to businesses to decide how they’ll react going forward. But SD-WAN’s greatest feature may now have to be an unchecked box on that dashboard.

Fast Friday- Black Hat USA 2019

I just got back from my first Black Hat and it was an interesting experience. It was crazy to see three completely different security-focused events going on in town all at once. There was Black Hat, B-Sides Las Vegas, and DEFCON all within the space of a day or so of each other. People were flowing back and forth between them all and it was quite amazing.

A wanted to share a few quick thoughts about the event from my perspective being a first timer.

  • The show floor wasn’t as bit as VMworld or Cisco Live, but it was as big as it needed to be. Lots of companies that I’ve heard of, but several more that were new to me. That’s usually a good sign of lots of investment in the security space.
  • Speaking of which, I talked to quite a few companies about a variety of analytics, telemetry, and insider threat monitoring solutions. And almost all of them had a founder from Israel or someone that was involved in the cybersecurity areas of the IDF. That’s a pretty good track record for where the investment is going.
  • The Vegas booth gimmicks never change. I think I’ve spent too much time at Vegas conferences because I’m starting to recognize the magicians and other “performers” at the booths. I’m glad they can get some work but I don’t know if the companies realize that there needs to be some new blood out there.
  • I found it very different that you could print pretty much any name on your badge that you wanted. I saw a few El Chapos, Pablo Escobars, and even a generic “IT Buyer”. Consequently, people were a little curious about my Twitter badge flag. I guess the idea of announcing your identity to people is a bit strange at a security conference.
  • Being on the press list for the event meant that I got to see some cool briefings. But it also meant sorting through some things that didn’t make sense. And there there was the Quasi-Prime Number presentation spam that I got. I don’t go into much more detail other than to point you to this Twitter thread which is a comedy goldmine of the presentation referenced in said email. Thanks to @MalwareJake for pointing out the original thread and all the amazing comments about how the harmony of music can be an input into crypto randomization.
  • Lastly, I wish I would have had more time to go down and check out DEFCON. A lot of my friends that were in town were there and seemed to be having the time of their lives. DEFCON seems more in line with my Batman job instead of my Bruce Wayne job though. Guess I’ll have to take some vacation to check out DEFCON next year.

Ultimately, I had a great time checking out Black Hat. There were some parts that needed polish and some things about having 20,000+ in Vegas that I’m not keen on. But it’s a successful conference and likely will be one I attend in 2020. If for no other reason than to give my VPN a workout again!

 

Home on the Palo Alto Networks Cyber Range

You’ve probably heard many horror stories by now about the crazy interviews that companies in Silicon Valley put you though. Sure, some of the questions are downright silly. How would I know how to weigh the moon? But the most insidious are the ones designed to look like skills tests. You may have to spend an hour optimizing a bubble sort or writing some crazy code that honestly won’t have much impact on the outcome of what you’ll be doing for the company.

Practical skills tests have always been the joy and the bane of people the world over. Many disciplines require you to have a practical examination before you can be certified. Doctors are one. The Cisco CCIE is probably the most well-known in IT. But what is the test really quizzing you on? Most people will admit that the CCIE is an imperfect representation of a network at best. It’s a test designed to get people to think about networks in different ways. But what about other disciplines? What about the ones where time is even more of the essence than it was in CCIE lab?

Red Team Go!

I was at Palo Alto Networks Ignite19 this past week and I got a chance to sit down with Pamela Warren. She’s the Director of Government and Industry Initiatives at Palo Alto Networks. She and her team have built a very interesting concept that I loved to see in action. They call it the Cyber Range.

The idea is simple enough on the surface. You take a classroom setting with some workstations and some security devices racked up in the back. You have your students log into a dashboard to a sandbox environment. Then you have your instructors at the front start throwing everything they can at the students. And you see how they respond.

The idea for the Cyber Range came out of military exercises that NATO used to run for their members. They wanted to teach their cyberwarfare people how to stop sophisticated attacks and see what their skill levels were with regards to stopping the people that could do potential harm to nation state infrastructure or worse to critical military assets during a war. Palo Alto Networks get involved in helping years ago and Pamela grew the idea into something that could be offered as a class.

Cyber Range has a couple of different levels of interaction. Level 1 is basic stuff. It’s designed to teach people how to respond to incidents and stop common exploits from happening. The students play the role of a security operations team member from a fictitious company that’s having a very bad week. You learn how to see the log files, collect forensics data, and ultimately how to identify and stop attackers across a wide range of exploits.

If Level 1 is the undergrad work, Cyber Range Level 2 is postgrad in spades. You dig into some very specific and complicated exploits, some of which have only recently been discovered. During my visit the instructors were teaching everyone about the exploits used by OilRig, a persistent group of criminals that love to steal data through things like DNS exfiltration tunnels. Level 2 of the Cyber Range takes you deep down the rabbit hole to see inside specific attacks and learn how to combat them. It’s a great way to keep up with current trends in malware and exploitive behavior.

Putting Your Money Where Your Firewall Is

To me, the most impressive part of this whole endeavor is how Palo Alto Networks realizes that security isn’t just about sitting back and watching an alert screen. It’s about knowing how to recognize the signs that something isn’t right. And it’s about putting an action plan into place as soon as that happens.

We talk a lot about automation of alerts and automated incident response. But at the end of the day we still need a human being to take a look at the information and make a decision. We can winnow that decision down to a simple Yes or No with all the software in the world but we need a brain doing the hard work after the automation and data analytics pieces give you all the information they can find.

More importantly, this kind of pressure cooker testing is a great way to learn how to spot the important things without failing in reality. Sure, we’ve heard all the horror stories about CCIE candidates that typed in debug IP packet detail on core switch in production and watched it melt down. But what about watching an attacker recon your entire enterprise and start exfiltrating data. And you being unable to stop them because you either don’t recognize the attack vector or you don’t know where to find the right info to lock everything down? That’s the value of training like the Cyber Range.

The best part for me? Palo Alto Networks will bring a Cyber Range to your facility to do the experience for your group! There are details on the page above about how to set this up, but I got a great pic of everything that’s involved here (sans tables to sit at):

How can you turn down something like this? I would have loved to put something like this on for some of my education customers back in the day!


Tom’s Take

I really wish I would have had something like the Cyber Range for myself back when I was fighting virus outbreaks and trying to tame Conficker infections. Because having a sandbox to test myself against scripted scenarios with variations run by live people beats watching a video about how to “easily” fix a problem you may never see in that form. I applaud Palo Alto Networks for their approach to teaching security to folks and I can’t wait to see how Pamela grows the Cyber Range program!

For more information about Palo Alto Networks and Cyber Range, make sure to visit http://Paloaltonetworks.com/CyberRange/