You may have noticed last week that Ubiquiti added a new “feature” to their devices in a firmware updated. According to this YouTube video from @TomLawrenceTech, Ubiquiti built an new service that contacts a URL to “phone home” and check in with their servers. It got some heavy discussion going, especially on Reddit.

The consensus is that Ubiquiti screwed up here by not informing people they were adding the feature up front and also not allowing users to opt-out initially. The support people at Ubiquiti even posted a quick workaround of blocking the URL at a perimeter firewall to prevent the communications until they could patch in the option to opt-out. If this was an isolated incident I could see some manner of outcry about it, but the fact of the matter is that companies are adding these hidden features more and more every day.

The first issue comes from the fact that most release notes for apps any more are nothing aside from platitudes. “Hey, we fixed some bugs and stuff so turn on automatic updates so you get the best version of our stuff!” is somewhat common now when it comes to a list of changes. That has a lot to do with applications developers doing unannounced A/B testing with people. It’s not uncommon to have two identical version numbers of an app running two wildly different code releases or having dissimilar UIs just because some bean counter wants to know how well Feature X polls with a certain demographic.

Slipstreaming Stuff

While that’s all well and good for consumer applications, the trend is starting to seep into enterprise software as well. While we still get an exhaustive list of things that have been fixed in a release, we’re getting more than we bargained for on occasion as well. Doing a scrub of code to ensure that it actually fixes the bugs that you have in your environment is important for reliability purposes. When the quality of the code you’re trying to publish is of less-than-stellar quality, you have have to spend more time ensuring that the stated fixes aren’t going to cause issues during implementation.

But what about when the stated features aren’t the only things included? Could you imagine the nightmare of installing a piece of software to fix an issue only to find out that something that was hidden in the code, like a completely undocumented feature, caused some other issue elsewhere? And when I say “undocumented feature” I’m not using it as a euphemism for another bug. I’m talking about a service that got installed that no one knows about. Like a piece of an app that will be enabled at a later time for instance.

Remember Microsoft back in the early 90s? They were invincible. They had everything they wanted in the palm of their hand. And then their world exploded. They got sued by the US government in an anti-trust case. One of the things that came out of this lawsuit was a focus on undocumented features in Microsoft software. The government said that Microsoft was adding features to their application software that gave them performance advantages over other vendors. And only they knew about them.

Microsoft agreed to get rid of all undocumented features, including some of the Easter eggs that were hidden in their software. That irritated some people that enjoyed finding these fun little gifts. But in 2005, there was a great blog post that discussed why Microsoft has a strict “no Easter eggs” policy now. And reading it almost 15 years later makes it sound so prescient.

Security Sounds Simple, Right?

Undocumented features are a security risk. If there is code in your app that isn’t executing or hasn’t been completely checked against the interactions in your environment you’re adding a huge amount of risk. There can be interactions that someone doesn’t understand or that you have no way of seeing. And you can better believe that if the regulators of your industry find them before you do you’re going to have a lot of explaining to do.

That’s even if you notice it in the first place. How many times have you been told to whitelist a specific URL or IP stack to make something work? I can remember being told to whitelist 17.0.0.0/8 for Apple to make their push notification service work. That’s an awful lot of IP addresses to enable push notifications! And that’s a lot of data that could be corrupted or misused by a smart threat actor.

The solution we need to fix the problem is actually pretty simple. We need to push back against the idea that we can slip undocumented features into updates. Release notes need to list everyting that comes in the software package. We need to tell our vendors and companies that we have to have a full listing of the software features. And regulatory bodies need to be ready to share the blame when someone breaks those rules instead of punishing the people that had no idea there was something in the code that was misbehaving.

Tom’s Take

I’m not a fan of finding things I wasn’t expecting. At one point my wife and I had the latest Facebook app updates and somehow her UI looked radically different than mine. But if it’s on my phone or my home computer I don’t really have much to complain about. Finding undocumented apps and features in my enterprise software is a huge issue though. Security is paramount and undocumented code is an entry point to disaster. We are the only ones that can stem the tide by pushing back against this practice now before it becomes commonplace in the future.