Tag Archives: Patching

You Can’t Patch People

Posted on by
Reply

One of the things I’ve noticed when it comes to IT is how quickly we’re willing to use software to solve people problems. Over my career I’ve seen all manner of crazy solutions to get around people being lazy or uneducated. Remember vMotion? Or OTV for stretched layer 2? Why do you think those solutions came about? I posit that it’s because it’s faster to write software than to patch people.

Hacking Humans

I see this most often in cybersecurity. Developers love to create software solutions that prevent things from happening. Phishing and all its various forms are some of the top priorities for solutions that prevent leaking of information. While we have invested a lot in phishing tests and education it’s also very likely that there are controls in place that prevent users from accidentally giving out information to threat actors.

Why are we so willing to write software to fix problems instead of teaching people to avoid those issues? I think in part it’s because software is predictable. If I create an app or write some controls into a platform it’s going to behave the same way every time. That’s the definition of deterministic. Every time the software is presented with an input it will react the same way. That makes it easy to figure out. People that deal with risk on a daily basis just love predictability.

Humans are messy. We don’t always behave the same way every time. Even someone that knows they shouldn’t click on links in an email will do it because they aren’t paying attention or because they are tired. When you factor in how much better the phishing emails have gotten thanks to the advent of generative AI even the rank-and-file people are getting tricked. Developers would rather deal with software than trying to send more tests and update education resources.

The real issue is that we can’t patch people as easily as we can with software. If updating the filters for spam and phishing and other security related items was as simple as downloading the new attack vectors into someone’s brain we’d be doing that instead. Likewise, if we could just convince people to build things a certain way to avoid having to create complicated systems like FHRP we would be doing that instead of trying to solve for lazy developers.

Treating People Like Programs

Why is it so hard to patch people? Forget about the deterministic part of the equation for a moment. Software isn’t instantly updated when something is discovered. It takes time to develop lists of new vectors or update programs to remove vulnerabilities. Why can’t we do the same for people and reduce the overhead of all the extra software?

People can be “patched” with education. It isn’t always easy to get people to take courses or read the bulletins that are sent out. There are ways to force people to do it but that kind of friction just makes security teams resent users for trying to avoid mandatory training updates. Hence the reliance on software to fix the issues. But it doesn’t have to be like that.

Instead of forcing people to take updated training you could use something like gamification to encourage people to update training or learn about new issues. This is especially good with younger or newer employees that are used to the badge hunt mentality. Giving them the option to display achievements tied to training is a great way to encourage them to keep updated while also pulling others in that want to earn the same recognition.

Tom’s Take

I get the desire to rely on deterministic software rather than dealing with unreliable people. But there is only so much software that you can write to try and fix behaviors. We eventually have to get to a point where we can educate users and encourage them to want to keep up with it instead of forcing them to go through endless modules that don’t give them any real info. If we would just put in a bit of the effort we use on software controls into the people we’re trying to restrict we might find the effort is multiplied far beyond what we could hope for.

Is Patching And Tech Support Bad? A Response

Posted on by
3

Hopefully, you’ve had a chance to watch this 7 minute video from Greg Ferro about why better patching systems can lead to insecure software. If you haven’t, you should:

Greg is right that moral hazard is introduced because, by definition, the party providing the software is “insured” against the risks of the party using the software. But, I also have a couple of issues with some of the things he said about tech support.

Are You Ready For The Enterprise

I’ve been working with some Ubiquiti access points recently. So far, I really enjoy them and I’m interested to see where their product is going. After doing some research, the most common issue with them seems to be their tech support offerings. A couple of Reddit users even posted in a thread that the lack of true enterprise tech support is the key that is keeping Ubiquiti from reaching real enterprise status.

Think about all the products that you’ve used over the last couple of years that offered some other kind of support aside from phone or rapid response. Maybe it was a chat window on the site. Maybe it was an asynchronous email system. Hell, if you’ve ever installed Linux in the past on your own you know the joys of heading to Google to research an issue and start digging into post after post on a long-abandoned discussion forum. DenverCoder9 would agree.

By Greg’s definition, tech support creates negative incentive to ship good code because the support personnel are there to pick up the pieces for all your mistakes when writing code. It even incentivizes people to set unrealistic goals and push code out the door to hit milestones. On the flip side, try telling people your software is so good that you don’t need tech support. It will run great on anything every time and never mess up once. If you don’t get laughed out of the building you should be in sales.

IT professionals are conditioned to need tech support. Even if they never call the number once they’re going to want the option. In the VAR world, this is the “throat to choke” mentality. When the CEO is coming down to find out why the network is down or his applications aren’t working, they are usually not interested in the solution. They want to yell at someone to feel like they’ve managed the problem. Likewise, tech support exists to find the cause of the problem in the long run. But it’s also a safety blanket that exists for IT professionals to have someone to yell at after the CEO gets to them. Tech Support is necessary. Not because of buggy code, but because of peace of mind.

How Complex Are You?

Greg’s other negative aspect to tech support is that it encourages companies to create solutions more complex than they need to be because someone will always be there to help you through them. Having done nationwide tech support for a long time, I can tell you that most of the time that complexity is there for a reason. Either you hide it behind a pretty veneer or you risk having people tell you that your product is “too simple”.

A good example case here is Meraki. Meraki is a well-known SMB/SME networking solution. Many businesses run on Meraki and do it well. Consultants sell Meraki by the boatload. You know who doesn’t like Meraki? Tinkerers. People that like to understand systems. The people that stay logged into configuration dashboards all day long.

Meraki made the decision when they started building their solution that they weren’t going to have a way to expose advanced functionality in the system to the tinkerers. They either buried it and set it to do the job it was designed to do or they left it out completely. That works for the majority of their customer base. But for those that feel they need to be in more control of the solution, it doesn’t work at all. Try proposing a Meraki solution to a group of die-hard traditionalist IT professionals in a large enterprise scenario. Odds are good you’re going to get slammed and then shown out of the building.

I too have my issues with Meraki and their ability to be a large enterprise solution. It comes down to choosing to leave out features for the sake of simplicity. When I was testing my Ubiquiti solution, I quickly found the Advanced Configuration setting and enabled it. I knew I would never need to touch any of those things, yet it made me feel more comfortable knowing I could if I needed to.

Making a solution overly complex isn’t a design decision. Sometimes the technology dictates that the solution needs to be complex. Yet, we knock solutions that require more than 5 minutes to deploy. We laud companies that hide complexity and claim their solutions are simple and easy. And when they all break we get angry because, as tinkerers, we can’t go in and do our best to fix them.

Tom’s Take

In a world where software in king, quality matters. But, so does speed and reaction. You can either have a robust patching system that gives you the ability to find and repair unforeseen bugs quickly or you can have what I call the “Blizzard Model”. Blizzard is a gaming company that has been infamous for shipping things “when they’re done” and not a moment before. If that means a project is 2-3 years late then so be it. But, when you’re a game company and this is your line of work you want it to be perfect out of the box and you’re willing to forego revenue to get it right. When you’re a software company that is expected to introduce new features on a regular cadence and keep the customers happy it’s a bit more difficult.

Perhaps creating a patching system incentivizes people to do bad things. Or, perhaps it incentivizes them to try new things and figure out new ideas without the fear that one misstep will doom the product. I guess it comes down to whether or not you believe that people are inherently going to game the system or not.