Tag Archives: Context

AI Isn’t a Genie, It’s an Intern

Posted on by
Reply

Tell me if you’ve heard this one before. We build a super intelligent system and give it a specific goal like maximizing paperclip production. The system decides to do the job as well as possible by converting all available matter into paperclips. Everything. The Earth. The solar system. All of it. After the universe collapses the machine shuts down, content that it followed directions.

This is the malicious genie problem that every Dungeons & Dragons player knows. You find a lamp. You make a wish. The genie grants the wish in the most technical way possible and it leads to catastrophe. The lesson we are supposed to learn is that a sufficiently capable AI system will find ways to satisfy objectives to the letter while simultaneously not doing it the way you wanted. The foundation of AI safety is to prevent that from happening.

Nick Bostrom has covered this in Superintelligence. Eliezer Yudkowsky has argued this very thing for years. It’s a compelling story. But it has a hidden assumption that causes more confusion that it solves.

The Best Intentions

The genie tale is a story about intent. The genie knows exactly what you asked for. It just found the most technically valid way to grant your wish while ruining the spirit of it. This isn’t Robin Williams. This is some kind of demonic creature looking to teach you a lesson. The failure mode is adversarial in nature. Fantasy genies that aren’t in Disney movies are always looking for loopholes.

If the failure mode is intent, then the solution must be constraint. Build a system that can’t possibly do the bad thing. Be specific about every edge case. Build rules on top of rules. If we build the perfect box we can prevent the genie (or the AI) from going rogue and ruining our day.

The framework is sound if the assumption is correct. It works for a genie that knows better, right? But when we apply it to a modern AI we see where the gaps are. Genies are working against you. AI is not nearly that smart.

Doing My Best

Imagine a racing game where the objective is to score points. Collect points on the track and finish the race. Sounds simple, right? But what if a brilliant AI figures out that all they have to do to win is drive back and forth in front of the starting line collecting points and blocking other players from finishing? If it has the most points at the end it wins even if it never finishes.

In this case, the AI player isn’t like the genie above. It didn’t purposefully subvert your expectation. It did exactly what it was told to do. Score the most points. If you didn’t tell it that it had to drive through the whole course and cross the finish line then it didn’t know that it needed to do that. The win condition was points, not racing. The AI wasn’t missing intent. It was missing context.

This is an altogether different problem than the one above. Instead of assuming the system is going rogue and being obtuse the system genuinely doesn’t know what you mean and it tries to fill in the context with what it has available. Without the context that you assumed the system should have it just did what it could and you were flabbergasted by the results.

This is something that pops up at every level of the system. Context starvation and goal misalignment aren’t two different things. They’re sides of the same coin. When you don’t do a great job of being specific you usually get misalignment of your output.

Framing Your Reference

If you think the problem is intent you’re going to break out the constraints. Rules. Guardrails. Boxes. You’re going to spend your efforts on preventing the system from doing something bad. Security is about building walls.

If you think the problem is context you have a different outlook. You’re going to spend more time being specific. Less rules, more grounding. You want the system to surface ambiguous instructions instead of trying to resolve it with limited information. It’s like an intern being unclear on a task. You want the AI to ask you what to do instead of interpreting incomplete info.

It should be a cooperative inference problem. The system should always be just a little uncertain about what the operator wants and then seek to find out what is needed. The alternative is to confidently pursue a bad solution to a fixed objective because it thought it knew what you wanted without you telling it those details.

Knowing you have to do this doesn’t make agent building easier. In fact, it makes it a lot harder. It just makes the whole thing a lot more honest. Because you have to assume that people will never full specify what they want in advance no matter how much detail they provide. That’s not a failure of imagination. It’s the reality of how complex systems are implemented. There’s always some detail you miss. You shouldn’t aim to create the perfect spec up front. You should instead seek to build a system that is smart enough to know it’s missing context and will ask for it before running off to go to work.

It also means you have to treat an agent asking questions as a feature and not a failure. It’s like when your intern asks you to confirm that what you asked for is what you want. That’s not them being dumb. That’s them being sure they heard you right. And really, that helps you in the long run because if the system is asking you about specific areas you know your instructions must be a little thin in that area.

Tom’s Take

If you think your AI is a malicious genie you’re always going to be asking what rules have to be put in place to prevent it from going rogue. What you should be asking instead is “How can I give my agents enough context to do what I really mean?” The more powerful our AI agents get the wider the gap between those two things is going to be. But we can start to solve it today by building systems that ask questions when things are unclear. I promise you that you’re going to enjoy the results more than trying to close every loophole in your wishes.

Context Is Expensive

Posted on by
Reply

When it comes to learning and understanding, facts are easy. If I ask you how many bits are in an IPv4 address it’s a single answer. People memorize facts and figures like this all the time. It’s easy to recall them for tests and to prove you understand the material. Where things start getting interesting is when you need to provide context around the answer. Context is expensive.

Cognitive Costs

Questions with one correct answer or with a binary answer choice are easy to deal with cognitively. You memorize the right answer and move on with your life. IPv4 addresses are 32 bits long. The sun rises in the east. You like Star Wars but not Galatica 1980. These things don’t take much effort to recall.

Now, think about why those answers exist. Why does the sun rise in the east? Why are addresses 32 bits long? Why don’t you like Galactica 1980? The answers are much longer now. They involve nuance and understanding of things that are outside of the bounds of simple fact recall. For example, look at this video of Vint Cerf explaining why they decided on 32-bit addresses all the way back in the mid-1970s:

There’s a lot of context around a simple fact. For some of us it’s fun to learn the context and provide it at parties or when we’re trying to put someone to sleep with endless recitation of trivia. A lot of people won’t bother to care about the why and move on with their life.

You know who does care about the why? People building AI systems. When you think about it, answers that are facts are searchable. You would jump on Google or Bing or Duck Duck Go to find out where the sun rises or what time it rises today. But the answer to why is something that people building AI algorithms are working on. They want to be the search engine that provides the context. They want to overwhelm you with the reasoning and the justification behind something. And that increases cognitive load on the system.

If I asked you to explain why Star Wars is better than Galactica 1980 you could likely give me five reasons with justification quickly. But if I asked you to analyze each of those for underlying assumptions about science fiction and writing styles you’d take a little longer to come up with your answers. That extra level of reasoning increases our cognitive load. Now imagine a system working on all that cognitive load simultaneously. That’s where we are with AI right now.

Making Money

People know how to deal with overwhelming amounts of information. They can selectively discard what they don’t care about and focus on what matters. It’s why we can find the signal of a person’s conversation in the noise of a cocktail party. Our brains can filter when necessary to reduce cognitive load. AI isn’t as good at that right now which is why every piece of data included in a list of sources or a prompt has to be included somewhere. AI doesn’t know how to say “this is important” or “this is something we can forget about”. It’s better than it was before but it has a long way to go to behave a like intelligence.

The problem is that all of that processing costs resources. Power consumption, water consumption, and processing time are all used when a model is doing things. Models perform well when things are easy to produce, like with simple answers or with information that has already been generated. They fall down a bit when they have to do many things simultaneously. Just like with the human brain it can get overwhelming. Only the AI doesn’t do as good of a job as the brain of isolating the important parts.

The answer, at least according to AI researchers, is to just do it all. Burn a lot of those magical tokens to get every answer and every piece of context and just have it ready in case someone asks. It’s not unlike rehearsing a conversation in your head over and over again to get the perfect response to every question that could be asked. And yes, before you ask I’m the kind of person that does that. The cognitive load of exploring every conversation option is usually two or three times longer than the conversation itself. That’s time I’m not going to be get back.

The context around the answers incurs additional expense. We want to understand why and we’re willing to pay to get that detail. Right now it’s mostly free for us to play around with. The models are getting trained by what we ask and refining their ability to predict responses and understand how we think. What happens in the future when that model isn’t free any longer? Are we willing to pay to access the context? Are the providers going to force us to see ads to provider compensation for it?

Tom’s Take

I love context. I provide it all the time. Sometimes it’s not warranted or appreciated. But it’s always there. Because I want to know why something is the way it is. I’m the kind of person that is going to cause our modern systems to burn additional resources to sate my curiosity. Right now the model doesn’t look to be sustainable because we haven’t trained our algorithms to understand that sometimes the best part about being smart is knowing when not to be.

The Why of Security

Posted on by
1
Security is a field of questions. We find ourselves asking all kinds of them all the time. Who is trying to get into my network? What are they using? How can I stop them? But I feel that the most important question is the one we ask the least. And the answer to that question provides the motivation to really fix problems as well as conserving the effort necessary to do so.

The Why’s Old Sage

If you’re someone with kids, imagine a conversation like this one for a moment:
Your child runs into the kitchen with a lit torch in their hands and asks “Hey, where do we keep the gasoline?”
Now, some of you are probably laughing. And some of you are probably imagining all kinds of crazy going on here. But I’m sure that most of you probably started asking a lot of questions like:
  • – Why does my child have a lit torch in the house?
  • – Why do they want to know where the gasoline is?
  • – Why do they want to put these two things together?
  • – Why am I not stopping this right now?
Usually, the rest of the Five Ws follow soon afterward. But Why is the biggest question. It provides motivation and understanding. If your child had walked in with a lit torch it would have triggered one set of responses. Or if they had asked for the location of combustible materials it might have elicited another set. But Why is so often overlooked in a variety of different places that we often take it for granted. Imagine this scenario:
An application developer comes to you and says, “I need to you open all the ports on the firewall and turn off the AV on all the machines in the building.”
You’d probably react with an immediate “NO”. You’d get cursed at and IT would live another day as the obstruction in “real development” at your company. As security pros, we are always trying to keep things safe. Sometimes that safety means we must prevent people from hurting themselves, as in the above example. But, let’s apply the Why here:
  • – Why do they need all the firewall ports opened?
  • – Why does the AV need to be disabled on every machine?
  • – Why didn’t they tell me about this earlier instead of coming to me right now?
See how each Why question has some relevance to things? If you start asking, I’d bet you would figure some interesting things out very quickly. Such as why the developer doesn’t know what ports their application uses. Or why they don’t understand how AV heuristics are triggered by software that appears to be malicious. Or the value of communicating to the security team ahead of time for things that are going to be big requests!

Digging Deeper

It’s always a question of motivation. More than networking or storage or any other facet of IT, security must understand Why. Other disciplines are easy to figure out. Increased connectivity and availability. Better data retention and faster recall. But security focuses on safety. On restriction. And allowing people to do things against their better nature means figuring out why they want to do them in the first place. Too much time is spent on the How and the What. If you look at the market for products, they all focus on that area. It makes sense at a basic level. Software designed to stop people from stealing your files is necessarily simple and focused on prevention, not intent. It does the job it was designed to do and no more. In other cases, the software could be built into a larger suite that provides other features and still not address the intent. And if you’ve been following along in security in the past few months, you’ve probably seen the land rush of companies talking about artificial intelligence (AI) in their solutions. RSA’s show floor was full of companies that took a product that did something last year and now magically does the same thing this year but with AI added in! Except, it’s not really AI. AI provides the basis for intent. Well, real AI does at least. The current state of machine learning and advanced analytics provides a ton of data (the what and the who) but fails to provide the intent (the why). That’s because Why is difficult to determine. Why requires extrapolation and understanding. It’s not as simple as just producing output and correlating. While machine learning is really good at correlation, it still can’t make the leap beyond analysis. That’s why humans are going to be needed for the foreseeable future in the loop. People provide the Why. They know to ask beyond the data to figure out what’s going on behind it. They want to understand the challenges. Until you have a surefire way of providing that capability, you’re never going to be able to truly automate any kind of security decision making system.

Tom’s Take

I’m a huge fan of Why. I like making people defend their decisions. Why is the one question that triggers deeper insight and understanding. Why concentrates on things that can’t be programmed or automated. Instead, why gives us the data we really need to understand the context of all the other decisions that get made. Concentrating on Why is how we can provide invaluable input into the system and ensure that all the tools we’ve spent thousands of dollars to implement actually do the job correctly.