A field report. The sequel to the lab.
The last post was the lab. I mapped the attack surface of an AI system: where untrusted input gets in, how it reaches capability, what the model will and won’t do under pressure. That post answers the question what could go wrong?
This one answers a different question, and it’s the one I get asked in actual rooms: what is going wrong? The gap between the attack surface on a whiteboard and the attack surface in a running deployment is enormous, and it’s almost always the same handful of gaps.
So this is a field report. Patterns, not customers: everything here is generalized and composited, because the specifics belong to the people who trusted me with them. But if you run security at a company shipping AI right now, you’re going to recognize at least three of these, and you’re probably living in one of them today.
There’s a single pattern underneath all of them. Enterprises are taking a trust model they spent twenty years building for deterministic software and human operators, and applying it unchanged to a component that is neither. An LLM, especially an agentic one, is a probabilistic system whose entire job is to consume untrusted input and emit fluent, authoritative-sounding output, and increasingly to act on it. Every mistake below is a version of the same mistake: trusting that component somewhere you’d never trust the equivalent untrusted input, and failing to bound what it can do when it’s wrong.
They treat model output as trusted
This is the most common one by a wide margin. A team spends a decade drilling a single rule into everyone who touches the stack: never trust user input. Validate it, bound it, assume it’s hostile. Then they wire in a system whose whole function is to ingest untrusted input and produce output, and they trust the output completely.
It shows up as model output flowing straight into a consequential path with no gate. The model classifies a ticket and routing acts on the label. The model extracts fields from an uploaded document and those fields land in a database, or a workflow, or a number someone pays. The model “approves” something and something gets approved. In each case the output is treated as a verdict from a trusted control system, when it’s actually a probabilistic guess produced by a component an attacker can influence through its input.
Two ways that bites you. The benign one: the model is confidently, fluently wrong, and that confidence is indistinguishable from the times it’s right. The adversarial one, the one that connects straight back to the lab, is that the output was shaped by content the model ingested, which means whoever controlled that content just wrote directly into your trusted path.
The fix isn’t to stop using the output. It’s to demote it. Treat what the model emits the way you’d treat input from an untrusted user: validate it against something deterministic, constrain its range, and never let it be the sole authority for anything that matters. Model output is data. It is not a decision.
They hand agents broad tool access without thinking about the input-to-capability bridge
In the lab post I called this the bridge: the moment you connect untrusted input to a real capability, you’ve built a path where whoever controls the input can reach the capability. In the field, almost nobody builds that bridge on purpose. They build it by accident, one convenient tool grant at a time.
The story is always roughly the same. Someone stands up an agent. It needs to read email, so it gets email read. It would be handy if it could reply, so it gets send. It needs context, so it gets a tool that fetches web pages or pulls from a shared drive or reads tickets, arbitrary content from sources the company doesn’t control. And it needs to be useful, so it gets a tool that does something: files the ticket, hits the internal API, moves the record, runs the code. Nobody in that sequence ever asks the only question that matters for security: what is the worst thing someone who controls this agent’s inputs could make it do with these tools?
That’s the whole game. Prompt injection isn’t a model defect you’re waiting on a vendor to patch. It’s a structural property of the design. If content the agent reads can steer what the agent does, then anyone who can get content in front of the agent (an email, a web page, a document, a poisoned tool result) is a candidate operator of its tools. Enterprises scope tools to the developer’s convenience. They almost never scope them to the trust boundary.
The fix is to make capability a default-deny question. Every tool grant is who can reach this through the model’s inputs, and is that acceptable? Split the agent that swims in untrusted content from the agent that takes irreversible action, so the dangerous tools aren’t sitting on the same surface that reads the open internet. Keep consequential capabilities behind a deterministic check or a human. Earn each grant; don’t hand over the whole keyring because scoping it was more work.
They run AI systems with no logging and no detection
You would never put a payments service into production with no logs, no monitoring, and no alerting. People put AI systems into production exactly that way, constantly, because the AI gets treated as a product feature rather than as infrastructure that can be attacked.
The result is a black box in the worst sense. When something goes wrong (and with the first two patterns in play, something will), there’s no trail. Nobody recorded what went into the model, what it said, what tools it called and with what arguments, or what got acted on as a result. And here’s the part teams miss even when they do log something: the interesting security signal in an AI system usually isn’t in the final output. It’s in the inputs and the tool-call sequence. The retrieved document that carried the injection, the tool result that turned the agent, the moment an assistant that spent all day answering questions suddenly calls its send-external tool: that’s the signal. But it’s only a signal if you captured the sequence.
Most “AI logging” I see captures the wrong layer. It records that an API call was made to the provider, maybe with latency and token counts, and nothing about the semantic reality: what content drove what action. And there’s no detection tuned to this system at all: no alerting on anomalous tool sequences, no signatures for known injection patterns, no monitor for data leaving through the model’s output.
The fix is to log the semantic layer (inputs including retrieved content and tool results, outputs, tool calls with their arguments, and the actions taken) and then actually build detection on top of it, because logs nobody watches are just a more expensive black box. You can’t catch the abuse you didn’t record, and you can’t respond to the incident you can’t reconstruct.
They have no idea how much AI is already running
While the security team is carefully reviewing the one official AI project, the rest of the company has quietly deployed twenty more. This is shadow IT, but faster and far harder to see, because AI is the easiest powerful thing an employee has ever been able to adopt without asking.
It comes in three flavors and they’re all live in most orgs right now. Employees pasting sensitive data into consumer chatbots, which is a data-egress event that looks like nothing: no file moved, no alert fired, just a paste. Teams standing up their own agents and integrations with a corporate card and an API key, complete with tools and access that nobody reviewed. And vendors switching on AI features inside SaaS you already use, so software you approved two years ago is now processing your data through a model you never evaluated. The security team’s map of where does our data go, and what is allowed to act on it went stale the moment adoption outran governance, and it always outruns governance.
The two instinctive responses both fail. Ban it, and you drive it underground, where you have even less visibility. Ignore it, and it compounds. The thing that actually works is unglamorous: get visibility into what’s already running, then make the sanctioned path good enough and easy enough that people don’t route around it. The demand is real and it isn’t going away. You can meet it, or you can pretend it isn’t there while it grows. Those are the only two options, and only one of them keeps you in the loop.
They let the model be the only thing that approves a consequential action
The last one is the most expensive when it goes wrong. A system takes an action that’s hard or impossible to walk back (moves money, sends an external message, changes a permission, deletes a record, alters a config), and the only thing that decided to do it was the model. The thing that judges and the thing that approves are the same thing, which means there’s no independent check at all.
The principle this violates is old and simple: for anything that matters, whatever decides shouldn’t be the only thing that approves. You want a gate that doesn’t share the failure mode of the decider. If the model can be wrong or manipulated, the verification has to be something that can’t be wrong or manipulated in the same way: a deterministic rule, an independent system, a person.
And the subtle failure is the one I see most: the human-in-the-loop who isn’t actually verifying anything. There’s a person in the flow, but all they see is the model’s summary, and they approve based on that. That isn’t independent verification. It’s the model’s judgment with a rubber stamp attached, and if the summary is the thing that got manipulated, the human is approving the manipulation. Real verification means the person sees the underlying facts, not the model’s gloss on them. The other half of this is that teams don’t tier their actions at all: irreversible and reversible, high-stakes and trivial, everything gets the same gate, which is usually no gate.
The fix is to sort actions by consequence and reversibility, and put genuine independent verification (something that doesn’t share the model’s failure mode) in front of the ones that can hurt you. Show the human the raw facts. Make the irreversible things earn a real check.
The thread
Read those back to back and the common root is hard to miss. None of them is exotic. Each one is a team trusting the model somewhere they’d never trust an equivalent untrusted input, and failing to bound the damage when the model is wrong. That’s the whole field report.
Which is the part of my LinkedIn line I want to expand. Enterprises are racing to deploy faster than most of us are learning to secure it. The deploying is easy now. That’s the entire point of the tools. The securing lags because the discipline for this specific kind of component hasn’t been internalized yet, even by good teams, because the component breaks the categories those teams built their instincts on. It isn’t deterministic software and it isn’t a human operator, and the trust model for both fails on it in ways that look fine right up until they don’t.
The good news, and I mean this, is that the gap is closeable with discipline that mostly already exists. Least privilege. Log what matters. Verify consequential actions independently. Know what you’re running. None of that is new. It’s the oldest security advice there is. The only new part is recognizing that the model is a component these rules apply to, not a magic feature they don’t.
The teams getting this right aren’t the slow ones. They’re the ones who looked at the model, saw it for what it is, and built accordingly.