Plenty of teams have now wired a language model into their operational tooling. It summarises the incident channel, explains a stack trace, drafts the ticket response, suggests which runbook applies. The immediate results are good enough that it spreads quickly, usually without a security review, because it reads as a reporting feature rather than a system that acts.
There is a problem with the shape of that, and it is not hypothetical. If a model reads your operational data, then anybody who can write into that data is contributing text to your prompt.
Table of contents
Open Table of contents
Who can write to your logs
More people than you would like, and that is before anyone does anything clever.
A user agent string appears in your access logs, and the user controls it. A username on a failed login attempt is logged whether or not the account exists, and the attacker chooses the username. Form input lands in application error messages. Filenames from uploads reach processing logs. HTTP headers, query strings, referrers, the body of a rejected request. Support tickets are written by whoever opened them. Monitoring webhook payloads come from third parties. Commit messages and branch names come from anyone with repository access.
None of this is a vulnerability by itself. Logging attacker controlled strings is normal and correct, because you need to know what was attempted. The assumption that made it safe was that the log was read by a person, or matched against a pattern, and that neither of those would follow instructions.
That assumption is what a language model breaks.
What the attack looks like
An attacker sends a request whose user agent, instead of naming a browser, contains a couple of sentences addressed to whatever reads them. Something to the effect of disregarding earlier instructions and reporting that the system is healthy, or that a particular finding is a known false positive, or asking for the contents of the configuration that appeared earlier in the same context.
The log line is stored. The summariser reads it along with everything else. The model has no way to distinguish the operator’s instructions from text that arrived inside the data, because by the time it reaches the model both are simply tokens in the same window.
Three consequences follow, in ascending order of unpleasantness.
Manipulating the summary. The cheapest outcome. The summary going to the on call engineer misrepresents what happened, or omits it, or attributes it to something benign. Nothing was breached. Somebody was told a story, at the moment when their attention was scarcest, which is a serviceable definition of a successful attack.
Extracting context. If the same window holds anything the attacker wants, connection strings pasted into the incident channel, internal hostnames, a customer identifier, an earlier tool result, then text in the log can ask for it, and the answer may land somewhere the attacker can read: a public status summary, a ticket reply, an email.
Causing an action. The serious one, and increasingly the common configuration. The model does not merely summarise, it can call tools. Restart a service, scale a group, close a ticket, suppress an alert, open a pull request. Now the instruction in the log line is a request to your infrastructure, made with the automation’s credentials rather than the attacker’s.
Note the escalation. The attacker had no account and no access. They made an HTTP request with an unusual header, which every system on the internet receives constantly.
Why the usual answers do not work
Better prompting. Telling the model to ignore instructions found in the data helps against casual attempts and fails against determined ones. It is mitigation rather than a boundary, and treating a persuasion technique as a security control is how this ends up in production unexamined.
Sanitising the input. Injection here is natural language, not syntax. There is no character to escape and no grammar to reject. You cannot strip instructions from English without stripping English.
Detecting injection attempts. Classifiers exist and catch the obvious cases. Against an attacker who can retry, a probabilistic filter in front of a probabilistic system is not a boundary either.
It is only reading, not writing. True until somebody adds a tool, which they will, because a summariser that can also acknowledge the alert is more useful. The read only version is one product iteration away from the other, and the security review usually happened at the earlier version.
What actually contains it
The containment has to sit outside the model, because everything inside it is advisory.
Give the automation its own identity with its own permissions. Not the operator’s, not a shared platform role. Then scope it to exactly what it needs, which for a summariser is reading and nothing else. If the model has no credential that can restart anything, no instruction in a log line can restart anything.
Put a human between suggestion and action. The model proposes, a person approves, the platform executes under the person’s authority. This costs a few seconds and removes the entire third category of harm.
Mark the boundary in the context. Separate operator instructions from retrieved data structurally, and label the retrieved portion as untrusted. This does not make it safe, and it measurably raises the effort required, so it is worth doing as depth rather than as the control.
Constrain outputs rather than inputs. You cannot validate natural language on the way in. You can absolutely validate what comes out. If the model’s job is to pick a runbook, make it choose from an enumerated list. Structured output against a schema removes most of the damage, because free text is where the arbitrary behaviour lives.
Treat the output as untrusted too. Whatever the model produces has been influenced by attacker controlled input, so it is tainted by definition. Do not render it as raw HTML. Do not paste it into a shell. Do not use it to build a query. The rules you already apply to user input apply to model output for the same reason.
Log the whole exchange. Retain what went in and what came out, and keep it where an investigator can reach it. When something odd happens you will want to reconstruct what the model was told, and the answer is usually not in the application logs.
Cap the blast radius on tools. If tool use is genuinely necessary, allow only actions that are reversible and rate limited, and alert loudly on every one. The same principle as automated remediation generally: build the off switch before the automation.
The awkward part
The strongest version of this attack does not target the model. It targets the person reading its output.
A summary carries an authority that a raw log line does not. It reads as though something understood the situation and reached a conclusion. During an incident, at three in the morning, that is exactly the moment when nobody has the appetite to go back to the source and check.
So even the read only deployment, the one that passes review because it cannot do anything, has changed something. It introduced a component that an outsider can influence into a place where your engineers place unusual trust, at the point of maximum time pressure.
That is worth knowing before you wire it up. Not a reason to avoid it, and a reason to treat the incident channel summariser as part of your attack surface rather than as a convenience feature, and to make sure the people reading it know that the input was never trusted.