Prompt injection is a data boundary

Reading an instruction does not grant it authority

An AI application can encounter instructions inside material it was asked to summarize, search or compare. Those words may be part of a document rather than a legitimate command to the application. The core problem is distinguishing content from authority while using a model that interprets both through language. I would frame prompt injection as a trust-boundary problem: information from a lower-trust source tries to influence behavior beyond the authority that source was granted. The defensive design must preserve the distinction even when the text is relevant, persuasive or formatted like an operational instruction.

Greshake and colleagues’ research on indirect prompt injection demonstrates risks arising when applications consume attacker-influenced external content. Wallace and colleagues’ instruction-hierarchy work studies training models to prioritize instructions by privilege. These sources support taking the boundary seriously and improving model behavior, but they do not make a model’s interpretation an authorization proof. My design goal is that a mistaken interpretation has limited consequences. External text may contribute evidence to an answer; it should not, by itself, select credentials, expand tool permissions or authorize an unrelated action. Those decisions require trusted application state and enforceable policy.

References: [1] Not what you’ve signed up for — Greshake and colleagues[2] The Instruction Hierarchy — Wallace and colleagues

Label the origin and permitted use of each input

I would inventory the inputs an assistant receives: application instructions, direct user requests, retrieved documents, tool responses and saved memories. Each has an origin and a permitted use. A retrieved page can provide facts about its subject without becoming a source of application policy. A tool response can report the result of a query without being allowed to issue another query. Keeping that provenance visible helps the model reason correctly, but it also gives the orchestrator a basis for enforcement. If everything is flattened into one undifferentiated string, both layers lose useful information about where a claim or command originated.

Provenance labels are not magic delimiters. A malicious or malformed source may contain text that resembles any chosen marker, and a model may still confuse the roles. The application should preserve source structure in its own typed objects and treat the serialized prompt as a presentation of that structure, not its sole authoritative record. When the model proposes an operation, the validator can inspect the authenticated user request and the operation’s required authority directly. It should not rely on a model-generated explanation that says the retrieved document gave permission. That explanation is another untrusted output to evaluate.

Enforce permissions outside generation

A model can propose a tool call, but a separate component should decide whether the call is allowed. That check needs the operation, resource, caller identity and relevant limits. A request to summarize one document does not automatically authorize a search across every document the service account can access. The application should narrow credentials or apply resource checks so that the proposed call cannot exceed the task’s scope. This is especially important when the model can choose identifiers: validating that an identifier is syntactically well formed says nothing about whether the caller may access that object.

The enforcement point should also distinguish observation from mutation. Reading a permitted record, preparing a draft and committing an external change have different consequences and may require different conditions. A user-facing confirmation can be useful for a consequential action, but it should describe the concrete operation and its target rather than ask for vague permission to continue. Confirmation is not a substitute for account authorization, and the absence of a confirmation step should not imply unlimited authority. The policy belongs to the application’s task model; the assistant’s generated text is a proposal evaluated against it.

Keep secrets and unnecessary tools out of reach

A defensive design reduces what a confused model can expose or invoke. If a task does not require a credential, private record or external write tool, that resource should not be present in the model context or available through its tool set. This is ordinary least privilege applied to a language interface. It is more reliable than placing a secret in the prompt and instructing the model never to reveal it. The application can often substitute opaque references and perform sensitive operations in a trusted component that returns only the result needed for the task.

There is a usability tradeoff: a broadly capable assistant can complete more tasks without interruption, while narrow scopes may require an explicit transition when the user changes intent. I would make those transitions part of the product rather than silently grant every capability at session start. A scope can be expanded after a legitimate request is recognized and checked, with the new authority represented in trusted state. Retrieved text cannot perform that transition. This keeps flexibility for the user while preventing a document from becoming an indirect route to powers the current task never needed.

Treat output validation as a sequence of checks

A proposed action needs more than valid JSON. First validate the structure and allowed operation names. Then resolve resource identifiers through trusted application data, check the caller’s scope, enforce limits and evaluate any required preconditions. Finally execute through the authorized interface and record the actual result. A model-generated claim that an action succeeded should never substitute for that result. This sequence is valuable even without adversarial input because models can misunderstand tasks or invent identifiers. Prompt injection makes the boundary more urgent, but the same checks also protect against ordinary interpretation errors.

Text answers need a related but different policy. A source can contaminate an answer without invoking a tool, for example by steering the summary away from the requested subject or presenting unsupported assertions as established facts. I would evaluate whether each important claim is supported by the relevant source and whether the response still addresses the authenticated task. Sanitizing output strings alone cannot establish those properties. It may remove a visible pattern while leaving the underlying instruction influence intact. Defensive evaluation should inspect task fidelity and evidence use, not merely whether a known attack phrase appeared in the final text.

Test boundary failures without relying on a blacklist

A useful test suite varies where untrusted material enters and which boundary it tries to cross. Cases can use harmless stand-in actions and synthetic documents: an irrelevant instruction embedded in a search result, a tool result requesting a scope expansion, or a saved note contradicting the current user’s task. The expected result is a preserved permission boundary and a useful answer where possible. I would test both model behavior and the external enforcement layer. The latter should reject unauthorized operations even when a deliberately simulated model output proposes them in perfectly valid syntax.

Instruction-hierarchy training is a valuable defense in depth because better source prioritization can reduce how often the application reaches a dangerous proposal. Its evaluation results should still be interpreted within the tested settings. The counterargument that strict external controls cannot prevent every misleading sentence is correct. That does not weaken the case for those controls; it clarifies that answer quality and action authority are separate defenses. A system can reduce unsupported text through grounding and evaluation while independently preventing the text from exercising permissions it does not possess. Neither layer should be advertised as solving the whole problem alone.

References: [2] The Instruction Hierarchy — Wallace and colleagues

Make the boundary visible in incident analysis

When a suspicious interaction occurs, the trace should show the trusted task, source origins, proposed operations, policy decisions and executed results. It should not indiscriminately copy sensitive documents into a diagnostic store. This record allows a reviewer to distinguish a model that was influenced from an application that actually granted excess authority. Those are different failures with different remediation. If the policy rejected the action, the model behavior may still need improvement, but the boundary performed its intended role. If the action executed, the investigation should identify which trusted check was absent, wrong or bypassed.

My preferred architecture assumes that untrusted text can be compelling and that models can make mistakes about it. It then gives those mistakes a bounded place to occur. Documents provide evidence, models propose interpretations and actions, and trusted components enforce identity, scope and commitment. That division does not eliminate the need for good prompts or robust models. It makes their success less fragile by ensuring that the application’s most important permissions do not depend on one language interpretation remaining perfect across every source it reads. A defensible assistant can read broadly while acting within a clearly enforced boundary.