Structured output is not semantic correctness

A valid object can describe an impossible world

Imagine an assistant returning a booking object with a start date, an end date, a room identifier and a positive guest count. Every field can satisfy its declared type while the end precedes the start, the room does not exist, or the room has already been reserved. A syntactically valid object is not yet a valid booking. This is not a defect in schemas. It is a mismatch between the property the schema checks and the stronger property the application hoped to obtain from it.

JSON Schema defines assertions about instance structure and values, including types, required properties and numeric bounds. Its validation vocabulary does not consult your current inventory or prove that a sentence agrees with a cited document. I would keep those responsibilities explicit instead of describing all successful checks as validation in one undifferentiated sense. The application needs to know whether it has established well-formedness, local consistency, external factual support or permission to execute. Each answer supports a different next step and can fail for a different reason.

References: [1] JSON Schema 2020-12 Validation Vocabulary

Build a sequence of narrower contracts

My preferred pipeline starts by decoding the transport, parsing the representation and checking the schema. Then it applies deterministic domain rules, resolves references against trusted state and evaluates any evidence-dependent claims. Only after those checks does it authorize an external action. The stages need not be separate network services. Their value is conceptual and operational: a failure should identify the contract that failed. That makes it possible to retry generation for a malformed object without pretending a permissions failure can be repaired by a more persuasive prompt.

Consider an illustrative cost estimate with quantity, unit price and total. The schema can require nonnegative numeric fields. A domain validator can recompute total from quantity and price under the application’s rounding rule. A reference check can confirm that the price belongs to the requested product version. An authorization check can decide whether the caller may issue the resulting quote. If the model supplies a mathematically consistent but invented price, only the reference check catches it. More elaborate typing alone would leave the central factual error intact.

Make unsupported claims representable

A schema that requires an answer string and a confidence number can pressure the system to provide both even when no supporting evidence exists. I would include explicit alternatives such as supported, insufficient evidence and ambiguous request. The supported branch can require source identifiers, while the others carry missing information or a clarification target. This does not ensure that the model selects the correct branch. It does make the desired behavior expressible and gives downstream code a finite set of outcomes it must handle.

The distinction should survive rendering. If every branch is flattened into a string called answer, the application loses the property it just worked to establish. Similarly, a confidence field should have a defined meaning and provenance. A model-generated decimal is not automatically a calibrated probability. I would usually prefer an explicit reason for uncertainty over an unexplained number. Where probabilities are operationally useful, their calibration needs an evaluation procedure outside the generation schema, using outcomes relevant to the actual decision being made.

type Proposal = { productId: string; quantity: number; quotedTotal: number };
type VerifiedQuote = { productId: string; quantity: number; total: number; priceVersion: string };
function verify(p: Proposal, price: { productId: string; unit: number; version: string }): VerifiedQuote {
  if (p.productId !== price.productId) throw new Error('Wrong reference');
  if (!Number.isSafeInteger(p.quantity) || p.quantity < 1) throw new Error('Invalid quantity');
  if (!Number.isSafeInteger(price.unit) || price.unit < 0) throw new Error('Invalid price');
  const total = p.quantity * price.unit; // Integer minor units in this example.
  if (!Number.isSafeInteger(total) || p.quotedTotal !== total) throw new Error('Invalid total');
  return { productId: p.productId, quantity: p.quantity, total, priceVersion: price.version };
}

Recompute what does not need a model

If a quantity can be calculated deterministically from trusted inputs, I would calculate it after generation rather than ask the model to predict it. In a hypothetical invoice, three units at 1,299 minor currency units produce 3,897 before any explicitly defined adjustment. A model can help identify the intended product or explain a discrepancy, but arithmetic should not become uncertain merely because language was involved earlier. The generated object can carry the inputs, while the authoritative total is produced by ordinary code.

This principle also applies to identifiers, permissions and state transitions. A model may propose a record to inspect, but a trusted lookup determines whether it exists and whether the caller may access it. A model may suggest moving a task to completed, but the domain service determines whether required prerequisites are satisfied. The illustrative code assumes a trusted price lookup and still checks safe-integer, nonnegative minor units; a real domain should add explicit quantity and total caps. Reuse the existing domain operations rather than build a parallel set of rules exclusively for AI. The model should enter through the same meaningful boundary as other clients.

Treat descriptions and retrieved text as untrusted inputs

An attacker does not need to break JSON syntax to influence an application. They can place an instruction inside a description field that the model later interprets, or persuade the model to choose a dangerous value that remains perfectly valid under the schema. OWASP’s prompt-injection guidance is relevant because the trust boundary concerns behavior, not only representation. A valid object containing an unauthorized action is still unauthorized. Schema success should never be used as evidence that the content originated from a trusted instruction source.

I would keep model-produced values as data when passing them to other systems. Avoid constructing shell commands, SQL fragments or executable templates from free-form fields. Resolve typed operations through narrow functions with independent authorization and parameter checks. Also preserve provenance: a requested operation from the user and an operation suggested by a retrieved page should not arrive at the executor with indistinguishable authority. The output schema can help represent that distinction, but the application must prevent the model from granting itself a more trusted provenance label.

References: [2] OWASP LLM01: Prompt Injection

Version the meaning as well as the shape

A schema version can change without adding or removing a field. Suppose a duration once meant elapsed wall-clock seconds and later means billable active seconds. Both values fit the same numeric type, but existing consumers would compute different results. I would document semantic units, reference frames, rounding and missing-value behavior alongside the schema. When those meanings change, use a deliberate compatibility strategy rather than assuming that unchanged JSON keys imply an unchanged interface. A stable shape can conceal a breaking domain change.

External references also need versions when decisions depend on mutable state. A quote checked against one price list should not silently execute against another. A proposed configuration change validated against an old resource revision may need revalidation before application. The important object is therefore not just the generated proposal but the proposal plus the state against which it was checked. Carrying that relationship allows the executor to reject stale work and request a fresh decision, instead of applying a previously valid interpretation to a world that has moved on.

Test plausible failures, not only malformed JSON

A useful evaluation suite should include structurally valid outputs with wrong identifiers, reversed dates, unsupported claims, stale versions and unauthorized transitions. Malformed JSON is easy to notice; plausible errors are more likely to pass through a polished interface. I would generate fixtures for each validation layer and verify that the failure is attributed correctly. A missing property should not be confused with a nonexistent account, and a nonexistent account should not be confused with an account the user lacks permission to inspect.

Retries deserve separate tests. If a repair prompt receives a validation error, it should correct the failed proposal without inventing new evidence or changing the user’s intended operation. After a bounded number of attempts, the system should return an explicit failure state. Repeatedly regenerating until something passes can select an answer that satisfies superficial rules while drifting away from the request. I would log the versioned proposal and validator outcomes within an appropriate privacy boundary so that successful execution can be traced to the checks that actually justified it.

Keep the useful guarantee precise

Structured generation is still valuable. It reduces ambiguity at an interface, makes invalid states easier to reject and lets ordinary software consume a model’s proposal without fragile text extraction. The counterargument to my emphasis on limits is that it can undersell a practical improvement. I do not want to do that. I want the guarantee stated precisely enough that other components can rely on it. Knowing that a response conforms to a schema is much better than guessing where a value appears in prose.

The design becomes stronger when each component makes a smaller, testable promise. The generator proposes a representable result. The schema checks its shape. Domain logic checks relationships. Trusted references check facts and freshness. Authorization checks whether an effect is permitted. The executor applies the change under its own consistency rules. No single stage has to pretend it established everything. A well-typed answer is a useful object to inspect, and semantic correctness is the additional work that turns that object into a defensible decision.