Getting Reliable JSON Out of a Language Model

The moment a language model stops writing prose for a human and starts producing input for a program, the requirement changes completely. Prose can be a bit off. A payload that your code will parse is either the shape you expected or an exception in production.

Why asking nicely fails

The first attempt is always some variation of:

Respond ONLY with valid JSON matching this shape. No markdown, no prose.
{ "sentiment": "positive" | "negative", "score": number }

This works most of the time, which is the trap — it works well enough in testing to ship, and then fails at some rate you never measured. The failures are consistent across models:

• The JSON arrives wrapped in a markdown code fence.
• A helpful preamble — “Here is the JSON you requested:” — precedes it.
• A trailing comma, or single quotes, or an unescaped quote inside a string.
• The response is truncated mid-object because it hit the token limit, producing valid-looking JSON that simply stops.
• score comes back as the string "0.8" instead of a number, or as "high".
• An enum value that is not in the enum — "mixed" when you allowed two options.

The response to this is usually a cleanup function that strips fences, finds the first { and last }, and attempts a repair. That code grows forever, because it is chasing a long tail of formatting accidents rather than addressing the cause: you asked for a constraint and received a suggestion.

Constrained decoding, which changes the guarantee

The mechanism worth understanding: a model generates one token at a time by sampling from a probability distribution over the vocabulary. Constrained decoding masks that distribution at each step so that only tokens allowed by your schema can be chosen.

If the schema says the next thing must be a number, every non-numeric token has its probability set to zero. Malformed output is not corrected after the fact — it is unreachable. The difference is between a request and a type system.

const Result = z.object({
  sentiment: z.enum(["positive", "negative", "neutral"]),
  score: z.number().min(0).max(1),
  rationale: z.string().max(280),
});

const parsed = await callModel({
  input,
  responseFormat: zodResponseFormat(Result, "result"),   // enforced at decode
});

Where the provider supports this, use it. It removes fences, preambles, trailing commas and invalid enum values as a category — not as bugs you handle, but as outputs that cannot be produced.

Two limits to know. Constrained decoding guarantees the shape, never the content — a schema-valid object can still say the wrong thing, and a required field the model has no basis for gets filled in with something plausible. And a truncated response is still possible if you run out of tokens mid-generation, so a length failure needs handling separately.

The schema is a prompt

Because the schema is passed to the model, its structure and naming shape the output as much as your instructions do. A few things reliably help.

Name fields as instructions. A field called oneSentenceSummary produces different output than one called summary. Field descriptions are read by the model — use them rather than repeating the rules in the prompt.

Enums over free text wherever the answer belongs to a known set. This is the single biggest reliability win, because it removes the entire space of near-miss values you would otherwise normalise.

Make uncertainty representable. If every field is required, the model must invent a value when the input does not support one. Give it somewhere honest to go — an optional field, or an explicit "unknown" member — and you convert silent fabrication into a signal you can branch on.

Keep it shallow. Deeply nested schemas produce more errors and consume more tokens. Two levels is comfortable; five is asking for trouble.

Put reasoning first if you want it. Fields are generated in order, so a rationale field before the verdict lets the model condition its answer on its own reasoning. Placed after, it is a post-hoc justification of an answer already committed to — the field order is doing real work either way.

Tool calling is the same mechanism, better framed

Tool or function calling is structured output with a different label: you describe the functions available and their parameter schemas, and the model emits a call rather than prose. The parameters are constrained the same way.

The framing matters because it changes what you are designing. Instead of “extract these fields,” you are defining an API that a non-deterministic caller will use — which means the same discipline as any public API:

• Few tools, clearly distinct. Twenty tools with overlapping purposes produce wrong selections. If two tools could plausibly serve the same request, merge them or rename until they could not.
• Descriptions written for the caller, stating when to use the tool and — more usefully — when not to.
• Validate inside the tool anyway. The schema constrains types, not semantics. A valid string can be another tenant's identifier, and the model is not an authorisation boundary.
• Return errors the model can act on. “No customer with that id; try searching by email” lets it recover. A stack trace does not.

This is also the boundary where an interoperability protocol earns its place: describing your tools once in a standard form, so any compatible client can discover and call them, rather than re-declaring the same schemas per integration.

Validate anyway, and retry deliberately

Even with constrained decoding, parse against your schema on receipt. The provider might not have applied the constraint, the model version might change under you, and content-level rules — a date that must be in the future, an id that must exist — are yours to enforce regardless.

const result = Result.safeParse(raw);
if (!result.success) {
  // one retry, with the actual error as context — not a blind re-roll
  return retryOnce({ raw, issues: result.error.issues });
}

Retry once, and feed the validation error back so the second attempt has information the first lacked. Blind retries burn latency and money to sample from the same distribution that just failed. And count these failures as a metric — a rising invalid-output rate is how you find out a model version changed beneath you.

Plan for the model declining

The case that gets skipped: what happens when the input does not contain what you asked for. A model handed an empty document and a schema requiring three fields will produce three plausible values, because the schema said they were required.

Design the refusal path into the schema itself — a top-level status field with an explicit “insufficient information” member, so declining is a valid, cheap, representable outcome rather than something the model has to fight the constraint to express. Then handle that branch properly instead of treating every response as an answer.

The short version

Prompting for JSON produces a failure rate you have not measured; use constrained decoding so malformed output is unreachable rather than repaired. Treat the schema as part of the prompt — descriptive names, enums over free text, shallow, with reasoning fields before conclusions. Design tools like an API for an unreliable caller, and never treat the model as an authorisation boundary. Validate on receipt, retry once with the error attached, and make “I cannot answer this” a representable result.