Do I still need to validate JSON from structured outputs?

Yes, validate the values in JSON from structured outputs, even when the provider guarantees the schema. A matching reply can still hold a value out of range or invented for a required field.

Does a JSON schema guarantee correct values from the model?

A JSON schema guarantees the shape and types of a model's output, not correct values. An integer field always holds an integer, but that integer can be wrong.

Should I call the real model to test structured outputs?

Call the real model only in the labelled-set check that runs on schema, prompt or model changes. Unit tests of parsing use recorded replies, so they run on every push without token cost.

How do I stop the model from inventing values for required fields?

Make a field nullable when the input may not contain its value, so the model can return null instead of a guess. Add inputs without that value to your labelled set.

Should I test structured outputs from an LLM?

Verdict

Yes

Test structured outputs from an LLM: unit-test the code that parses each output with recorded replies, including a refused reply, a cut-off reply and values the schema allows but your rules reject, and compare extracted fields with labelled inputs in CI.

Why

Blast radiususers
Change frequencyregularly
Detectabilityeventually
Reversibilitywith-effort
Test costmoderate

Test the code around structured outputs. My typical case is a feature where a hosted model reads a CV or a support email, returns JSON that matches a schema, and code stores the fields. Blast radius is users, because customers act on those fields. Change frequency is regularly, because the schema and prompt change about once a month. Detectability is eventually, because a wrong value of the right type passes the schema and stores without an error, and Reversibility is with-effort, because a script repairs the stored records. Test cost is moderate, because a recorded reply and a labelled field both compare exactly, so rule R11 gives Test.

When the decision changes
When	Decision	Why
The schema also has a free-text field, such as a summary or a suggested reply, where a wrong value is harder to catch	Test the free-text field differently: score it in an evaluation set on every prompt or model change	Test cost rises to heavy, because a summary has no exact expected value to compare
The structured output sets a refund amount that code pays out to the customer	Test mandatory: recorded outputs with an amount above the order total and a negative amount, asserting that code caps or rejects the refund	Blast radius rises to money and Reversibility to costly, because an overpaid refund needs a claw-back
The structured output names the customer account that an emailed document is filed under	Test mandatory: labelled emails that mention two customers, plus a check in code that the sender belongs to the named account	Blast radius rises to safety-or-legal and Detectability to never, because a document filed under another customer reads like a normal record
The structured output fills a form that the customer reads and corrects before pressing Save	Test minimally: one recorded-reply test of the main path, plus a regression test for each field that users report	Detectability moves to same-day and Reversibility to trivial, because the customer sees each value and nothing is stored before Save
Staff check every extracted record in an internal review screen before it is saved	Do not write structured-output tests; the staff member who checks each record is the check	Blast radius falls to internal, Detectability to same-day and Reversibility to trivial
The structured output runs once to tag an archive of old support tickets	Test the structured output differently: rehearse on a sample of 200 tickets and check the tags by hand	Change frequency moves to once, so a test in the suite would never run again

What breaks if you don't test

The schema guarantees shape, and shape is the part that rarely goes wrong. The model returns an integer where the text held no number, or a date in the wrong year. Your code stores it and nothing logs an error. A user finds the wrong value weeks later.

What you lose if you over-test

Tests that the provider returns valid JSON check code that the provider owns. Unit tests that call the real model cost tokens on every push and fail when a free-text field changes wording.

How to test

Save real replies as fixtures, plus a refused reply and a reply cut off at the token limit. OpenAI's structured outputs guide states that both cases can return output that does not match your schema.
Unit-test the parsing code with them: a refused or cut-off reply stores nothing.
Test each rule the schema cannot express. Anthropic's structured outputs documentation lists minimum and maximum among the constraints that decoding does not enforce, so put range checks in code, such as Pydantic validators.
Run the real model in CI against 50 labelled inputs, including inputs that lack a field, on every schema, prompt or model change, and fail below your pass rate.

Procedure and references

When the answer changes

A field in the output moves money or picks whose records the code reads.
A person reads every output before anything is stored.
The field you care about is free text with no single correct value.

Real incident + Code example

The candidates with zero years

On a recruiting app I worked on, the CV schema made years_experience a required integer. CVs that listed projects without dates came back with 0, because a required integer left the model nowhere to put "unknown". Recruiters filtered for three or more years, and those candidates vanished. Five weeks later a candidate asked a recruiter why nobody had called her. We found 260 profiles with an invented 0, made the field nullable, re-ran those CVs, and added these tests:

import json, pytest
from cv_parser.extract import parse_reply, ExtractionFailed

def reply(payload, stop="end_turn"):
    return {"stop_reason": stop, "text": json.dumps(payload)}

def test_cv_without_dates_keeps_experience_empty():
    cv = parse_reply(reply({"name": "Ana Ruiz", "years_experience": None}))
    assert cv.years_experience is None

def test_value_the_schema_allows_but_rules_reject():
    with pytest.raises(ExtractionFailed, match="years_experience"):
        parse_reply(reply({"name": "Ana Ruiz", "years_experience": 140}))

@pytest.mark.parametrize("stop", ["max_tokens", "refusal"])
def test_cut_off_or_refused_reply_is_not_stored(stop):
    with pytest.raises(ExtractionFailed):
        parse_reply({"stop_reason": stop, "text": '{"name": "Ana'})

Sources

Should I test LLM tool calls?Yes
Should I mock LLM calls in tests?Code under test: Yes
Should I test for hallucinations?Test it differently
Do I need evals for my LLM app?Test it differently
Should I test input validation?Yes

FAQ

Do I still need to validate JSON from structured outputs?: Yes, validate the values in JSON from structured outputs, even when the provider guarantees the schema. A matching reply can still hold a value out of range or invented for a required field.
Does a JSON schema guarantee correct values from the model?: A JSON schema guarantees the shape and types of a model's output, not correct values. An integer field always holds an integer, but that integer can be wrong.
Should I call the real model to test structured outputs?: Call the real model only in the labelled-set check that runs on schema, prompt or model changes. Unit tests of parsing use recorded replies, so they run on every push without token cost.
How do I stop the model from inventing values for required fields?: Make a field nullable when the input may not contain its value, so the model can return null instead of a guess. Add inputs without that value to your labelled set.