Structured extraction from documents with JSON Schema, validation and bounded retries

Goal

Turn documents (invoices, contracts, forms, tickets) into records that downstream code can trust, with every field either valid or explicitly missing, and a measured accuracy per field.

Prerequisites

A JSON Schema for the record: required for fields that must be present; additionalProperties: false so the model cannot invent fields (the JSON Schema documentation describes the keyword as controlling properties not listed under properties); enumerations for closed value sets; explicit null where a value may be absent. A validator in the host language (Pydantic's model_validate_json in Python raises a ValidationError that names the failing fields). A labelled sample of documents for measuring.

Steps

1. Put the schema, field-by-field extraction rules and two or three examples in the prompt; state that missing values must be null, not guessed.
2. If the provider supports schema-constrained output, use it. The Claude documentation describes structured outputs as guaranteeing schema-compliant responses through constrained decoding and a strict mode for tool inputs. This removes syntax errors, not factual errors.
3. Validate every response against the same schema in code, even with constrained output, and add semantic checks the schema cannot express (totals add up, dates fall in range, an ID exists in the master data).
4. On failure, retry at most twice, feeding back the validator's message and the offending value; count retries.
5. Ask for provenance with each field: page number and a short quote. Check that the quote occurs in the extracted text of that page; a field whose quote is not found is marked unverified.
6. Route records that still fail after the retry limit, or whose critical fields are unverified, to a review queue; never fill defaults silently.
7. Measure per-field precision and recall on the labelled sample after every prompt or model change; version the sample.

Expected result

Records that pass schema and semantic validation, a per-field accuracy table, a retry rate, and a review queue that holds the genuinely ambiguous documents rather than random failures.

Limits and test basis

Constrained decoding enforces shape, not truth; a schema-valid record can still contain a wrong amount. Provenance checks by string match fail on scanned documents with OCR errors and on values the model normalised (dates, currencies); compare normalised forms. Accuracy figures are the reader's to measure; none are claimed.