Structured output and schemas

A program cannot use «The category is probably an invoice, and it seems urgent». It needs {"category": "invoice", "urgent": true}.

Three ways of getting that, in increasing reliability:

1. A prompt: «Answer ONLY with JSON according to … ». It usually works, it sometimes fails — the model adds explanations or code fences.
2. Function calling / JSON mode: the provider takes a JSON schema and guarantees syntactically valid JSON.
3. Grammar-constrained decoding (GBNF in llama.cpp, Outlines): at every token all the tokens that would break the schema are masked. The output cannot become invalid.

But valid JSON is not the same thing as correct JSON. Always validate semantically too.

import json, re
from pydantic import BaseModel, ValidationError, Field
from typing import Literal

class Case(BaseModel):
    category: Literal["invoice", "technical", "other"]
    urgent: bool
    summary: str = Field(max_length=200)

def extract(llm, text, attempts=2):
    schema = json.dumps(Case.model_json_schema(), ensure_ascii=False)
    prompt = f"Answer ONLY with JSON according to the schema.\nSCHEMA: {schema}\nTEXT: {text}"
    last_error = None
    for _ in range(attempts):
        out = llm(prompt if not last_error else f"{prompt}\n\nThe previous answer was invalid: {last_error}. Answer with valid JSON only.",
                  temperature=0, json_mode=True)
        cleaned = re.sub(r"^```(?:json)?|```$", "", out.strip(), flags=re.M).strip()
        try:
            return Case.model_validate_json(cleaned)
        except (ValidationError, json.JSONDecodeError) as e:
            last_error = str(e)[:200]
    raise ValueError(f"could not extract: {last_error}")

Four practical rules: use Literal instead of a free string where the set of values is known (then the model cannot invent categories), strip the code fences before parsing, send the error back on a retry (the model often corrects itself immediately), and have a defined behaviour when every attempt has failed — never a crash in the user's face.