The context window, system prompts and few-shot

The model sees only what fits in the context window (8 k–200 k tokens, say): the system prompt + the history + your question + the documents. Anything outside it does not exist.

• The system prompt sets the role, the rules and the format — it holds for the whole conversation.
• Few-shot: 2–5 examples of in → out steer the format better than descriptions do.
• Structured output: ask for JSON with a schema → the program can read the answer.

Placement: models are best at the beginning and the end of the context («lost in the middle»). Put the instructions first, the most important document last, and the question at the very end.

messages = [
  {"role": "system", "content": "You classify customer messages. Answer with JSON ONLY: {\"category\": \"invoice|technical|other\", \"urgent\": true|false}."},
  {"role": "user", "content": "My invoice is twice as high as usual"},
  {"role": "assistant", "content": '{"category": "invoice", "urgent": false}'},
  {"role": "user", "content": "The app crashes every time I log in!!"},
  {"role": "assistant", "content": '{"category": "technical", "urgent": true}'},
  {"role": "user", "content": "How do I change my password?"},
]
# → '{"category": "technical", "urgent": false}'
import json; answer = json.loads(text)   # always validate — and have a fallback if it is not JSON

The budget: count the tokens (with the tokenizer) before the call; cut the history from the start, never the system prompt.