Citation and grounding

A RAG answer without a citation is no better than an ordinary LLM answer — the user cannot check anything.

Three levels of citation:

1. Document level: «according to the Handbook ch. 3». Easy to implement, hard to verify.
2. Passage level: [1] points at a specific fetched chunk that is shown to the user. The standard.
3. Claim level: every sentence has its own citation, and the system automatically checks that the passage really supports the sentence. The most expensive, the safest.

Grounding (groundedness) is the measure: what share of the claims in the answer are actually supported by what was fetched?

PROMPT = """Answer the question ONLY from the passages below.
- Follow every claim with [n], where n is the number of the passage.
- Use several citations if several passages support the claim: [1][3]
- If the passages are not enough, answer exactly: "That does not appear from the material."
- Never invent passage numbers.

{passages}

QUESTION: {question}
ANSWER:"""

import re

def validate_citations(answer, n_passages):
    """Catches the most common error: the model citing passages that do not exist."""
    nums = {int(n) for n in re.findall(r"\[(\d+)\]", answer)}
    invalid = {n for n in nums if not 1 <= n <= n_passages}
    sentences = [s for s in re.split(r"(?<=[.!?])\s+", answer.strip()) if s]
    uncited = [s for s in sentences if not re.search(r"\[\d+\]", s) and len(s.split()) > 5]
    return {"invalid_citations": sorted(invalid), "sentences_without_citation": len(uncited),
            "ok": not invalid and not uncited}

print(validate_citations("The battery life is 12 hours [1]. The price is 4 990 kr [7].", n_passages=4))
# {'invalid_citations': [7], 'sentences_without_citation': 0, 'ok': False}

A design choice worth making: let «That does not appear from the material» be an accepted answer in the eval suite, and measure how often it is used rightly and wrongly. An assistant that never declines sometimes lies; one that always declines is useless. That is a trade-off to be measured, not guessed.