Reranking with cross-encoders

A bi-encoder encodes the question and the document separately and compares the vectors. That makes it possible to pre-encode the whole index — but the model never sees the question and the document together.

A cross-encoder feeds them in together and gives a relevance score. It sees how the words in the question relate to the words in the document, and is therefore considerably more accurate.

	Bi-encoder	Cross-encoder
The input	the question and the document separately	together, in the same sequence
Can pre-encode the index	yes	no
Calls per question	1	one per document
The quality	good	better
1 M documents	~10 ms	several hours

The solution is two-stage retrieval: let the bi-encoder fetch 50–100 candidates broadly and cheaply, and let the cross-encoder rerank them. You get the better model's quality at a cost that scales with 50, not with a million.

The architecture in practice:

question → BM25 + vector search → RRF → 50 candidates → cross-encoder → top 5 → LLM
            (~10 ms)                                     (~100 ms)

What it is worth. Typical improvements in nDCG@10 when reranking the top 50 lie at 10–20 % relative. The decisive effect is a different one, though: the precision in the top 3 rises sharply, and those are the documents that actually end up in the LLM prompt's context.

Three families of reranker:

The type	How	The cost
Cross-encoder (BGE, mxbai, Cohere Rerank)	one call per pair	medium
ColBERT (late interaction)	token-wise similarity, pre-encodable	low, but a larger index
An LLM as judge	ask a model to rank	high

ColBERT is an interesting middle way: it keeps one vector per token and computes the maximum similarity per query token. That gives nearly cross-encoder quality with pre-encoding — at the price of an index ten to a hundred times larger.

How many candidates should be reranked? It is a trade-off:

Candidates	The recall from stage 1	The latency
20	may miss the right document	low
50	usually the right balance	~100 ms
200	nearly always included	several hundred ms

The basic rule: a reranker can only improve what stage 1 fetched. If recall@50 is only 0.7 no reranking in the world can get above that — then it is the retrieval that should be improved, not the ranking.

Always measure both stages separately:

The measure	Tells you
recall@50 (stage 1)	the ceiling for the whole system
nDCG@10 before reranking	the starting point
nDCG@10 after	the gain
precision@3	what actually ends up in the prompt
The latency p50 and p95	the cost
The share of questions that got worse	the regressions

The last row holds here as it does for hybrid search: an improved average can hide that a fifth of the questions have deteriorated.

import time
import numpy as np
from sentence_transformers import CrossEncoder, SentenceTransformer

bi = SentenceTransformer("intfloat/multilingual-e5-base")
cross = CrossEncoder("BAAI/bge-reranker-v2-m3", max_length=512)

def retrieve(question, index, k=50):
    q = bi.encode(f"query: {question}", normalize_embeddings=True)
    return index.search(q, k)                      # [(doc_id, text, score), ...]

def rerank(question, candidates, top=5):
    pairs = [(question, c["text"]) for c in candidates]
    scores = cross.predict(pairs, batch_size=32)
    order = np.argsort(-scores)
    return [candidates[i] | {"reranker_score": float(scores[i])} for i in order[:top]]

def two_stage(question, index, retrieve_k=50, top=5):
    return rerank(question, retrieve(question, index, retrieve_k), top)

# Measure both stages separately — and the regressions
def evaluate(key, index, retrieve_k=50):
    ceiling, before, after, worse = 0, 0, 0, 0
    latency = []
    for f in key:
        cand = retrieve(f["question"], index, retrieve_k)
        ids = [c["id"] for c in cand]
        ceiling += int(f["right"] in ids)                    # the ceiling: recall@retrieve_k
        before += int(f["right"] in ids[:3])
        t0 = time.perf_counter()
        reranked = [c["id"] for c in rerank(f["question"], cand, top=3)]
        latency.append((time.perf_counter() - t0) * 1000)
        hit_after = f["right"] in reranked
        after += int(hit_after)
        worse += int((f["right"] in ids[:3]) and not hit_after)
    n = len(key)
    latency.sort()
    return {
        f"recall@{retrieve_k} (the ceiling)": round(ceiling / n, 3),
        "precision@3 before": round(before / n, 3),
        "precision@3 after": round(after / n, 3),
        "regressions": round(worse / n, 3),
        "reranker_p50_ms": round(latency[n // 2], 1),
        "reranker_p95_ms": round(latency[int(0.95 * n) - 1], 1),
    }

# How many candidates pay off? Run the curve on your own data.
def sweep_candidates(key, index, candidates=(10, 20, 50, 100, 200)):
    for k in candidates:
        r = evaluate(key, index, retrieve_k=k)
        print(f"  k={k:>3}: ceiling {r[f'recall@{k} (the ceiling)']:.3f}  "
              f"p@3 {r['precision@3 after']:.3f}  p95 {r['reranker_p95_ms']:>6.1f} ms")

The first row in the evaluation is the most important. If recall@50 sits at 0.70, then 0.70 is the ceiling for the whole system — the reranker can never find something stage 1 did not fetch. If that figure is low it is the retrieval that should be improved first.