Reranking: The Second Pass That Finds Relevance

Your vector search returned 50 chunks for one question. The chunk that actually answers the question is ranked 37th. Your model reads the top 5. Do the math on what your users are getting.

This is not an embedding failure. It is a job-description failure. Retrieval and reranking are two different jobs, and most RAG systems only hire for the first one.

Why similarity is not relevance

A retriever scores fast and blunt. It embeds the question and each chunk separately, then compares vectors. That design is what makes it fast enough to search millions of chunks, and it is also what makes it imprecise: the question and the chunk never actually meet.

The failure mode is predictable. A chunk that name-drops the question's keywords floats to the top whether or not it answers anything. A chunk with the real answer, written in different words, sinks. Cosine similarity measures topical nearness. Your users need relevance.

The second pass

Reranking adds a second, more careful pass over a small candidate set:

1. Retrieve broad. Pull a generous candidate set, say the top 50, with the fast retriever. Recall is the goal here; precision can come later.
2. Rerank precisely. A cross-encoder reads the question and each candidate together, as a pair, and scores true relevance. The chunk with the reset steps jumps to the top; the keyword-stuffed chunk sinks to the bottom, where it belongs.
3. Cut smart. Hand the model the top 5 after reranking, not before.

In the 60-second version of this lesson on our channel, the same eight chunks are scored both ways. Under retrieval scores they sit in one order; under reranker scores the answer-bearing chunk moves from the middle of the pack to position one. Same corpus, same question. The only change is the second pass.

What it costs, what it buys

The honest trade: reranking adds latency and a model call per query. On 50 candidates that cost is small, typically tens of milliseconds with a hosted reranker, and the quality jump is the largest single improvement most RAG systems have left on the table. Teams tune chunk sizes for weeks to gain less than one reranking pass delivers in an afternoon.

Rules of thumb from production work:

• Rerank when your retriever's top-k has the answer somewhere in it but not at the top. If the answer is not in the candidate set at all, your problem is retrieval or chunking, and reranking cannot save you.
• Keep the candidate set modest. Reranking 50 is cheap; reranking 500 is a latency budget with a search feature attached.
• Measure before and after with the same eval set. Reranking is easy to verify, which is exactly why it should be verified.

The rule

Retrieval finds candidates. Reranking finds relevance. Skip the second pass and your model reads the loudest chunks, not the right ones.