Why do we need rerankers after vector search?

#The problem

A retrieval pipeline that does the obvious thing works well enough in a demo to feel finished: embed the corpus, embed the query, take the nearest neighbours, hand them to the model.

In use it fails in a way that is oddly hard to describe. The right document is usually somewhere in the results. It is just not near the top, and not often enough inside the slice that fits in the context window. Answer quality is not bad in a way that suggests a bug. It is bad in a way that suggests the system does not quite understand the question.

The fix is a reranking stage. This note is about why that stage has to exist at all, rather than being something a better embedding model would eventually make unnecessary.

#What I thought was happening

My mental model was a similarity ranking. The embedding model, I assumed, read the query, read each document, and produced a score for how well the second answers the first. Nearest neighbours were therefore the best answers, in order, and anything wrong with the ordering was a model quality problem: use a better encoder, get a better ordering.

That model is wrong in one specific way, and everything else follows from it.

#What is actually happening

A bi-encoder never sees a query and a document together.

Documents are embedded at index time, before any query exists. A document's vector is a fixed summary of that document, written without knowing what will be asked of it. At query time the query is embedded independently, and the comparison is a dot product between two vectors produced in isolation.

That constraint is not incidental. It is the entire reason vector search is fast: the expensive work happens once, offline, and a query becomes an approximate nearest-neighbour lookup over a prebuilt index. Millions of documents, single-digit milliseconds.

But it means a document vector has to be a general-purpose summary. A page about connection pooling that happens to contain three sentences on timeout defaults gets one vector, dominated by connection pooling. Ask about timeout defaults and that page sits somewhere in the neighbourhood, competing against pages that are entirely about timeouts and are much closer to the query in the space, whether or not they actually answer it.

The dot product answers "is this document about the same sort of thing as this query?" I was reading it as "does this document answer this query?" Those two questions diverge exactly where it matters: specific questions asked against long, mixed documents.

#What a cross-encoder does differently

A cross-encoder takes the query and one document as a single input and runs attention across both. Every token of the question can attend to every token of the passage. There is no fixed summary in between, so nothing has to be decided before the question is known.

# Bi-encoder: two independent passes, compared afterwards.
doc_vec   = encode(document)        # once, at index time
query_vec = encode(query)           # once, per query
score     = dot(query_vec, doc_vec)
 
# Cross-encoder: one pass over the pair.
# There is nothing to precompute here, because the input does not exist
# until the query arrives. That is the whole cost, and the whole benefit.
score = model(query, document)      # per query, PER DOCUMENT

The cost is in that last line. Scoring becomes O(candidates) model passes per query instead of one vector lookup. Running a cross-encoder over a whole corpus is not slow, it is infeasible.

Hence the two-stage shape. Stage one is allowed to be approximate but must be cheap enough to run over everything. Stage two is allowed to be expensive because it only ever sees a shortlist.

#The experiment worth running

The cheapest way to see whether a reranker will help you is to stop measuring the pipeline and start measuring the boundary between its stages.

For a set of questions with known good answers, ask two separate questions of your own system:

1. Is the right document in the top k at all? That is recall@k, where k is the shortlist size you would feed a reranker.
2. Where in that shortlist does it land? That is MRR or nDCG@k.

Those two numbers tell you which problem you have, and they point at completely different fixes:

#The mental model that finally made it click

The two stages are not two attempts at the same job. They optimise different metrics, and they should be measured separately.

• Stage one is a recall problem. Its only job is to get the right document into the candidate set. Whether it lands at position 4 or position 80 does not matter, because stage two is going to reorder them anyway.
• Stage two is a precision problem. It receives a set that usually already contains the answer, and its job is to move it into the top few.

Once those came apart, the failures stopped being mysterious. I had been treating "did we answer correctly" as a single property of the pipeline, so every failure looked like the same failure. They are two different failures with two different fixes, and no end-to-end number can tell them apart.

#What I would check first now

In this order, when retrieval quality is poor:

1. Recall of the first stage at the shortlist size. One number, and it decides which of the remaining steps are worth doing at all.
2. Chunking. A chunk spanning three topics produces a vector that represents none of them. Most of what gets blamed on the embedding model turns out to be this.
3. Whether lexical matching is missing. Dense retrieval is weak on exact identifiers, error codes, version numbers and rare proper nouns. Hybrid search rescues a class of failure no reranker can recover from, because those documents never reach the shortlist.
4. Then the reranker, and then the shortlist size feeding it, which becomes the main latency and quality dial once the stage exists.

#What I would remember six months from now

When a system has stages, the stages usually have different jobs, and a single end-to-end number will hide which one is failing. Instrument the boundary between them. That generalises well beyond retrieval, and it is the part of this I have reused most.

#References

• Sentence-BERT, which sets out the bi-encoder and cross-encoder distinction directly.
• Multi-Stage Document Ranking with BERT on the two-stage retrieve-then-rerank architecture.
• Your own evaluation harness, which is the only source here that knows anything about your corpus.