In the previous post I explained hybrid search and Reciprocal Rank Fusion. The idea was to combine keyword search and vector search and merge their ranking results in a practical way.
That gives us better candidates. But candidates are not the same as final evidence. In other words: the retrieval system may have found documents that are related to the query, but not necessarily the documents that best answer the query.
This is where reranking comes in.
A reranker takes the top candidate documents from the retrieval phase and scores them again with a more precise model.
The first retrieval phase is optimized for recall. It is supposed to find everything that could be relevant.
The reranker is optimized for precision. It is supposed to decide which of these candidates are actually the most useful pieces of evidence.
This is important because the first retrieval phase typically has to search through a large document collection. It must be fast. That is why vector search uses pre-computed document embeddings, and keyword search uses an index.
A reranker is slower and more computationally expensive, so we do not run it over millions of documents. Instead, we only run it over the top candidates.
For example: we retrieve the top 100 documents, run reranking, keep the best 5 or 10, and send those to the LLM. This offers a good balance between speed and quality.
To understand why reranking helps, it is useful to compare bi-encoders and cross-encoders. A bi-encoder processes the query and the document separately.
The query is converted into a vector. The document is converted into another vector. The system then compares the vectors using cosine similarity or another similarity function. That is fast and scalable.
Document embeddings can be computed once and stored in a vector database. When a user asks a question, the system only computes the query embedding and compares it to the stored document embeddings.
That is why bi-encoders are useful for first-stage retrieval. But they have a weakness. The query and the document do not interact within the model. They are compressed separately and only then compared.
A cross-encoder works differently.
It processes the query and the document together.
This means the model can directly compare words, numbers, entities, negations, and relationships. That makes it slower, but generally more precise.
For example, consider this query:
machine stops with servo alarm F217 after gripper replacement And these two documents:
Document A:
After replacing the gripper module, recalibrate the servo end positions.
Document B:
Servo alarm F271 can occur after replacing the drive module. A bi-encoder might consider both documents similar. Both mention servo alarms, replacement, and machine behavior. But a cross-encoder can look more carefully.
It can recognize that Document A fits the repair situation. It can also recognize that Document B mentions a different alarm code and a different module.
So the reranker might score them like this:
Document A → 0.94
Document B → 0.31 That is why cross-encoders are often used as rerankers. They are too computationally expensive to search the entire document collection, but very useful for re-scoring the top candidates.
At this point we have a strong retrieval pipeline:
keyword search
+ vector search
+ RRF
+ reranking That is already significantly better than a simple top-K vector search.
In summary: Reranking makes retrieval more precise. The first retrieval step finds candidates, and the reranker decides which candidates are actually useful.
At this point the retrieval pipeline is already quite strong. But some user questions are not one question. They are several questions packed into one sentence. In the next post, I will explain query planning, decomposition, and agentic retrieval.
