The final part of our series about modern Retrieval-Augmented Generation systems.
So far in this series we have been building a stronger retrieval pipeline step by step.
We started with keyword search and BM25. Then we looked at vector search. Then we combined both with hybrid search and RRF. Then in the previous post I explained reranking and the difference between bi-encoders and cross-encoders. The key point was that retrieval should first find good candidates, and reranking should then select the strongest pieces of evidence.
That gives us a solid retrieval stack, but even a strong retrieval pipeline can struggle when the user asks a complex question with multiple information needs.
Real user questions are messy. They often contain multiple information needs in a single sentence.
For example:
That is not one retrieval task. It is several. The system must investigate the alarm. It must check the replacement context. It must search for calibration instructions. It must also check part compatibility for a specific machine variant. If we send this entire question as a single search query, we may get incomplete results.
A better approach is query planning and decomposition.
The system breaks down the original question into smaller sub-queries.
For example:
Each sub-query is focused.
Each sub-query can be sent to the source where the answer is most likely to be found.
That is already significantly better than searching everywhere for everything.
A service ticket system is useful for known incidents. A maintenance manual is useful for procedures. A parts catalog is useful for part numbers and compatibility. A variant table is useful for machine configurations.
Good source selection improves both quality and latency. This is where retrieval starts to become more agentic.
Agentic retrieval means the system does not just search once.
It plans. It searches. It reviews the results. It decides whether something is missing. Then it may search again.
A classic RAG pipeline looks like: query, retrieval, and generation phase. An agentic retrieval flow looks more like this:
That matters for real-world questions.
The first retrieval pass might find the F217 alarm description and the calibration instruction, but not the affected variants. Or it finds the part number, but not whether it is still used in the AX-360.
An agentic retriever can reflect on the retrieved evidence and ask:
If something is missing, it can perform a follow-up search.
For example:
The final step is evidence consolidation.
When multiple sub-queries search across multiple sources, the system should not simply dump all passages into the LLM prompt.
It should structure the evidence.
For example:
This kind of structured evidence is significantly more useful than a random list of passages.
The LLM can now generate a response that is complete, grounded, and easier to verify.
RAG should not be reduced to vector search. That is the core point of this entire series. Vector search is useful, but it is only one part of the retrieval stack.
Real users ask messy questions. They mix symptoms, assumptions, part numbers, variants, and follow-up questions in a single sentence.
The information needed to answer them can be distributed across manuals, tickets, catalogs, configuration tables, and internal documentation.
That is why strong RAG systems are built as a stack:
In my view, this is the direction practical RAG is heading. Not simply larger models. Not simply larger context windows.
But better retrieval systems that know what to search for, where to search, when to search again, and how to deliver the right evidence to the model.
Richard is a member of the SABO core team, where he combines deep technical expertise with a focus on clean design and best practices. As a Senior Machine Learning Engineer and Software Architect, he works on intelligent systems such as conversational assistants and AI-driven industrial solutions. His work bridges modern software architecture with real-world machine learning applications, often involving MLOps and scalable infrastructure. In his free time, he enjoys learning new languages and exploring how they shape the way we think and communicate.
