But there is an important limitation.
On their own, they have no access to your private documents, internal knowledge bases, product catalogs, machine manuals, service tickets, project specifications, or Confluence pages.
That is why many real-world AI systems use Retrieval-Augmented Generation, commonly referred to as RAG.
The basic idea is simple:
Before the LLM responds, the system first searches for relevant information. This information is then passed to the model as context. The model responds based on this retrieved context.
That sounds simple, but in practice the difficulty usually does not lie in generation. The challenge lies in retrieval.
Finding the right information is much harder than it looks.
A simple RAG system might work well for a short search query like:
servo alarm F217But real users rarely ask perfect search queries. They ask things like:
The machine stops with servo alarm F217 after the gripper module was replaced.
What is causing it, and which machine variants are affected?This is not a simple search. It contains multiple information needs.
The system must understand what alarm F217 means. It must connect the alarm to the replacement of the gripper module. It must search for possible root causes. It may also need to determine which machine variants use the same servo or gripper configuration. And finally, it must deliver a well-founded answer with evidence.
A simple vector search may only return the most similar-looking text passages. It might find a document about alarm F217, but miss the information about the machine variant. Or it finds a service ticket, but not the official maintenance manual.
That is why I believe RAG should not be reduced to “vector database plus LLM”.
That is too simplistic.
A reliable RAG system needs a retrieval stack.
It must search different types of sources: manuals, tickets, service reports, PDFs, meeting notes, Confluence pages, product catalogs, and structured databases.
And this must happen in a way that delivers the right evidence to the LLM – not just any similar-looking text passages.
The goal is not to produce a fluent response.
The goal is to produce a grounded response.
This means: the model should not fabricate missing facts. It should respond by using the retrieved information as evidence. That is the real challenge in enterprise RAG.
The LLM is only the last step. The quality of the response depends heavily on what the system retrieves before the LLM starts writing.
In this series, I want to go through step by step the retrieval techniques that make modern RAG systems more reliable.
In this first part, we started with the basic idea: RAG is not primarily about text generation. It is about finding the right information before the model responds.
In the next post, I will start with the oldest retrieval technique, keyword search, and explain why BM25 is still very relevant in modern RAG systems.
