Almost every company we talk to has the same idea: put an assistant on top of our documents so people stop searching and start asking. Contracts, product manuals, internal policies, support tickets, a decade of Confluence. The demo takes an afternoon. The version people trust takes a few careful weeks, and the difference between the two is the subject of this article.
Retrieval-augmented generation (RAG) is simple to describe. When someone asks a question, the system finds the passages most likely to contain the answer and hands them to a language model together with the question. The model writes an answer grounded in those passages and points to its sources. The model does not need to have been trained on your data, and your data does not leave your control any more than you allow.
What follows is what we pay attention to when a RAG system has to work on Monday morning, not just in a meeting.
Start with the questions, not the documents
The most common mistake is to begin by indexing everything. A better first step is to collect fifty to a hundred real questions from the people who will use the assistant, together with the answers an expert would give and the documents those answers come from.
That list does three jobs at once:
- it tells you which sources actually matter, which is usually a fraction of what exists;
- it reveals the shape of the questions (lookups, comparisons, “what is our policy if…”, calculations), which decides the architecture;
- it becomes the test set you will use for every change you make later.
If nobody can write down what a good answer looks like, no amount of engineering will produce one. This is the cheapest week of the project and the one that saves the most money.
Document preparation is most of the work
Language models are good at reading. They are not good at reading a PDF that was scanned sideways, a table that was flattened into a single line, or a page whose header repeats on every chunk. Retrieval quality is decided long before the model sees anything.
Parse structure, not just text
Headings, lists, tables and section numbers carry meaning. A clause that says “subject to section 4.2” is useless if section 4.2 ended up in a different chunk with no label. We convert documents into a structured intermediate format first, keep the heading path for every passage (“Employee handbook › Leave › Parental leave”), and treat tables as tables. For scanned material that means OCR with layout detection, and for the hardest documents a vision-capable model that describes figures and diagrams in words.
Chunk along meaning
Fixed-size chunks of a few hundred tokens are a reasonable default and a poor final answer. Splitting along sections, keeping a little overlap and attaching the section path and document metadata to each chunk makes a measurable difference. So does adding a one-sentence summary of the surrounding document to every chunk, so that a passage about “the notice period” still knows which contract it belongs to.
Keep it fresh
Documents change. A RAG system needs an ingestion pipeline that notices new and updated files, re-processes only what changed, removes what was deleted and records which version of a document an answer came from. Without that, the assistant slowly becomes confidently out of date, which is worse than having no assistant at all.
Retrieval: hybrid beats clever
Vector search finds passages that mean something similar to the question. It is excellent for paraphrases and weak for exact things: product codes, names, article numbers, error messages. Keyword search (BM25) is the opposite. In practice the robust setup is both, combined, followed by a re-ranking step that looks at the question and each candidate passage together and orders them by how well they answer it.
A few further techniques earn their place in most production systems:
- Metadata filters for things the user already knows: country, product line, document type, date range. They are cheaper and more reliable than hoping similarity will sort it out.
- Query rewriting, where the model turns a conversational follow-up (“and for contractors?”) into a standalone search query.
- Multi-step retrieval for questions that need several sources, where the assistant searches, reads, and searches again. This is where a RAG system starts to look like an agent, and it should be used deliberately because each step adds latency and cost.
You do not need an exotic vector database to start. PostgreSQL with pgvector handles a very large share of real-world corpora, keeps your data in a system you already operate and makes permission filters a plain SQL question. Dedicated engines make sense at larger scale or with specific performance needs.
Permissions are not optional
If a person cannot open a document, the assistant must not quote it to them. This sounds obvious and is the requirement most often discovered late.
The reliable pattern is to carry access rights into the index: every chunk knows which groups or users may see its source, the retrieval query filters by the identity of the person asking, and the model never receives a passage the user is not entitled to. Filtering after generation is not enough, because the model has already read the text. For systems connected to SharePoint, Google Drive or a document management system, we sync permissions together with content and treat a permission change like a content change.
Answers people can check
Trust comes from being able to verify. Every answer should show its sources, ideally linking to the exact section, and the model should be instructed to say “I could not find this in the documents” when retrieval returns nothing relevant. A well-behaved assistant that declines is far more useful than one that improvises.
We also design the answer format around the task. A support agent wants a short answer and the relevant steps. A lawyer wants the clause quoted verbatim. A sales team wants a comparison table. The same retrieval layer can serve all three with different instructions.
Evaluation turns opinions into numbers
Without evaluation every change is a matter of taste. With it, you can tell whether a new chunking strategy, a different model or a prompt change made things better or worse.
We measure two things separately:
| Layer | Question | How we check |
|---|---|---|
| Retrieval | Did the right passages reach the model? | Recall of expected sources on the test set |
| Generation | Is the answer correct, complete and faithful to the sources? | Expert review on a sample, plus model-graded checks against reference answers |
Running this suite on every change, the way you would run unit tests, is what keeps quality from drifting. Once the system is live, real questions with thumbs up or down and the occasional expert review feed new cases back into the set.
Cost, latency and the choice of model
Most of the cost of a RAG system sits in two places: the tokens you send to the model (retrieved context is the bulk of it) and the engineering time to maintain the pipeline. Sending fewer, better passages lowers cost and usually improves answers too. Prompt caching helps when the same long instructions or documents are reused. Routing simple questions to a smaller, faster model and reserving the strongest model for hard ones keeps both bills and response times in check.
We often use Anthropic’s Claude for this kind of work because it reads long, dense documents well and follows instructions about citing and declining carefully. We combine it with other providers or open models where cost, latency or data residency call for it. Being able to switch models without rebuilding the system is a design goal in itself, because the best choice this year may not be the best choice next year.
Data protection in the EU
For European companies the questions are predictable and answerable: where is data processed, is it used for training, how long are prompts and outputs kept, and who can see the logs. Major model providers offer EU data processing options and contractual commitments not to train on business data. Personal data in source documents should be minimised or masked where it is not needed for the answer. A short record of the data flows and the purpose of the system makes the conversation with your data protection officer, and any assessment under the GDPR or the EU AI Act, much easier.
A sensible first version
A first production release that we would be comfortable putting in front of real users usually includes:
- one or two well-chosen sources, not the whole intranet;
- structured ingestion with automatic updates;
- hybrid search with re-ranking and permission filters;
- answers with citations and an honest “not found”;
- an evaluation set of real questions run on every change;
- a feedback button and a dashboard of what people ask.
Everything else, such as more sources, agents that take actions, or integration into Slack, Teams or your CRM, builds on that foundation. If you are weighing an assistant for your own documents, our AI and LLM team can start with a short discovery and a working prototype on your real data, and our data engineering practice takes care of the pipelines underneath. Or simply tell us what you have in mind.