RAG Done Right: Sources, Chunking, Evaluation, and Hallucination Control (with Simple Examples)

March 13, 2026

Many teams are adopting Retrieval-Augmented Generation (RAG) to build internal assistants, intelligent search tools, or systems that answer questions based on documentation.

The idea is straightforward: instead of relying only on the general knowledge of a language model, the system retrieves relevant information from a knowledge base and uses it as context to generate an answer.

In theory, it sounds simple. In practice, building a RAG system that works reliably requires making good decisions in several areas: the data sources, how the content is chunked, how the system is evaluated, and how hallucinations are controlled.

In this article, we review some key principles for implementing RAG in a robust way, with simple examples.

‍

1. Data sources matter more than you might think

A RAG system is only as good as the sources it uses.

One common mistake is to start indexing every document available without curating the content first. This often leads to inconsistent or contradictory answers.

Before thinking about embeddings or models, it’s important to define:

which documents are trusted sources
which ones are up to date
which content is relevant for the use case

For example, imagine an assistant designed to answer questions about internal company policies.

A poor approach would be to index:

old email threads
duplicate documents
multiple versions of the same policy

A better approach would be to:

index only the official and current version of each document
organize sources by category
maintain a clear update process for the knowledge base

A well-designed RAG system starts with a curated knowledge base.

‍

2. Chunking determines whether the system finds the right information

Once the sources are defined, the next step is splitting the content into smaller parts that can be indexed in a semantic search system.

This process is known as chunking.

The goal is for each fragment to contain enough context to be useful, but not so much that it mixes different topics.

Simple example.

Original document:

"To request vacation time, the employee must submit a request in the internal system at least 10 days in advance. The supervisor must approve it before it is confirmed."

If the chunk is too large, it may mix multiple processes.

If it is too small, it may lose context.

A good chunk might look like this:

Indexed chunk

"To request vacation time, the employee must submit a request in the internal system at least 10 days in advance."

This allows the system to quickly retrieve the relevant information when the user asks:

"How many days in advance do I need to request vacation?"

Well-designed chunking significantly improves retrieval quality.

‍

3. Evaluating the system is as important as building it

One of the biggest problems in AI projects is that evaluation is often informal.

Teams test the system with a few questions, it seems to work, and the project is considered finished.

But RAG systems require systematic evaluation.

Some useful practices include:

defining a set of test questions
verifying whether the system retrieves the correct documents
evaluating the quality of the generated answers

Example:

Test question:

"How many days in advance should I request vacation?"

Possible evaluation:

Did the system retrieve the correct chunk?
Is the answer faithful to the document?
Did it invent information?

Building small evaluation datasets helps improve the system over time.

‍

4. Controlling hallucinations

One of the risks of any system based on LLMs is that the model may generate information that is not present in the sources.

In RAG systems, this risk can be mitigated with several strategies.

One of the simplest is forcing the model to answer only using the retrieved context.

For example, a system prompt might state:

"Answer only using the information provided in the context. If there is not enough information, say that you cannot answer."

Another useful practice is showing the sources used in the response.

Example:

Question

"How many days in advance do I need to request vacation?"

Answer

"You must request vacation at least 10 days in advance."

Source

Vacation policy – section 2.

This improves transparency and increases trust in the system.

‍

5. RAG is engineering, not just prompts

A robust RAG system combines several components:

well-curated knowledge sources
effective chunking strategies
semantic retrieval mechanisms
systematic evaluation
hallucination control

For this reason, building good RAG systems is not just about prompts or models. It is fundamentally a systems engineering problem focused on knowledge retrieval and generation.