Does RAG train or change the LLM?

No. RAG does not change the model's weights. It only changes what goes into the prompt at question time, so you can add or remove documents without retraining anything.

What is a vector database used for in RAG?

A vector database stores the embedding of every chunk and finds the chunks whose embeddings are closest to the question's embedding. Small projects can keep vectors in memory; larger ones use a dedicated vector store.

How many chunks should a RAG system retrieve?

A common starting point is three to five chunks. Retrieve too few and the answer may be missing; retrieve too many and the model gets distracted by noise and the prompt gets expensive.

Can RAG show where an answer came from?

Yes. Because every retrieved chunk carries metadata such as file name and page, a RAG app can list its sources next to the answer. This is one of the main advantages of RAG over fine-tuning.

What is RAG? A practical guide to retrieval-augmented generation

What is retrieval-augmented generation (RAG)?

Retrieval-augmented generation (RAG) is a way to make a large language model (LLM) answer questions from your own data. At question time, the app searches your documents for the most relevant passages and pastes them into the prompt. The LLM then writes an answer based on those passages instead of guessing from its training data.

RAG solves two common problems. An LLM does not know your private documents, and its training data stops at a fixed date. RAG fixes both without retraining the model, because the knowledge lives in a search index you control.

A RAG system has two phases. The indexing phase runs ahead of time: load documents, split them into chunks, turn each chunk into an embedding, and store the embeddings. The query phase runs on every question: embed the question, find the closest chunks, build a prompt, and call the LLM.

How does chunking work?

Chunking is the step that splits long documents into smaller pieces, called chunks, before they are indexed. A chunk is usually a few hundred tokens long. Chunking matters because retrieval returns whole chunks, so the chunk is the unit of knowledge the model will see.

Chunk size is a trade-off. Small chunks are precise but can cut a sentence away from the context that explains it. Large chunks keep context but bring in unrelated text and use more of the prompt.

Most frameworks add chunk overlap, where the end of one chunk is repeated at the start of the next. Overlap reduces the chance that an important fact is split across two chunks and lost. Splitting on natural boundaries such as headings, paragraphs or table rows usually works better than splitting at a fixed character count.

What are embeddings and vector search?

An embedding is a list of numbers that represents the meaning of a piece of text. An embedding model turns similar texts into vectors that sit close together in space. "How do I reset my password?" and "I forgot my login" end up near each other even though they share few words.

Vector search is how RAG finds relevant chunks. The app embeds the user's question with the same embedding model used for the chunks. It then asks the vector store for the chunks whose vectors are most similar, usually by cosine similarity, and returns the top few.

Many production systems combine vector search with keyword search, an approach called hybrid search. Keyword search catches exact terms such as product codes, error messages and names, which pure vector search can miss. A reranker can then re-score the combined results so the best chunks land at the top.

How is the prompt assembled?

Prompt assembly is the step that turns retrieved chunks into instructions the LLM can follow. A typical RAG prompt has three parts: a system instruction, the retrieved context, and the user's question. The system instruction tells the model to answer only from the context and to say it does not know when the context is not enough.

Each chunk should be labeled with its source, such as the file name and page. Labels let the model cite sources and let the app show those sources to the user. They also make debugging much easier when an answer is wrong.

The order and amount of context matter. Put the most relevant chunks first, keep the total within a sensible budget, and remove duplicates. More context is not always better, because irrelevant text can pull the answer off course.

What does a minimal RAG app look like in Python?

LlamaIndex is an open-source Python framework for connecting LLMs to your data. The example below loads every file in a ./data folder, chunks and embeds them, builds an in-memory vector index, and answers a question. By default LlamaIndex uses OpenAI for embeddings and the LLM, so it expects an OPENAI_API_KEY environment variable.

pip install llama-index
export OPENAI_API_KEY="sk-..."

from llama_index.core import Settings, SimpleDirectoryReader, VectorStoreIndex

# Chunking settings used when the index is built
Settings.chunk_size = 512
Settings.chunk_overlap = 50

# Indexing phase: load, chunk, embed and store
documents = SimpleDirectoryReader("./data").load_data()
index = VectorStoreIndex.from_documents(documents)

# Query phase: retrieve the top 3 chunks and ask the LLM
query_engine = index.as_query_engine(similarity_top_k=3)
response = query_engine.query("What is our refund policy?")

print(response)
for source in response.source_nodes:
    print(source.score, source.node.metadata.get("file_name"))

The last loop prints the retrieved chunks with their similarity scores and file names. Checking these sources is the fastest way to tell whether a bad answer came from retrieval or from the model. In the no-code AI bot framework I built, LlamaIndex handled RAG over different data sources in the same way, with the LLM being configurable per bot.

What are the common RAG failure modes?

RAG failures usually fall into a few repeatable patterns. Knowing them makes debugging faster, because each one has a different fix.

Failure mode	What you see	Usual fix
Wrong chunks retrieved	Confident answer about a related but wrong topic	Better chunking, hybrid search, a reranker
Answer split across chunks	Partial or incomplete answers	Larger chunks, more overlap, split on headings
Missing document	"I don't know" for a question the docs should cover	Check the loader, file types and indexing logs
Model ignores context	Answer contradicts the retrieved text	Stricter system prompt, fewer and cleaner chunks
Hallucinated answer	Fluent answer with no support in the sources	Require citations and an explicit "I don't know" rule
Stale index	Old prices, policies or names	Re-index on document changes, store update dates
Poor PDF or table parsing	Garbled numbers and broken tables	A better parser, or convert tables to text first

The most important habit is to look at the retrieved chunks, not just the final answer. If the right text never reaches the prompt, no amount of prompt tuning will fix the answer.

How do you know if a RAG system is working?

A RAG system is working when it retrieves the right chunks and the model answers faithfully from them. These are two separate checks. Retrieval quality asks whether the correct source appears in the top results. Answer quality asks whether the answer is correct, grounded in the context and honest when the context is missing.

Build a small test set of real questions with known correct sources. Run it after every change to chunking, embeddings or prompts. The article on testing LLM apps covers how to set this up, and RAG vs fine-tuning explains when RAG is the right tool in the first place.

Summary

RAG gives an LLM access to your own documents by chunking them, embedding the chunks, retrieving the closest matches for each question and adding them to the prompt. It keeps answers current, supports citations and needs no model training. Most quality problems come from retrieval, so inspect the retrieved chunks first. If you want a RAG chatbot built on your data, see the AI applications service.