What RAG is and when to use it

A language model learns general knowledge during training, but it does not know your company's handbook, last week's product release or a specific customer contract. There are three main ways to give it that knowledge:

1. Put the documents in the prompt. Models now accept long inputs, so for a few documents this is often the simplest option.
2. Fine tune the model on your material. This teaches style and formats well, but is a poor way to add facts that change, and it does not give citations.
3. Retrieve relevant passages at question time and include only those in the prompt. This is retrieval augmented generation.

Why retrieval

RAG works well when:

• The knowledge base is too large to fit in a prompt, or too expensive to send every time.
• Content changes often, so you need updates without retraining.
• Users need citations to check answers.
• Different users may see different documents, so access control matters.

The basic pipeline

Indexing, done ahead of time:

1. Load documents from their sources.
2. Clean and split them into chunks.
3. Compute an embedding vector for each chunk.
4. Store chunks, vectors and metadata in an index.

Answering, done per question:

1. Turn the question into a search query and embedding.
2. Retrieve the most relevant chunks.
3. Optionally rerank them.
4. Ask the model to answer using only those chunks, with citations.

Where RAG fails

Most poor RAG results come from retrieval, not the model:

• The answer is in a table that the PDF parser turned into gibberish.
• The relevant sentence was split across two chunks.
• Twenty near duplicate pages crowd out the one that matters.
• The question uses different words from the document, and keyword search misses it.
• The document is outdated, but nothing marks it as such.

When retrieval delivers the right passage, modern models usually answer well. When it does not, no prompt can save the answer. That is why this course spends most of its time on retrieval and on measuring it.

When not to use RAG

If users need precise numbers from a database, query the database, perhaps with a model generating safe SQL or calling an API. If the corpus is small and stable, try long context first. If the task is classification, a classifier may be better. RAG is a tool for open questions over text collections.