The short version
LLMs know a lot, but they don't know everything. They don't know about your company's internal docs, yesterday's news, or the specific PDF you uploaded. RAG fixes this by adding a retrieval step: before the model generates a response, it searches a knowledge base for relevant information and includes it in the prompt.
The result is an AI that can answer questions about your data, not just general knowledge.
How it works
RAG has two stages:
1. Retrieval: the system takes the user's question and searches for relevant content in a knowledge base. This might be a database of documents, a collection of web pages, or a set of files. The search usually works through embeddings, numerical representations of text that capture meaning, so "how do I deploy?" matches a document about deployment even if it doesn't use that exact phrase.
2. Generation: the retrieved content is inserted into the prompt alongside the user's question. The LLM reads both and generates a response grounded in the retrieved information. The prompt typically includes an instruction like "answer based on the following context" to keep the model focused on the provided material.
A simple RAG pipeline:
User question
→ Search knowledge base (embeddings + similarity)
→ Retrieve top 3-5 relevant chunks
→ Insert into prompt: "Context: [chunks]. Question: [user question]"
→ LLM generates answer grounded in context
The quality of RAG depends heavily on the retrieval step. If the wrong documents are retrieved, the model generates a confident answer based on irrelevant information. Good chunking (how you split documents), good embeddings (how you represent them), and good ranking (which results make it into the prompt) all matter.
Why it matters
RAG is how most "chat with your docs" tools work. It's how AI assistants answer questions about company knowledge bases, product documentation, and private data. If you've ever used a tool that seems to know about your specific documents, it's almost certainly using RAG. Understanding the pattern helps you evaluate these tools and spot their limitations.