What Is RAG (Retrieval-Augmented Generation)?
RAG connects large language models to outside documents so they can answer with current, private, or cited information instead of guessing from memory alone.
RAG, short for retrieval-augmented generation, is a technique that connects a large language model to an outside source of information — a document store, a database, a wiki — so the model can look things up before it answers instead of relying only on what it memorized during training. A retriever finds the most relevant passages for a question, those passages are inserted into the model's prompt as context, and the model then generates its answer grounded in that retrieved text. It's the difference between asking someone to answer from memory alone and letting them open the right book to the right page first.
What Is RAG, Exactly?
Retrieval-augmented generation pairs two systems that used to be separate: an information-retrieval engine (the kind that powers search) and a generative language model. According to Wikipedia's overview of the technique, RAG lets large language models "retrieve and use information from external data sources" rather than answering purely from the parameters they learned during pretraining, and it was first proposed in a 2020 research paper by Patrick Lewis and collaborators that combined a pretrained language model with an external memory searched at inference time, as Wikipedia's entry on retrieval-augmented generation describes.
AWS frames the problem RAG solves in plain operational terms: foundation models are trained on broad public data, but organizations constantly need answers grounded in their own proprietary documents — internal wikis, SharePoint pages, Confluence spaces, PDFs that were never part of any training run. As AWS's prescriptive guidance on RAG options and architectures puts it, "with RAG, the foundation model references an authoritative data source that is outside of its training data sources... before generating a response." Nothing about the model's underlying weights changes; what changes is what gets placed in front of it at the moment it's asked a question.
Why LLMs Need RAG: Knowledge Cutoffs, Hallucinations and Private Data
Three practical problems push teams toward RAG instead of just shipping a raw model.
Knowledge cutoffs. Every model is trained on a dataset that stops at a certain date. Ask it about something that happened afterward and it either says so or, worse, guesses. IBM's explainer notes that a model's training data is finite and reflects only what its developers could access at training time, so without a way to refresh that knowledge, the model's answers quietly go stale, according to IBM's topic page on retrieval-augmented generation. Retraining or fully fine-tuning a model every time the world changes is slow and expensive; retrieval lets you swap in fresh documents on the fly instead.
Hallucination. When a model doesn't know something, it can still produce a fluent, confident-sounding sentence that happens to be wrong. NVIDIA's engineering blog uses a courtroom analogy for this: the LLM is like a judge who can reach a fair verdict on general knowledge, but for case-specific facts, a clerk — the retriever — needs to go find the relevant precedents first. Grounding the model's output in retrieved passages, and letting it cite them, gives users something to check, which NVIDIA describes as a way to reduce (though not eliminate) hallucinated answers, per NVIDIA's blog post on what retrieval-augmented generation is.
Private and proprietary data. No general-purpose model was trained on your company's internal support tickets, contracts, or engineering runbooks — and for good reason; that data shouldn't be anywhere near a public training set. RAG is how a model answers questions about that private information without ever being retrained on it: the documents stay in your own index, and only the small number of relevant snippets get pulled into a given prompt. This is also where RAG overlaps with what practitioners call context engineering — the broader discipline of deciding exactly what information a model sees in its context window, of which retrieval is one major source.
How the RAG Pipeline Actually Works
Strip away the branding and a RAG system is really two pipelines: one that happens ahead of time to prepare a knowledge base, and one that happens live, in milliseconds, every time a user asks a question.
Chunking and embeddings
Before anything can be searched, source documents have to be broken into pieces small enough to be individually useful and large enough to still make sense on their own — a process called chunking. IBM's guidance is specific about why size matters: chunks that are too large become vague and dilute the signal a retriever is trying to match, while chunks that are too small lose the surrounding context that gave them meaning. Each chunk is then run through an embedding model, which converts its text into a numerical vector — a long list of numbers positioned in a high-dimensional space such that chunks with similar meaning end up near each other, even if they don't share any of the same words. Those vectors are stored in a vector database purpose-built for similarity search at scale.
Vector search and retrieval
When a user asks a question, that same embedding model converts the query into a vector, and the system searches the vector database for the stored chunks whose vectors are closest to it — typically using cosine similarity or dot-product distance, often sped up with approximate-nearest-neighbor indexing so the search stays fast even across millions of chunks. This is semantic search: it matches on meaning rather than exact keyword overlap, which is why a question phrased as "time off for remote staff" can still retrieve a policy document that only ever says "telecommute" or "PTO." Microsoft's documentation for Azure AI Search describes exactly this gap — traditional keyword search fails when a query's wording doesn't match the document's wording, and a RAG retriever has to understand intent, not just match tokens — per Microsoft Learn's RAG and generative AI overview for Azure AI Search. Many production systems run hybrid search, combining this vector similarity pass with a traditional keyword search in parallel, so that both semantic matches and exact terminology (product codes, legal citations, error strings) get a fair chance to surface.
Re-ranking
The first retrieval pass usually casts a wider net than the model needs — say, the fifty most similar chunks — because similarity search alone is an imperfect proxy for relevance. A separate re-ranking step, often a smaller and more precise model, re-scores that shortlist against the actual question and keeps only the handful that matter most. Azure AI Search's documentation describes semantic ranking as re-scoring results "based on meaning, not just keywords," and recommends pairing it with configurable result limits so the system returns highly relevant, concise results instead of an exhaustive document dump. Re-ranking matters because of the next constraint: the context window.
Context injection and generation
Whatever survives retrieval and re-ranking gets inserted into the prompt that's actually sent to the LLM, usually with instructions to answer using only the supplied material and to cite it where possible. The model then generates its response the normal way, token by token, except now its "knowledge" for this one answer includes whatever was just handed to it. AWS's guidance on Amazon Bedrock splits this cleanly into two options: a fully managed retrieve_and_generate call that handles both the search and the model call in one step, or a standalone retrieve call that returns only the matching documents so a team can post-process them — filter, deduplicate, re-rank further — before writing their own prompt and calling the model, per AWS's prescriptive guidance for RAG architectures. Either way, nothing about this process touches the model's weights — it's entirely a matter of what gets assembled into that one request.
RAG vs. Fine-Tuning vs. Long Context
RAG is frequently confused with two other ways of getting a model to use information it wasn't originally trained on: fine-tuning and simply relying on a longer context window. They solve different problems and are often combined rather than chosen between.
Fine-tuning means further training a model's weights on a domain-specific dataset so it internalizes a style, a format, or specialized vocabulary. IBM's framing is direct: "fine-tuning trains an LLM on domain-specific data" to build familiarity with a domain and its output requirements, while RAG instead supplies relevant information for each individual response, and the two are commonly used together — a fine-tuned model that also retrieves, per IBM's comparison of the two approaches. Long-context models, meanwhile, simply expand the window a model can read at once, so in principle you could paste an entire knowledge base into the prompt rather than retrieving a subset of it — but that gets expensive and slow fast, and research on how models actually use very long contexts shows relevant details can still get lost in the middle of a huge prompt even when they're technically "in there."
| Approach | What it changes | Best for | Trade-off |
|---|---|---|---|
| RAG | What the model sees in its prompt, per query | Fast-changing or proprietary data, citing sources, swapping knowledge bases instantly | Answer quality depends entirely on retrieval quality |
| Fine-tuning | The model's own weights | Teaching a consistent tone, format, or specialized task behavior | Slow and costly to update; doesn't add fresh facts well |
| Long context only | Nothing — just stuffs more text into one prompt | Small, static knowledge bases that fit comfortably in a window | Expensive at scale, and relevant details can get lost in a huge prompt |
Where RAG Is Used Today
RAG shows up anywhere a language model needs to answer questions about information that's too large, too private, or too fresh to bake into the model itself.
Enterprise search and internal knowledge bases. This is the canonical use case AWS and Microsoft both build dedicated services around: letting employees ask natural-language questions over SharePoint sites, internal wikis, HR policies, and contract repositories, with answers that cite exactly which document they came from. Microsoft's Azure AI Search documentation notes that enterprise content typically spans many different systems — SharePoint, databases, blob storage — and that a unified, permission-aware retrieval layer is what makes RAG workable across all of them at once, according to Microsoft's documentation. Increasingly, these systems are exposed to models as tools through standardized interfaces like the Model Context Protocol, which lets an assistant call out to a retrieval service the same way it would call any other tool.
Coding assistants. Modern AI coding tools retrieve relevant snippets from a codebase — the function being edited, its callers, related tests, internal style guides — and inject them into the model's context before it writes or reviews code, so suggestions actually match the project's existing patterns instead of generic textbook code. This retrieval-plus-generation loop is also the backbone of many AI agents that need to look things up mid-task rather than relying on a single static prompt.
Customer support. Support chatbots grounded in product manuals, troubleshooting guides, and prior support tickets can answer specific product questions accurately and point to the exact policy or manual page involved, rather than guessing from general training data. IBM lists "specialized chatbots and virtual assistants" for customer support among the clearest production use cases for RAG, alongside research assistance, content generation with verifiable citations, and internal onboarding tools.
Limitations and Challenges of RAG
RAG is not a magic fix for accuracy, and the vendors building these systems are candid about where it still struggles. Microsoft's Azure AI Search documentation lists five concrete challenges any RAG implementation has to solve: understanding vague or conversational queries rather than just matching keywords, unifying content that lives across many different source systems, staying within the token budget of the underlying model instead of dumping entire documents into the prompt, meeting users' expectations for answers in seconds rather than minutes, and enforcing security so that a retrieval system never surfaces content a given user isn't authorized to see. IBM is equally direct that RAG reduces hallucination without eliminating it, since the model can still misread or misuse retrieved context even when the right passage was successfully found. In practice, the quality of a RAG system is bounded by the quality of its retriever: if the wrong chunk gets retrieved, a perfectly capable language model will still generate a confident, well-written, wrong answer from it.
What to Read Next
RAG is the plumbing underneath a huge share of today's production AI systems, but it's one piece of a larger stack. The retrieved passages have to come from somewhere — typically a vector database built for similarity search — and increasingly they're surfaced to models through standardized tool-calling layers rather than hand-rolled prompts. Readers who want to go deeper on how models decide what information to act on at all should look at how context engineering differs from simple prompt writing, and how autonomous AI agents chain retrieval, tool use, and reasoning together into multi-step workflows rather than single question-and-answer turns.
Frequently asked questions
What does RAG stand for in AI?
RAG stands for retrieval-augmented generation. It is a technique that lets a large language model search an external knowledge source, such as a document index or vector database, and use what it finds to help write its answer, instead of answering only from what it learned during training.
How is RAG different from fine-tuning?
Fine-tuning changes a model's internal weights by training it further on domain-specific data, which is useful for teaching style, format, or specialized behavior. RAG leaves the model's weights untouched and instead supplies relevant outside information at the moment of each query, which makes it far faster and cheaper to update when facts change. The two approaches are often combined.
Does RAG stop AI models from hallucinating?
RAG reduces hallucination by grounding answers in retrieved text and allowing citations, but it does not eliminate it. A model can still misinterpret or misuse the retrieved passages, and if the retrieval step pulls the wrong document, the model can still generate a confident but incorrect answer from it.
What role does a vector database play in RAG?
A vector database stores numerical representations (embeddings) of document chunks so that a retrieval system can quickly find the chunks most similar in meaning to a user's query, using similarity search rather than exact keyword matching.
Is RAG still needed if a model has a very long context window?
Often, yes. While a long-context model can technically accept huge amounts of text in a single prompt, feeding it an entire knowledge base is expensive, slow, and can bury the relevant details inside a mass of irrelevant text. RAG retrieves and injects only the small number of passages that are actually relevant to a given question.
Where is RAG used in practice?
Common production uses include enterprise search over internal documents and wikis, AI coding assistants that pull in relevant code and documentation, and customer support chatbots grounded in product manuals and past support tickets.
Sources
- AWS Prescriptive Guidance: Retrieval Augmented Generation options and architectures on AWSdocs.aws.amazon.com
- Microsoft Learn: RAG and Generative AI - Azure AI Searchlearn.microsoft.com
- IBM: What Is Retrieval Augmented Generation (RAG)?ibm.com
- NVIDIA Blog: What Is Retrieval-Augmented Generation, aka RAG?blogs.nvidia.com
- Wikipedia: Retrieval-augmented generationen.wikipedia.org
Felix Moreau writes Pandromeda's software coverage and how-to guides. He covers Windows, macOS and Linux updates, the apps people rely on, emulators and developer tools, and turns official documentation into clear, numbered steps that work on the current version.
