Retrieval Augmented Generation (RAG) solves a concrete problem: LLMs know nothing about your internal process documents, current product specifications, or proprietary knowledge bases. RAG connects the language understanding of modern AI models with structured access to your own data. This works without expensive fine-tuning and without sending sensitive documents to external services.
The result: employees can ask questions about internal documents in natural language and receive precise, source-backed answers.
What Separates RAG from Fine-Tuning
Fine-tuning teaches a model new knowledge through retraining. This sounds intuitive but has serious drawbacks: it’s expensive, time-consuming, and must be repeated every time documents are updated. Fine-tuned models also frequently suffer from catastrophic forgetting, losing previously learned knowledge when new information is introduced.
RAG works differently. The LLM itself remains unchanged. Instead, relevant context is retrieved from a knowledge store at runtime and added to the prompt. The model sees not just the question, but also the matching document excerpts.
| Criterion | Fine-Tuning | RAG |
|---|---|---|
| Update effort | High (retraining) | Low (add document) |
| Cost | High | Low |
| Freshness | Frozen at training time | Continuously updatable |
| Transparency | Black box | Source citations possible |
| Data privacy | Data enters training | Data stays on-premise |
For most enterprise applications (internal Q&A systems, support bots, document analysis), RAG is the more pragmatic choice.
Architecture: The Three Core Components
A RAG system consists of three phases: indexing, retrieval, and generation.
Indexing
Documents are split into chunks, converted into vectors by an embedding model, and stored in a vector store. This step runs once (or incrementally as new documents arrive).
Retrieval
When a query arrives, the question is also converted into a vector. The vector store finds semantically similar chunks via approximate nearest neighbor search. These chunks are passed as context to the LLM.
Generation
The LLM answers the question based on the retrieved context. Well-implemented RAG systems include the source chunks as citations for traceability and trust.
from anthropic import Anthropic
import chromadb
client = Anthropic()
vector_db = chromadb.Client()
collection = vector_db.get_collection("company_docs")
def rag_query(question: str, n_results: int = 5) -> dict:
# 1. Retrieval: fetch relevant chunks
results = collection.query(
query_texts=[question],
n_results=n_results,
)
context_chunks = results["documents"][0]
sources = results["metadatas"][0]
# 2. Build prompt with context
context = "\n\n---\n\n".join(context_chunks)
prompt = f"""Answer the following question based solely on the provided context.
If the answer is not contained in the context, say so explicitly.
Context:
{context}
Question: {question}"""
# 3. Generation
response = client.messages.create(
model="claude-opus-4-6",
max_tokens=1024,
messages=[{"role": "user", "content": prompt}],
)
return {
"answer": response.content[0].text,
"sources": sources,
}Chunking Strategies: The Underrated Detail
The quality of a RAG system stands or falls with its chunking strategy. Chunks that are too large introduce too much irrelevant context. Chunks that are too small break apart meaningful passages.
Fixed Window with Overlap
The simplest approach: split text into blocks of roughly 512 tokens, with 50–100 token overlap between adjacent chunks. The overlap prevents important sentence boundaries from being lost.
from langchain.text_splitter import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter(
chunk_size=512,
chunk_overlap=64,
separators=["\n\n", "\n", ".", " "],
)
chunks = splitter.split_text(document_text)Semantic Chunking
For structured documents (manuals, process documentation), semantic chunking is more effective. Documents are split at natural boundaries: headings, paragraphs, and lists. This keeps the meaning intact within each chunk.
Hierarchical Chunking (Parent-Child)
A more advanced pattern: small chunks for retrieval precision, but when a chunk matches, the surrounding larger context is passed to the LLM. This improves both retrieval precision and answer quality.
In practice, for flowing prose (policies, FAQs), 512 tokens with overlap works well. For structured documents, semantic chunking is clearly superior.
Choosing an Embedding Model and Vector Store
Embedding Models
For multilingual enterprise documents, several models perform well. The Microsoft multilingual-e5-large offers strong performance and runs locally. OpenAI’s text-embedding-3-large provides very high quality but requires API access. Nomic AI’s nomic-embed-text is open source, runs locally, and offers a good balance.
For sensitive company data, prefer locally running models. This ensures no data leaves your infrastructure.
Vector Stores
Several vector store options exist for different use cases. ChromaDB works well for prototypes and local development. Qdrant is ideal for production systems, self-hosted deployments, and GDPR compliance. PostgreSQL with pgvector makes sense if PostgreSQL is already in your stack. Pinecone is suitable for managed cloud deployments where fast setup is prioritized.
For enterprise deployments in Europe, Qdrant self-hosted is a proven choice: GDPR-compliant, scalable, with a solid Python SDK.
Common Pitfalls
Measuring Retrieval Quality
Many teams invest heavily in generation (prompt engineering) but neglect retrieval. When the wrong chunks are retrieved, even the best LLM can’t help. Retrieval metrics like MRR (Mean Reciprocal Rank) and Recall@K should be part of your evaluation pipeline.
Hallucination Despite Context
LLMs can draw incorrect conclusions even with provided context. Countermeasures:
- Instruct the system prompt to answer only from the provided context
- Tie answers to source citations (chunk ID, document name)
- Use low temperature values (0.0–0.3) for factual queries
Don’t Forget Chunk Metadata
Each chunk should carry metadata: source document, creation date, department, approval status. This enables targeted filtering (“only documents from Q1 2026”) and traceable citations.
Conclusion
RAG is no longer a hype topic. It’s a production-ready pattern for enterprise applications that connect AI with internal knowledge. The barrier to entry is low: with ChromaDB, a local embedding model, and an LLM API, a working prototype can be set up in a day.
The real work lies in the details: clean chunking, robust retrieval evaluation, and a well-thought-out update strategy for new documents. These are exactly the questions we work through with clients in projects.
For the next step: our article on AI Agents in the Enterprise shows how RAG systems fit into larger agent architectures. And for the infrastructure side of deploying AI workloads, the principles in our Cloud Migration guide apply equally well.
Talk to us about your RAG project →
Frequently Asked Questions
What is the main advantage of RAG over fine-tuning?
RAG keeps the LLM unchanged and retrieves relevant context at runtime without costly retraining. Documents update easily, costs stay low, and systems remain transparent with source citations. Fine-tuning requires expensive retraining, causes knowledge loss, and becomes outdated when documents change.
How large should chunks be in a RAG system?
Fixed-window chunking typically uses 512 tokens per chunk with 50 to 100 tokens of overlap between adjacent chunks. This size balances retrieval precision with context relevance. Semantic chunking at natural document boundaries works better for structured documents like manuals and policies.
Why is retrieval quality more important than generation quality?
When wrong chunks are retrieved, even the best LLM cannot help. Retrieval metrics like MRR and Recall@K should be part of your evaluation pipeline. Many teams invest heavily in prompt engineering but neglect retrieval, which is a missed opportunity for quality improvements.
Which embedding models work well for enterprise data?
Microsoft’s multilingual-e5-large runs locally and performs well on multilingual documents. OpenAI’s text-embedding-3-large provides very high quality but requires API access. Nomic AI’s nomic-embed-text balances quality, local execution, and open-source benefits.
How do you prevent hallucinations in RAG systems despite providing context?
Use system prompts instructing the model to answer only from provided context. Tie answers to source citations by chunk ID and document name. Use low temperature values between 0.0 and 0.3 for factual queries. Include chunk metadata for filtering and traceable citations.
