RAG(Retrieval-Augmented Generation) end-to-end

Pull the relevant passages from your knowledge base into the prompt, so the LLM is reading the answer instead of trying to recall it. Three steps, in order: Retrieve → Augment → Generate.

What RAG actually does

A vanilla LLM answers from its compressed memory of training data. It has no access to your private documents and no awareness of anything written after its training cutoff. RAG fixes both by bolting a retrieval step in front of the model: before the model sees your question, the system finds the most relevant passages from your documents and pastes them into the prompt as context.

That one change does three big things:

• Hallucinations drop sharply — the model has the actual passage to read, not a fuzzy memory to compress.
• Citations become possible — the answer can point back to the exact document it came from.
• Your private knowledge becomes accessible — wikis, PDFs, ticket histories, transcripts, code. If you can index it, the model can read it.

That's the whole concept. Everything below this point is the engineering of how to do that retrieval step well — which sounds simple but is where most production RAG systems either thrive or quietly fail.

When you'd reach for it

Build RAG any time you want an LLM to answer questions grounded in your data:

• A support bot over your help docs
• A "chat with your codebase" tool
• A research assistant over scientific papers or legal cases
• A company-internal Q&A over wikis and runbooks

You wouldn't reach for RAG when general knowledge is all that's needed — a creative writing assistant, a brainstorming partner. Adding retrieval there just slows things down with no quality gain.

How it's actually built

The whole pipeline has two halves — offline (build the index) and at query time (read it).

Offline (run when knowledge changes):

At query time:

[Chunk #1: "Refunds are processed within 5 business days…"]
[Chunk #2: "To cancel a subscription, go to Settings → Billing…"]
[Chunk #3: "Our support hours are 9am–6pm UTC…"]

Where it bites in real life

Common failure modes

Under the hood (optional)

The full RAG pipeline as one function: embed the query, retrieve top candidates, rerank to the best 5, build a grounded prompt, generate. About 30 lines of Python — and the comments map directly to the steps in the analogy above.

def answer(question: str, tenant_id: str) -> dict:
    # 1. Embed the query
    q_emb = voyage.embed([question], model="voyage-3").embeddings[0]
 
    # 2. Hybrid retrieve
    candidates = db.execute("""
        SELECT id, text, source FROM chunks
        WHERE tenant_id = %s
        ORDER BY embedding <=> %s::vector
        LIMIT 50
    """, (tenant_id, q_emb)).fetchall()
 
    # 3. Rerank top-5
    ranked = cohere.rerank(query=question,
                           documents=[c["text"] for c in candidates],
                           top_n=5)
    top_chunks = [candidates[r.index] for r in ranked.results]
 
    # 4. Build a grounded prompt
    context = "\n\n".join(f"[{c['id']}] {c['text']}" for c in top_chunks)
    prompt = f"""Answer using ONLY the context below. Cite chunk IDs in [brackets].
If the context doesn't contain the answer, say "I don't know."
 
Context:
{context}
 
Question: {question}"""
 
    # 5. Generate
    resp = client.messages.create(
        model="claude-sonnet-4-6", temperature=0,
        messages=[{"role": "user", "content": prompt}],
    )
    return {"answer": resp.content[0].text, "sources": top_chunks}

Frequently asked questions

What is RAG in simple terms?

RAG means retrieving relevant documents first, then asking the model to answer using that retrieved context.

When should I use RAG instead of fine-tuning?

Use RAG when knowledge changes frequently or you need citations. Fine-tuning is better for stable behavior or style changes.

Further reading

• RAG vs fine-tuning: how to choose — a practical decision framework.

• Retrieval-Augmented Generation — Lewis et al. 2020 — the original RAG paper.

• Anthropic — Contextual retrieval — significant improvements over vanilla chunking. Required reading.

• LlamaIndex — Production RAG — the production-RAG checklist.

• OpenAI — Question answering with embeddings — concrete cookbook recipe.