Capstone: build a tiny RAG bot

Take everything you've learned in the Medium track — embeddings, vector DBs, RAG, prompting, evals — and build the smallest version that actually works on your data.

Why this matters

You've read the recipe for RAG. Cooking it is different. By the time you finish this, you'll have:

• A working RAG bot grounded in your documents.
• A 5-question eval set that catches regressions.
• The hands-on intuition for which knob does what when results aren't great.

That's a real skill. It's also the foundation for almost every "AI on your data" product worth building.

The challenge (~2–3 hours)

Step 1 — Pick a corpus (10 min)

Choose 5–10 documents you actually know well:

• Your team's internal wiki / runbooks.
• Your own writing — blog posts, talks, reports.
• A specific section of public docs (a programming framework, a reference manual).
• A handful of PDFs on a topic you care about.

Keep it small. A 10-doc corpus you understand deeply is far more useful for learning than 1,000 docs you don't.

Step 2 — Build the index (30 min)

Minimum-viable pipeline:

chunk each doc (200–500 tokens) → embed each chunk → store {text, embedding}

Choices that don't matter much for this exercise:

• Embedding model: any of text-embedding-3-small (OpenAI), voyage-3 (Voyage), or a local one via Ollama.
• Storage: in-memory list of dicts is fine. pgvector if you want production-shaped from day one.
• Chunking: split on H2 headings, paragraph breaks, or every 500 tokens with 50-token overlap.

Choices that matter:

• Use the same embedding model for indexing and querying. Mix them and recall collapses.
• Keep the original text alongside the embedding so you can show citations.

Step 3 — Build the answer function (30 min)

question → embed it → top-5 nearest chunks → put them in a grounded prompt → get answer

The grounded prompt template (cribbed from lesson 6):

You are answering questions ONLY using the supplied context.
If the answer isn't in the context, reply exactly: "I don't know."
Cite the chunk IDs in [brackets] for each claim.

Context:
[1] {chunk1}
[2] {chunk2}
[3] {chunk3}
…

Question: {question}

Step 4 — Build a 5-question eval (20 min)

Write five questions you know the answer to, covering:

1. One easy match — the answer is verbatim in one chunk.
2. One that requires combining two chunks.
3. One the docs don't cover — the bot should refuse and say "I don't know."
4. One adversarial — a question that sounds like it's about your domain but isn't (try to trick it into hallucinating).
5. One paraphrase — the answer is in the docs, but the user's wording is very different from the doc's wording.

Write the expected behaviour for each.

Step 5 — Run, score, fix (45 min)

Run all 5 questions. Score:

• ✅ Right answer with right citation
• 🟡 Right answer but missing/wrong citation
• ❌ Wrong answer
• ✅ Correctly refused (for question 3)

If anything fails, fix one thing at a time. Common fixes:

After fixing, re-run all 5. Don't optimise just the failing one — make sure you didn't break the working ones.

Step 6 — Reflect (15 min)

Write 3 bullets:

1. What surprised you? (Often: how much chunking strategy matters.)
2. What was easier than expected? (Often: the actual code; LLM APIs are simple.)
3. What would you change for a 100-doc, 1000-question version? (Hint: where would you add a reranker, prompt caching, an LLM-as-judge eval?)

Self-check

What's next

You've now built the core architecture behind most "AI on your data" products. From here:

• Scale it up: more docs (10 → 1,000 → 100,000) brings reranking, hybrid search, and proper monitoring into play. The Advanced track covers the production layer.
• Productionise it: add caching, retries, fallback providers, observability — covered in Production deployment.
• Explore agentic RAG: instead of one-shot retrieval, let an agent decide when and what to retrieve over multiple turns.

Or: ship the tiny version. Boring, working, grounded RAG over your real docs is more useful than a never-shipped fancy version.