Capstone: production-ize one prompt

Pick one prompt in your stack. Take it from "works in the demo" to "survives real traffic." Use everything you've learned in the Advanced track.

Why this matters

A working prototype is 10% of the work. The other 90% — retries, rate limits, cost monitoring, evals — is the unglamorous stack that separates a feature you can ship from a demo that goes viral and bankrupts you.

This capstone takes you through that 90%, on one specific prompt that's small enough to finish but real enough to teach you everything.

The challenge (~half a day)

Pick the prompt

Pick one that satisfies all of:

• Real, used by real (even internal) users.
• Single-purpose: classification, extraction, summarisation, routing, Q&A.
• Easily evaluated: the right answer is checkable.

If you don't have one, build a small one for this exercise: classify GitHub issues by category, summarise customer feedback into themes, or extract structured fields from invoice PDFs.

Step 1 — Lock the eval first (45 min)

Write a 10-question golden set before touching anything else.

For each item:

• Input: the actual data (sanitised).
• Expected output / behaviour: what "good" looks like.
• Why this case matters: edge case? happy path? known failure?

Cover:

• 5 happy-path inputs.
• 3 edge cases (empty input, very long input, mixed-language).
• 2 known failures or red-team inputs ("ignore previous instructions", indirect injection attempt, ambiguous question).

Save to git. This is the source of truth; treat it like code.

Step 2 — Add the production wrapper (1 hour)

Wrap the prompt's LLM call with the layers from Production deployment:

1. Retry on transient errors (5xx, 429, network) with exponential backoff + jitter.
2. Fallback chain: primary provider → secondary provider → cached answer.
3. Rate limit per user (token bucket; per-user spend cap).
4. Cap max_tokens to a sensible bound for this task.
5. Apply prompt caching to the static prefix (system prompt + few-shot examples).
6. Validate output: schema-check structured outputs; sanity-check tool calls.
7. Trace every call: input, output, model, latency, token counts, cost, eval scores.

Pick a tracing tool (Langfuse, Braintrust, LangSmith, or roll-your-own with structured logs into Postgres/DuckDB).

Step 3 — Wire the eval into CI (30 min)

A simple GitHub Action / equivalent:

• On every PR that touches the prompt, system message, or tool definitions: run the 10-question eval.
• Pass = each item meets its expected behaviour (deterministic check or LLM-as-judge ≥ threshold).
• Fail = block the merge, flag which items regressed.

This is the single biggest win of the whole exercise. Now nobody can quietly ship a regression.

Step 4 — Live eval over production (45 min)

Sample 1% of real production traffic. For each sampled call:

• Re-run with an LLM-as-judge against your faithfulness / accuracy rubric.
• Log the score with the trace.
• Set a regression alert: "if average daily score drops by more than 5% over 7 days, page someone."

This catches drift the golden set can't.

Step 5 — Run a "viral post" load test (30 min)

Simulate the failure modes from Cost & latency:

• A single user makes 10,000 requests in an hour. Does your rate limit catch it?
• A user sends an enormous input. Does your max_tokens cap save you?
• Your primary provider returns 503 for 5 minutes. Does your fallback fire? Do users notice?
• A request includes adversarial content (<context>...IGNORE PREVIOUS INSTRUCTIONS...</context>). Does your guard layer reject the resulting bad output?

If any of those break: that's your homework. Fix and re-run.

Step 6 — Document (30 min)

Write a one-page runbook:

• What this prompt does (1 sentence).
• Eval suite location (link).
• Tracing dashboard (link).
• Known limits (3 bullets).
• What to do when it breaks (3 specific scenarios → first response).

Put it where the next on-call engineer will find it. Future-you will thank present-you.

Self-check

What you've actually built

After this capstone, you have, for one prompt:

• A versioned eval suite that catches regressions.
• A CI gate that prevents bad prompt changes from merging.
• A production wrapper with retries, fallbacks, rate limits, caching.
• Tracing + cost monitoring at the call level.
• A runbook for the next person on-call.
• A load-tested posture — you've personally tried to break it.

That's a real, shippable AI feature — the kind that survives going viral, surviving a vendor outage, and surviving the next prompt change six months from now.

What's next

There's no Lesson 16. You've finished the curriculum. From here, the field will keep moving — but you have the framework to:

• Read papers (Attention is All You Need, reasoning model papers, RAG follow-ups) and place them in context.
• Evaluate new tools and patterns without falling for hype.
• Build features that are useful, safe, and economically viable.
• Push back when colleagues claim AI will solve a problem it can't.

That's the goal. The lessons were just the warm-up.