1. In RAG, what does the 'retrieve' step actually do?
Calls the LLM to generate. Embeds the user's query and finds the top-k most similar chunks in your vector DB. Re-tokenises the input. Streams the response.
2. Why is metadata filtering essential in multi-tenant vector search?
It speeds up queries. It's the security boundary preventing one customer from retrieving another's data. Required by the embedding model. It's not — vector similarity is enough.
3. Most reliable way to get strict JSON from an LLM:
Beg in the prompt. Use the API's structured-output / tool-calling support with a schema. A regex. Manually parse markdown code fences.
4. Few-shot prompting is most useful when:
Prototyping a generic chatbot. The task has tricky edge cases or a specific format the model needs to see demonstrated. You want lower latency. Training a new model.
5. Tool calling: who actually runs the tool?
The model executes it internally. Your code, in response to a structured request from the model. A separate billing service. Nobody; it's just a hint.
6. Anthropic's 'Building Effective Agents' argues that:
Agents are always better than fixed pipelines. Most use cases are better served by simpler workflows than by full agentic loops. Frontier models eliminate the need for tool calling. Agents must always run multiple models in parallel.
7. What does `cache_control: ephemeral` on a static prefix do?
Caches the output on the client. Caches the static prefix server-side; subsequent matching calls re-read those tokens at a deep discount. Makes the model deterministic. Nothing — caching is automatic.
8. LLM-as-judge evals are best for:
Determining the model's parameter count. Scoring qualitative properties like faithfulness or instruction-following at scale, calibrated against human labels. Bit-for-bit identical output. Replacing all human review.
9. Why is hybrid search usually better than pure vector?
Vector misses exact-match queries (SKUs, IDs, error codes); keyword catches them. Keyword search is faster. Hybrid is the only mode vector DBs support. Marketing term, no real difference.
10. Building a flight-booking assistant — safest way to expose the booking action:
Trust the model to confirm. Have a `confirm_booking` tool requiring explicit user approval / re-auth. Make booking automatic so the experience is fast. Limit tool calls to one per session.
11. Right chunks retrieved, but the model answers from training data anyway. Fix:
Bigger model. Strengthen the prompt to use ONLY the supplied context, place context next to the question, and verify groundedness with an eval. Higher temperature. More chunks.
12. Cost is dominated by re-sending a 5k-token system prompt every turn. Best fix:
Switch model. Apply prompt caching to the static prefix. Strip whitespace. Use top_p=1.