Write Evals Before You Write Prompts

Every RAG system I’ve seen fail in production failed the same way: it worked in the demo, passed a vibe check, and then quietly returned worse answers than the keyword search it replaced. The root cause is almost never the model. It’s the absence of an evaluation harness that would have caught the regression before users did.

The eval set is the spec

Before you touch a prompt or pick an embedding model, write down 50–200 question/answer pairs that represent the queries your system must handle. These are not test data — they are the specification. If you cannot write them, you do not yet know what you are building.

• Include the easy wins — they protect against regressions.
• Include the adversarial cases — wrong answers that look plausible.
• Include the unanswerable cases — queries where the right answer is “I don’t know.”

Measure retrieval and generation separately

A RAG failure has two parents: the retriever returned the wrong chunks, or the generator misused the right chunks. If you only score the final answer, you cannot tell which. We score recall@k on the retriever and faithfulness on the generator, then combine them into a single number we trend over time.

def recall_at_k(retrieved_ids: list[str], relevant_ids: set[str], k: int = 5) -> float:
    if not relevant_ids:
        return 0.0
    top_k = retrieved_ids[:k]
    hits = sum(1 for doc_id in top_k if doc_id in relevant_ids)
    return hits / len(relevant_ids)
# Trend this number across every index change, embedding swap,
# or chunking strategy update. If it drops, you ship nothing.

LLM-as-judge, with guardrails

For generation quality, human grading doesn’t scale. We use an LLM-as-judge with a rubric, but we anchor it: a small human-graded set calibrates the judge, and we track agreement. If the judge drifts from humans, we re-calibrate. Never trust a judge you don’t measure.

If you cannot write the eval set, you do not yet know what you are building.

Production is an eval that never stops

Once the system is live, the eval set keeps running on a sample of real traffic. Drift in recall or faithfulness triggers a rollback, not a post-mortem. The same harness that gated the launch now guards it. That is the difference between a demo and a product.