Evals & Quality
The part that separates a demo from a product — knowing whether a change made the thing better.
6 topics
Why Evals Are the Real Work
Without a way to measure quality you cannot tell improvement from regression, which means every change after the demo is a guess.
core20 minBuilding an Eval Set
Where the cases come from, how many you need, and why a set built from imagination tests a product nobody is using.
core20 minLLM as Judge
Using a model to score outputs makes subjective quality measurable at scale — provided you treat the judge as a component that itself needs validating.
core20 minRegression Testing Prompts
Wiring the eval suite into CI so a prompt, model, or retrieval change cannot ship a silent regression.
core15 minOffline & Online Evaluation
Offline evals tell you whether to ship; online signals tell you whether it worked — and when they disagree, production is right.
core15 minEvaluating Agent Trajectories
Agents need evaluating on the path as well as the destination, because a correct answer reached through twelve wasted tool calls is a different product than one reached in three.
deep20 min