You can run a useful evaluation in an afternoon: fifty real inputs, one check that runs automatically, and a scoreboard you keep in the repository.
The check is the hard part, and it is the part that makes the number mean anything.
1. Take fifty real inputs
Pull them from your logs, not from a benchmark. Real inputs carry the mess: the typo, the missing field, the two-sentence request that assumes context.
Fifty is enough to see a difference of ten points and small enough to finish today.
2. Write the check
Four kinds, in order of how much they prove:
- Exact match, when there is one right answer.
- A schema or type check, when the output has a shape.
- A unit test, when the output is code that runs.
- A rubric applied by a second model, when the output is prose. Keep the rubric to five yes-or-no questions.
If you cannot write any of these, you are not ready to evaluate. You are ready to read outputs, which is a different and slower activity.
3. Run both models the same way
Same prompt, same tools, same temperature, same number of attempts. Change one thing between runs or the comparison is a story.
4. Count, do not average
Record pass or fail per input. A model that passes 43 of 50 and fails the same 7 is different from one that passes 43 and fails a random 7 — the first has a hole you can route around.
5. Keep the set
Add every production failure to the file. In a month it is the only benchmark that describes your product.
Tools, if you want them
EleutherAI's lm-evaluation-harness runs standard academic benchmarks. Promptfoo compares prompts and models with assertions. Inspect, from the UK AI Safety Institute, is built for agent evaluations with tools.
All three are open source. None of them will write your check for you.