LLM evaluation: tests for the non-deterministic.

By Mokshify Engineering · Reviewed by the team · Updated 17 Jul 2026 · 6 min read

The problem

Code has tests; AI features usually have vibes. A prompt change that improves one answer silently ruins twelve others, a model upgrade shifts behaviour overnight, and nobody can say whether the feature got better or worse - because nobody defined what better means. Evaluation is how AI features earn the release discipline the rest of the codebase already has.

The eval suite, concretely

A test set of real cases. Ten to fifty genuine inputs from the actual workflow - real documents, real questions, including the awkward ones that made someone escalate. Synthetic test cases test your imagination, not your product.
Expected outcomes, not expected strings. For extraction: the correct fields. For answers: the facts that must appear, the claims that must not, the citation that must ground it. Exact-match assertions break on phrasing; fact-level checks survive it.
Scoring you can defend. Deterministic checks first (schema validity, required fields, forbidden content, citation presence) - they are free and unarguable. Model-graded scoring on top for fluency and correctness, calibrated against a sample of human judgments so the grader itself is validated, not just trusted.
A gate, not a dashboard. Evals run in CI like unit tests; a release that regresses the score does not ship. This is the same principle as the rest of our QA - quality is gated, not hoped.

What to measure per use case

RAG: retrieval hit rate (did the right passage arrive?) separately from answer faithfulness (did the model stick to it?) - conflating them hides which stage failed. See the RAG pipeline.
Extraction: per-field precision and recall; a 98% overall score can hide a 60% failure on the one field finance cares about.
Agents: task completion, unnecessary-action count, and refusal correctness - an agent that never refuses is more dangerous than one that sometimes fails.

Advantages & disadvantages

Advantages: prompt and model changes become safe, comparable engineering decisions; provider switches (ADR-008) become measurable; drift gets caught by the suite, not by customers. Disadvantages: building the first test set takes real domain time - it is the honest cost of AI in production; model-graded scores need periodic human recalibration; evals measure what you encoded, so unencoded failure modes still need human QA passes.

Technology selection

Plain pytest-style harnesses in the product repo - eval cases are versioned fixtures, scores are CI artifacts, and the whole thing runs in the same pipeline as everything else. No exotic eval platform until scale demands one; the discipline matters more than the tooling.