Eval Suite
A structured set of test cases with known expected outputs that verifies an AI agent's behavior before deployment and after every change.
Also known as: eval · test suite · agent evals
What is Eval Suite?
An eval suite is a structured set of test cases with known expected outputs that verifies an AI agent’s behavior before deployment and after every change. Because LLM outputs vary between calls, exact-match tests fail, so a grader scores each output against a rubric for accuracy, tone, escalation and format. It separates an AI demo from an AI deployment.
Christos Papadimitriou, theagency47 · Updated May 15, 2026Why eval suites exist
LLM outputs are non-deterministic, the same input can produce slightly different outputs across calls. Traditional software testing (exact-match assertions) does not work. Eval suites use rubric-based scoring instead: each test case has expected behavior described in dimensions (accuracy, tone, escalation, format), and a separate grader scores each output against the rubric.
This is the difference between “AI demo” and “AI deployment.” Most AI agency failures in 2026 come from skipping this step.
The four buckets
- Common-case (60%), everyday work the agent must handle correctly
- Edge-case (20%), rare-but-real inputs the agent will see eventually
- Adversarial (15%), prompt injection, false data, scope-creep attempts
- Regression (5%, grows), specific bugs we have already fixed once
Full breakdown with examples in our blog post on the methodology. OS47 is where clients see eval results for each of their agents.
FAQ
Who builds the eval suite?
theagency47 builds it during discovery (days 1–5 of a Spark engagement). The client validates test cases. We do not deploy without sign-off.
How often does the suite run in production?
Before every change (automatic). Weekly in background as health check. On-demand when investigating production tickets.
What if an eval fails after a production change?
Automatic revert to the last passing version. The change that broke the eval is investigated. Production fixes go through the same eval gate as initial deployment.
Related terms
AI Agent
A software system that takes a goal, plans steps, calls tools, and acts autonomously to produce an output, without step-by-step human prompting.
System Prompt
The instructional text that defines an AI agent's role, voice, boundaries, and decision rules. The single most important customization layer of an agent.
Hallucination
An AI agent producing output that sounds confident but is factually wrong, typically because the model invented information not present in its inputs.
