Eval Suite
A structured set of test cases with known expected outputs that verifies an AI agent's behavior before deployment and after every change.
Also known as: eval · test suite · agent evals
What is Eval Suite?
An eval suite is a structured set of test cases (typically 20 to 30 per agent) with known expected outputs, used to verify the agent's behavior before deployment and after every change. theagency47's standard structure has four buckets: 60% common-case, 20% edge-case, 15% adversarial (including prompt injection), and 5% regression (specific bugs we have already fixed once). Agents must achieve 90%+ pass rate on the full suite before reaching production, and the suite re-runs automatically after every prompt or knowledge-base change. Without an eval suite, you are demoing in production.
Christos Papadimitriou, theagency47 · Updated May 15, 2026Why eval suites exist
LLM outputs are non-deterministic, the same input can produce slightly different outputs across calls. Traditional software testing (exact-match assertions) does not work. Eval suites use rubric-based scoring instead: each test case has expected behavior described in dimensions (accuracy, tone, escalation, format), and a separate grader scores each output against the rubric.
This is the difference between “AI demo” and “AI deployment.” Most AI agency failures in 2026 come from skipping this step.
The four buckets
- Common-case (60%), everyday work the agent must handle correctly
- Edge-case (20%), rare-but-real inputs the agent will see eventually
- Adversarial (15%), prompt injection, false data, scope-creep attempts
- Regression (5%, grows), specific bugs we have already fixed once
Full breakdown with examples in our blog post on the methodology.
FAQ
Who builds the eval suite?
theagency47 builds it during discovery (days 1–5 of a Spark engagement). The client validates test cases. We do not deploy without sign-off.
How often does the suite run in production?
Before every change (automatic). Weekly in background as health check. On-demand when investigating production tickets.
What if an eval fails after a production change?
Automatic revert to the last passing version. The change that broke the eval is investigated. Production fixes go through the same eval gate as initial deployment.
Related terms
AI Agent
A software system that takes a goal, plans steps, calls tools, and acts autonomously to produce an output, without step-by-step human prompting.
System Prompt
The instructional text that defines an AI agent's role, voice, boundaries, and decision rules. The single most important customization layer of an agent.
Hallucination
An AI agent producing output that sounds confident but is factually wrong, typically because the model invented information not present in its inputs.