Human-in-the-Loop
A deployment pattern where every AI agent action is reviewable by a human before downstream effects, used during the first weeks of production.
Also known as: HITL · human-on-the-loop · supervised deployment
What is Human-in-the-Loop?
Human-in-the-loop (HITL) is a deployment pattern where every AI agent action is reviewable by a human before downstream effects take place. theagency47 uses a 14-day HITL window after every production launch: the agent drafts outputs, a human approves them, and the agent's pass rate accumulates against the eval suite under real traffic. After 14 days, if pass rate exceeds the agreed threshold (typically 95%), the agent moves to autonomous mode with sampling-based audit. HITL is what makes production deployment safe, it bounds the worst-case mistake to a reviewable draft.
Christos Papadimitriou, theagency47 · Updated May 15, 2026How HITL works in practice
For the first 14 days after deployment:
- The agent processes inputs and produces a draft output.
- The draft is queued for a human reviewer (assigned per agent).
- The human approves, edits, or rejects.
- Approved drafts go out; edited drafts go out with the edit; rejected drafts feed back into eval refinement.
After 14 days:
- If pass rate ≥95%, the agent moves to autonomous mode for that category of action.
- If pass rate is under 95 percent, HITL continues for another period and we investigate failures.
- High-stakes actions (refunds above policy, external communications to executive contacts) stay HITL forever, regardless of pass rate.
Why HITL is non-negotiable
Without HITL, the first production week of any AI agent surfaces issues that did not show up in eval, new input patterns, integration quirks, edge cases of business logic. HITL turns those into approval-queue items rather than production incidents.
FAQ
Does HITL slow down the agent?
For the first 14 days yes, by adding human review time. But humans review faster than they produce, so the team is still net-faster than baseline.
What if there is no human available to review?
Then the agent waits, or escalates differently. We design queues with timeout escalation. No silent failures.
Related terms
Bounded Autonomy
The engineering pattern of giving an AI agent specific permissioned actions and explicit escalation rules, letting it act independently within tight, defined boundaries.
Escalation Rule
A configured condition that hands an agent task off to a human when the agent is uncertain, out of scope, or facing high-stakes input.
Eval Suite
A structured set of test cases with known expected outputs that verifies an AI agent's behavior before deployment and after every change.