Scale Labs
BACK
AgentsEnterprise9/3/2026

Introducing READY: What It Takes to Deploy an AI Agent

By Yuan (Emily) Xue, Veronica Chatrath, Bryan Zhu

View paper

Enterprises everywhere are putting AI agents to work on business-critical workflows. However, measuring whether those agents are any good is harder than it looks. Today's benchmarks are capability tests: they score how often a model gets the right answer working alone, with no people involved. That tells you what it can do. It doesn't tell you whether it's ready to deploy, which is a different question. Big organizations now need to know whether an agent can hit the reliability a workflow requires, at a cost they can plan for, and how much human review it takes to get there.

Say an agent scores 80% on one of those capability tests. That number alone tells you almost nothing about whether it's ready to ship. What matters is the other 20%: whether the agent can flag its own weak answers, whether a person catches the ones it misses, and what that human review costs.

Introducing READY 

Today we're introducing READY (Reliable Enterprise Agent Deployment), a suite of industry-specific benchmarks that measure agents the way enterprises actually use them: working alongside people, inside real workflows. For any given workflow, READY Benchmarks measure three things together:

  • Reliability: How often the agent and its human operators get the work right.
  • Human Oversight: How much human involvement is needed to reach that level of reliability.
  • Cost: What it takes to run the agent and the human oversight together.

Measured this way, the result isn't a score. It's a deployment profile: this agent, in this workflow, with this much human oversight, reaches this level of reliability at this cost. Workflows differ enormously from one industry to the next, and so do the roles people play in checking the work. That’s why we’ll release more industry-specific benchmarks under READY over the coming months.

How it Works

How READY Works: 3 stages to a deployment profile

READY starts with an enterprise's workflow.

The enterprise defines what counts as correct work and how reliable the system needs to be, and READY finds the most cost-effective way to get there. Every system runs through three phases:

  1. Workflow evaluation: The agent works through real cases from the workflow. READY records the whole investigation, including the tools it called and the evidence it pulled, and scores each case against that workflow's own standard for correct work. It also records how confident the agent was in each answer.
  2. Policy optimization: Given a reliability target, READY finds the most efficient way to split the work between the agent and the human (i.e., oversight policy). The agent keeps the cases it is most confident in, and the rest go to a human reviewer. READY searches across possible oversight policies and picks the one that meets the target at the lowest cost.
  3. Independent validation: READY freezes the chosen split and re-tests it on cases it never saw. The result is certified only when it clears the reliability target with statistical confidence, under the stated assumptions.

Research Findings

We ran agentic systems through READY, and three things stood out that a capability score would have missed:

  • Identical benchmark scores can have opposite deployment verdicts. Two systems separated by only 0.3 points in autonomous accuracy (72.8% vs. 72.5%) require 39.2% versus 29.6% human review, respectively, to qualify at the same 76% reliability target.
  • Accuracy, and knowing when to escalate, are different skills. Across the 16 systems, accuracy and routing quality were essentially uncorrelated (Pearson ρ = −0.12). Qwen-3.7-Plus is the clearest case, pairing one of the lowest accuracies with the best routing signal of any system tested.
  • Reliability tops out at the human reviewer's own accuracy. Under an assumed 90% human-review success rate, no policy exceeds 90% reliability, and every system is driven to full human review by a target of about 85%.

What This Means for Enterprise AI

A READY deployment profile is deliberately specific: this agent, in this workflow, with this oversight model, under these assumptions. It doesn't transfer to a different workflow, and it isn't meant to. Deployment decisions are always situational, and a single portable score is exactly what has been failing the teams trying to make one.

The testbed is open, and the method works the same way for any workflow. We're launching with two healthcare benchmarks, CliniCARE-Bench and PSEBench, with other industries, such as financial services, coming soon.

Building those out takes people who know the work. We're looking for domain experts to write realistic enterprise tasks and researchers to build evaluation environments. If that's you, sign up here.

Resources:

All posts