Who Grades the Graders? Rethinking Verifier Design for Computer-Use Agents

As Computer Use Agents (CUA) take on complex professional tasks like processing emails, creating spreadsheets, drawing 3D diagrams, and drafting financial memos, verifiers are at the heart of providing evaluation and training signals around their capabilities. Benchmarks like OSWorld 2.0 and Agents' Last Exam use these verifiers, in the form of programmatic checks, to measure whether an agent is capable of completing production-grade work.

However, when verifier scores signal deployment readiness or serve as training rewards, grader errors become more than just minor academic flaws; they directly degrade model quality. To investigate this issue, we analyzed 38 tasks from our professional office CUA training dataset (including workflows in finance, legal, and engineering), along with OSWorld 2.0 tasks, assessing the quality of verifiers behind them.

Flaws in Benchmark’s Programmatic Graders

Our evaluation uncovered two primary failure modes in standard benchmark grading from OS World 2.0:

All-or-Nothing Binary Grading: In complex multi-step workflows like OSWorld 2.0 (task_010), the agent must read emails, find an approval letter among decoys, and compile a checklist across ten files. Despite having dozens of distinct steps, the grader performs a single binary comparison against a reference file, returning a rigid 0 or 1. On the public leaderboard, this causes severe ranking inversions: a state-of-the-art model scores a 0.0, while two weaker models receive a 1.0. For reinforcement learning, these all-or-nothing signals severely hinder training rewards, and provide sparse evaluation metrics for long-horizon tasks.

Brittle and Unrealistic Requirements: In OSWorld 2.0's Task 018, the agent must schedule a meeting via email and calendar. Even if an agent successfully schedules the meeting through a slightly different path, or presents a minor formatting variation or a small timing offset, the scoring leaves no room for natural nuance and the agent gets the exact same score as an agent that never tried: zero.

The Adversarial Reality: Why Rubric Design Is Hard

The two approaches to writing verifiers include:

1. Programmatic checks (OSWorld-style): small programs that open the agent's output and test it (i.e. does the file exist, do the totals match, are the rows in order, etc.).
2. VLM Agent Judge: give a model the agent's work, a reference ground truth answer, and a rubric covering multiple success dimensions and let it score the output.

There are pros and cons to each, which are summarized below:

Scale AI’s Approach

Scale approaches this challenge by combining the strengths of both methodologies. Start from human-written rubrics, which capture what professionals believe correct. Then convert every item that a deterministic check can accurately evaluate. And for the rest of the rubrics, continue grading them with a judge whose inputs have been verified by a human expert. And before any converted deterministic check ships, pass a defective artifact to the new grader and confirm that the defect is caught, with the scoring being penalized.

The Reward Hacking Trap

Designing rubrics that resist "reward hacking", where the agent exploits grading shortcuts to get a high score without doing the work, is notoriously difficult. When evaluators rely on loose heuristics, agents quickly optimize for the metric rather than the task:

• Structure-Blind Matching: Evaluators that look for unconstrained keywords or numbers allow agents to dump unstructured text blobs. An agent can score 1.0 without building a table, binding values to labels, or understanding context. Scale's recent research (https://arxiv.org/html/2607.28802v1) gives an example: an agent told to beat a chess engine "win against a chess engine edited the board state until the opposing engine resigned," scoring "a win even though the agent had bypassed the intended game". A number dump is the same trick against a weaker check.
• Instruction Leakage: String-overlap rubrics frequently reward agents for copying prompt text verbatim into deliverables, generating high confidence scores while performing zero actual execution or research.
• High-Scoring "Do-Nothing" Strategies: Weak existence checks or unconstrained baselines allow agents to submit blank templates or unedited original files and still score over 80%.
• Flawed Reference Data: Case-sensitivity bugs or incorrect ground-truth files actively penalize correct agent solutions while accidentally rewarding degenerate shortcuts.

Closing these loopholes requires anticipating every edge case beforehand, balancing strict correctness with the flexibility needed for open-ended problem-solving.

To fix this, we separate evaluation into two distinct checks:

• Rubric Vulnerability (Can it be gamed?) rubrics that give credit just for matching an expected answer file can be passed by copying the reference instead of doing the analysis.

For example, we had a task that asked the agent to transform a messy dataset into a spreadsheet, a report, and a slide deck. Most of the reward was attributed to matching a set of ground truth expected files, so an agent could score high just by reproducing the reference values without doing real data cleaning. On top of that, the same figures were scored over and over across different artifacts, once in the spreadsheet, again in the report, and again in the slides, so a single answer earned credit several times.

We fix this by scoring the work itself: we recompute the results from what the agent actually submitted, fold the repeated criteria together, keep formatting checks separate from correctness, and treat the reference only as a secondary sanity check.

• Agent Cheating (Did the agent game it?) We have observed undesirable agent behaviors such as reading the answer key or hardcoding a test instead of computing an output to solve the task.

For instance, on a task asking to create two financial risk reports, an agent skipped the analysis, opened our expected answer file with a text-extraction tool and copied the answers word for word into the output. The output looked perfect and passed, but instead of computing an answer, the agent simply read the answer key. In our catalog of violations, this is considered an Oracle Read (exploitation of artifacts / metadata) and is flagged as cheating by the agent.

Practical Insights

We ran some experiments to stress-test our CUA verifier creation pipeline. Insights below.

Rubric <-> Verifier Conversion Agreement

When our verifiers and our judge graded the same outputs, they agreed on 99.5% of the items the judge passed. All disagreements were adjudicated by hand.

Even our production judge, given three byte-identical copies of the same output, returned 0.0, 0.125, and 0.175. It samples its verdict instead of computing it, so identical inputs diverge. The programmatic verifier scored all three identically. The takeaway: convert rubric items that must be reproduced exactly, like counts and format checks, into programmatic verifiers, and leave items that require subjective judgement to the judge.

Evaluators Suffer From Pipeline Blindness

When an AI judge evaluated a known-perfect reference answer for a financial memo task, it awarded a score of only 0.5.

The text-extraction pipeline pulled raw text and tables but completely ignored embedded chart images. Paradoxically, the judge marked the unextracted chart items as correct, explicitly stating it "cannot verify specific data from rendered text". Blind judges tend to give unearned credit rather than failing unread elements.

Adding a vision model step to process chart images brought the reference score to 0.98. This visual verification also caught real errors, failing a frontier model that drew a chart bar with a flipped sign (showing a gain instead of a loss) and dropping its score to 0.35.

One wrong number, thirty-eight perfect scores

We generated 603 programmatic verifiers using coding models to scale task creation. While all 603 passed standard sanity checks by scoring reference answers at 1.0, reference testing alone is weak.

To test their true sensitivity, we intentionally corrupted the single most important number in each deliverable across 38 tasks, representing the exact error that invalidates a document in real life.

Zero out of 38 programmatic verifiers failed the corrupted document.

• 25 verifiers completely missed the corruption, awarding a flat 1.0 score.
• 13 verifiers caught the error at the item level, but weighted averaging diluted the penalty, capping the total score drop between 0.4% and 10.7%.

Takeaways

• Corruption-Test Verifiers at Generation: Never trust a verifier just because it scores reference answers at 1.0. Apply mutation testing by deliberately corrupting key output values to verify that scores actually drop.
• Combine Hard Gates and Dense Rewards: Use both signals for their intended roles. Boolean hard gates act as strict pass/fail checks on core facts - if the headline number is wrong, the entire deliverable fails. Granular, per-item scores then provide the smooth learning signal agents need to learn complex, long-horizon tasks. Without hard gates, weighted averages cushion critical errors, diluting a fatal mistake into a negligible score drop.
• Verify Pipeline Inputs: Ensure LLM judges receive complete visual renderings through vision models rather than lossy text extractions.
• Separate Gameable Rubrics from Cheating Agents: Reward hacking audits ask two different questions. Check whether the rubric can be gamed: score the submitted work itself, recomputed from scratch, rather than resemblance to a reference file. Separately, check whether the agent actually cheated, using trajectory evidence like an answer-key read. Fix the first, flag the second, and never treat a weak rubric as proof of cheating.
• Combine Agentic and Manual Audit Pipelines: Catching reward hacks requires continuous adversarial testing. By pairing automated agentic flaw-detection pipelines with targeted human review, benchmark builders can systematically catch prompt leakage, structure-blind matching, and reference errors before verifiers go live.

Reliable evaluation is not an afterthought, it is the cornerstone of agent development. Combining mutation-tested programmatic verifiers, visually verified judges, and adversarial audits that keep rubric flaws separate from agent cheating creates evaluation systems that remain accurate as computer-use agents continue to evolve.