RUBRIC DROPOUT: A SIMPLE WAY TO MITIGATE REWARD HACKING IN RUBRIC-AS-REWARD RL

Rubric-based RL can quietly learn to game its reward: the training judge keeps assigning higher scores even as true quality declines. A one-line fix, inspired by neural-network dropout, mitigates the problem at virtually no additional cost.

TL;DR

• Train against a fixed rubric for long enough, and the policy learns to game it: the training judge's score keeps rising while a stronger gold judge shows that quality peaked much earlier and has since fallen—by as much as 22 points out of distribution.
• Rubric Dropout: at each step, randomly drop a fraction of the rubric's criteria before computing the reward. No single criterion remains consistently available long enough for the policy to exploit it.
• Same peak, far less decay: +2 to +7 points in gold score at every matched checkpoint across two domains and three model sizes, with one line of code and no additional judge calls.

The Quiet Failure Mode

When a task has no objectively verifiable answer—medical advice or research explanations, for example—a common approach is rubric-as-reward RL: write a list of criteria for each prompt, have an LLM judge grade each one, and use the weighted fraction satisfied as the reward. The method is effective and auditable, but it has a built-in weakness: the rubric is a fixed proxy for quality, and the same criteria are scored at every training step.

We trained Qwen3-8B with GRPO on medical and science rubrics, then evaluated held-out benchmarks with two judges: the training judge and a stronger judge from a different model family. Here is how their assessments diverge:

On our science task pair, the collapse is even steeper: the gold score falls 22 points from its peak within 600 steps. We can also see what the policy learns in place of genuine quality: late in training, the baseline's responses degenerate into templated scaffolding—tidy headers and placeholder lists with little substance—that satisfies the rubric's structural criteria while saying almost nothing.

A criterion rewarded in the same way at every step becomes a stable target. Once the policy finds a cheap way to satisfy it, that shortcut is reinforced forever.

The fix is one line

Neural-network dropout prevents co-adaptation by ensuring that no single unit can be relied upon. We apply the same idea to the objective: no single criterion can be relied upon.

# reward, before:
reward = weighted_score(rubric, response)

# reward, after: drop a random fraction f of criteria each step
kept = sample(rubric.criteria, frac=1 - f, seed=hash(prompt_id, step))
reward = weighted_score(kept, response)

Three rules make this work in practice:

1 · Share the mask within each rollout group. GRPO compares multiple rollouts for the same prompt. If each rollout were graded against a different sub-rubric, those comparisons would be meaningless. Seeding the mask with (prompt_id, step) gives every rollout for a prompt the same sub-rubric without requiring cross-worker communication. It also causes the choice of reward normalizer to cancel out of the advantage calculation entirely.

2 · Protect what must never be dropped. Safety-critical criteria—for example, negative-weight "pitfall" checks—belong to a protected set that dropout never touches.

3 · Always evaluate on the full rubric. Dropout is a training-time perturbation only. Because the judge already grades every criterion in a single call, dropout requires no additional judge calls.

Does it work?

All comparisons use the same fixed protocol: the same training horizon, a fixed evaluation window (steps 400–600), and matched checkpoints evaluated on identical prompts. The effect is not limited to 8B models. We applied the same recipe, unchanged, at three model sizes:

At 8B, dropout improves the gold score by +1.0 and +2.0 points on Medical and by +6.4 and +7.0 points on Science, winning at all 11 matched checkpoints in both domains. Both measures of reward hacking also fall by several points: the degree to which the training judge overrates the policy, and the share of criteria it credits that the gold judge rejects. In-domain training reward remains matched at 97–98%, so these gains do not come at the expense of the reward being optimized. At 4B, the windowed gold score improves by +0.7 to +5.3 points. At 1.7B, both dropout fractions win on Science, while only the smaller fraction helps on Medical. One pattern is consistent across the table: the smaller the model, the smaller the optimal dropout fraction. A weaker model needs more of the rubric to remain visible in order to learn what quality means.

Not a delay, a different trajectory

Does the separation last? We ran the baseline and 50% dropout for two full epochs on both task pairs.

And it is not the same path traveled more slowly: past one epoch, at matched levels of overclaim, the dropout run holds a higher gold score at all 13 matched points on Medical. The two runs lie on different frontiers. Dropout changes the trajectory; it does not just stretch it out.

What's actually happening

Averaged over the mask distribution, dropout leaves the expected advantage unchanged except for a global scale factor that GRPO's standardization removes. Its entire effect comes from the variance it injects, and that variance is targeted. It is largest when a response's advantage depends heavily on a single high-weight criterion—exactly what an exploit looks like—and close to zero when a response is broadly better than the rest of its group. Gradient noise slows optimization in a particular direction, suppressing single-criterion exploits while allowing broad improvements to pass through.

One satisfying detail is that the injected variance scales as f·(1−f), which peaks exactly at f = 1/2. Empirically, 50% is also the best dropout fraction at 8B.

As for the fraction itself, we used 50%, and we would not call it optimal: anything from 30% to 50% worked, smaller models preferred the smaller fraction, and only at 60% did the kept sub-rubric stop capturing what quality means. In the end, f is one hyperparameter, and the right value for a new setup is a single cheap experiment.

The opposite bet fails: reweighting criteria toward those most informative during training performed worse than doing nothing, with the lowest gold score and the highest overclaim of any run. Concentrating optimization pressure feeds the hack; spreading it fights it.

The rubric is a proxy. You can't stop a proxy from being a proxy, but you can stop it from being a stationary target.