To Critic or Not to Critic: GRPO vs PPO

As agentic tasks become more complex and rollouts become longer and more expensive, improving the efficiency of reinforcement learning (RL) becomes increasingly important. Learned critics can reduce the variance of policy gradient estimates, potentially improving sample efficiency, but they also introduce additional computational overhead and can introduce bias through bootstrapping. Understanding when and under what conditions critics improve overall training efficiency remains an important open research question.

We investigate this trade-off through an extensive empirical study spanning more than 1,200 training runs across diverse tasks and rollout horizons. Our experiments range from continuous-control environments with varying horizon lengths and state-space complexity to single-turn mathematical reasoning and long-horizon terminal agents that interact with external tools and receive delayed outcome rewards.

Three takeaways

1. Batch size and critic reliance decide the winner. PPO generally leads at small batch sizes. As batch size grows, GRPO catches up, often overtaking heavily bootstrapped PPO and reaching parity with lightly bootstrapped PPO.

2. More complex tasks push the crossover to larger batch sizes. With longer horizons and more complex domains, GRPO needs larger batch sizes before it catches up.

3. Critics pay off when compute is limited. At small compute budgets, PPO’s critic can save enough rollouts to justify its extra training cost. At large compute budgets, GRPO can match PPO with less training compute.

When should we use GRPO versus PPO?

Early RLHF pipelines were built on Proximal Policy Optimization (PPO) [1], which trains a separate critic network to estimate how good each state is. The critic lowers gradient variance and gives token-level credit assignment through Generalized Advantage Estimation (GAE) [2]. Since DeepSeek-R1, most open-source work has switched to Group Relative Policy Optimization (GRPO) [3, 4], which drops the critic and instead compares rewards across a group of trajectories sampled from the same prompt. GRPO saves memory, is simpler to implement, and avoids the bias that comes from bootstrapping off an imperfect value function.

More recently, critic-based methods such as VAPO and SAO have reported strong results against GRPO [5, 6]. However, these comparisons typically use a single batch size and focus on math and coding, leaving open whether these gains extend to other batch sizes and domains.

Critics can reduce the samples needed per update, but add compute and potential bias from bootstrapping. We study how this tradeoff depends on three factors:

1. Batch size: the number of trajectories per actor update. Larger batch sizes reduce variance but not bias.
2. Critic reliance: controlled by PPO's GAE parameter λ. Smaller λ bootstraps more from the critic, lowering variance at the cost of bias from critic error.
3. Task complexity: which we study through horizon and domain. We observe higher gradient noise at longer horizons and in the more complex control domain.

Setup

We compare PPO and GRPO in three settings: control tasks, short-horizon math, and long-horizon terminal agents.

• Control tasks. Fixed-horizon Pendulum and HalfCheetah with the reward given only at the final step, which mimics outcome rewards in LLM RL. These let us estimate gradients accurately and vary the horizon while holding everything else fixed.
• Short-horizon math. Single-turn DeepScaleR problems with Qwen2.5-3B Base, evaluated on MATH-500, GSM8K, AMC23, Minerva Math, and OlympiadBench.
• Long-horizon terminal agents. Multi-turn TMax-15K terminal tasks with Qwen3.5-4B, evaluated on Terminal-Bench 2.1 and TB-Lite [7].

To enable a controlled comparison, PPO and GRPO use the same actor optimization objective, policy initialization, and total trajectory batch size, while differing in their advantage estimation and associated sampling strategies. PPO additionally trains a critic.

We control PPO's critic reliance with length-adaptive GAE from VAPO, where λ = 1 − 1/(αH) [5]. Larger α means less bootstrapping. We test α ∈ {0.1, 0.5, 1.5}, which ranges from heavily bootstrapped (α=0.1) to lightly bootstrapped (α=1.5).

Finding 1: Batch size can change which algorithm wins

At the smallest tested batch sizes, PPO generally leads GRPO, with the exception of the shortest Pendulum task.

As batch size grows, GRPO catches up. How GRPO catches up depends on how much PPO relies on its critic. Against heavily bootstrapped PPO (α=0.1), 6 of 7 comparisons end with GRPO leading. Against lightly bootstrapped PPO (α ∈ {0.5, 1.5}), only 6 of 16 do, and the rest end in parity or a continued PPO lead.

The LLM settings show the same pattern. In math, GRPO overtakes heavily bootstrapped PPO at batch size 1,024 and reaches parity with the lightly bootstrapped variants. In terminal agents, PPO leads at batch size 128 and GRPO reaches parity by a batch size of 2,048.

The intuition: a critic reduces noise, and increasing the batch size also reduces noise. Once the batch size is large enough, the critic's variance reduction matters less, but any bias it introduced remains. Heavily bootstrapped critics carry the most bias, so they lose their advantage first.

Finding 2: Task complexity shifts the crossover

The more complex the task, the larger the batch size GRPO needs to catch up.

Horizon. In Pendulum, GRPO leads heavily bootstrapped PPO at a batch size of just 4 at a horizon of 5. At a horizon of 200, it doesn't catch up until a batch size of 65,536.

Domain. HalfCheetah, a higher-dimensional locomotion task, is consistently less favorable to GRPO than Pendulum. At a horizon of 200, lightly bootstrapped PPO still leads even at a batch size of 65,536.

The LLM settings follow the same pattern. GRPO catches up to lightly bootstrapped PPO by a batch size of 1,024 in single-turn math, while multi-turn terminal agents need a batch size of 2,048.

Finding 3: Training efficiency has two regimes

A critic is not free. PPO trains a second network alongside the policy, which increases compute and memory.

At small compute budgets, the critic substitutes for sampling. In math, heavily bootstrapped PPO sits on the Pareto frontier at small batch sizes and compute budgets. In terminal agents, PPO with α=0.5 at B=128 matches GRPO at B=512 with about a quarter of the estimated H100 GPU-hours (91 versus 342). The critic's extra training cost is more than offset by the rollouts it saves.

At large compute budgets, the critic's returns diminish. In math, GRPO at B=1,024 gets the highest test score with less compute than any run with a larger batch size. In terminal agents at B=2,048, GRPO matches both PPO variants with fewer GPU-hours.

Why this happens

Why should batch size change which method wins? The standard way to analyze a gradient estimator is the bias–variance decomposition of its mean squared error (MSE): squared bias plus a variance term that shrinks as 1/B. But MSE counts errors in both the direction and the magnitude of the gradient, while popular optimizers like AdamW and Muon make updates largely invariant to gradient magnitude. What matters is whether the batch gradient points in the right direction, so we study the bias–variance tradeoff in terms of gradient direction.

We can describe each advantage estimator with two numbers:

• Alignment c, the cosine similarity between the estimator's expected gradient and the true policy gradient. This is the directional counterpart of bias. In our control experiments, GRPO’s expected gradient points along the true gradient, while bootstrapping from an imperfect critic can tilt PPO’s away from it.
• Noise scale ν, the variance of a single-trajectory gradient relative to the squared norm of the expected gradient. This is the directional counterpart of variance, and it extends the gradient noise scale of McCandlish et al. [9] to biased estimators.

With batch size B, the expected cosine similarity between the batch gradient and the true gradient is approximately

When B is small relative to ν, noise dominates and the method with the lower noise scale generally wins. As B grows, the noise term ν/B shrinks, and the expected cosine approaches c. More sampling can't raise it further, so c acts as a ceiling. That is GRPO's regime. If PPO has the better small-batch tradeoff but GRPO has higher alignment, we can predict a crossover batch size from c and ν.

We show this across control tasks, where we can estimate the true gradient from about 2.1 million trajectories per checkpoint. The formula predicts measured cosine similarity with a median absolute error of 0.003 in HalfCheetah and 0.015 in Pendulum.

Discussion

Tasks are getting longer, but the signal per trajectory is not growing. LLM tasks have grown from single-turn math answers to agentic trajectories spanning hundreds of turns and hundreds of thousands of tokens. Under the current critic-free paradigm, each of these trajectories still ends in a single outcome reward, often a pass or fail. With binary outcome rewards, a 1,000-turn agent trajectory receives the same single pass-or-fail signal as a one-line math answer. Its gradient is far noisier because of how many actions share that one reward. Our results show the cost of this mismatch: the longer and more complex the task, the larger the batch size GRPO needs to catch up. A critic is one promising way to get more signal out of each trajectory, by assigning credit to individual steps rather than spreading one reward across all of them.

As rollouts get more expensive, getting more out of each one matters more. Rollout generation already dominates RL compute budgets. ScaleRL, for example, devotes 80% of its GPUs to generating rollouts and only 20% to policy updates [10]. The main reason is that generating tokens one at a time is memory-bandwidth bound, so it uses GPUs far less efficiently than compute-bound training [11]. Agentic rollouts can also incur costs from sandboxes, tool and API calls, and LLM judges that grade each rollout. In scientific discovery or robotics, a single rollout can mean running a real experiment or moving a real robot. When rollouts and rewards are this costly and updates are comparatively cheap, it is worth investing more compute at training time to learn more from each trajectory. Critics are a natural candidate, and our results show they already pay for themselves at small budgets. Even accounting for the critic's own training compute, PPO matches GRPO with fewer GPU-hours at small compute budgets.

We recommend that practitioners treat batch size and critic reliance as a joint choice based on their target task and compute budget. We use one starting model per LLM setting and haven't yet studied how actor and critic sizes affect the crossover points. As trajectories grow longer and more expensive, this choice will only matter more.

Read the full paper: https://labs.scale.com/papers/to-critic-or-not-to-critic

References

[1] Schulman et al., “[Proximal Policy Optimization Algorithms](https://arxiv.org/abs/1707.06347),” 2017.

[2] Schulman et al., “[High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://arxiv.org/abs/1506.02438),” ICLR 2016.

[3] Guo et al., “[DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://arxiv.org/abs/2501.12948),” 2025.

[4] Shao et al., “[DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://arxiv.org/abs/2402.03300),” 2024.

[5] Yue et al., “[VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks](https://arxiv.org/abs/2504.05118),” 2025.

[6] Hou et al., “[Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning](https://arxiv.org/abs/2607.07508),” 2026.

[7] Ivison et al., “[Tmax: A Simple Recipe for Terminal Agents](https://arxiv.org/abs/2606.23321),” 2026.

[8] Zhao et al., “[Small Leak Can Sink a Great Ship: Boost RL Training on MoE with IcePop!](https://ringtech.notion.site/icepop),” 2025.

[9] McCandlish et al., “[An Empirical Model of Large-Batch Training](https://arxiv.org/abs/1812.06162),” 2018.

[10] Khatri et al., “[The Art of Scaling Reinforcement Learning Compute for LLMs](https://arxiv.org/abs/2510.13786),” 2025.

[11] Austin et al., “[How to Scale Your Model](https://jax-ml.github.io/scaling-book/inference/),” Google DeepMind, 2025.