To Critic Or Not To Critic
Jason Wei, George Pu, Jingxuan Fan, Yuan (Emily) Xue, Yuan (Christy) Li
As agentic tasks become more complex and rollouts become longer and more expensive, improving the efficiency of reinforcement learning becomes increasingly important. Critics can reduce gradient variance, allowing models to learn from fewer trajectories, but they also introduce additional compute costs and potential bias. Understanding when a critic should be applied remains an important open question.
Abstract
As large language models tackle longer and more complex tasks, understanding when a critic improves the efficiency of reinforcement learning (RL) becomes increasingly important. Proximal Policy Optimization (PPO) trains a critic to reduce variance and provide fine-grained credit assign- ment, while Group Relative Policy Optimization (GRPO) drops the critic and avoids the bias of bootstrapping from the critic’s value estimates. Although recent critic-based methods have reported strong performance relative to GRPO, it remains unclear when each approach is preferable. We argue that there is no universal answer; instead, the choice depends on several interacting factors, including batch size, critic reliance, and task complexity. We study the bias–variance tradeoff in terms of gradient direction, measuring how closely each method’s gradient estimates align with the true policy gradient as batch size grows. Under a shared actor objective, we compare GRPO with PPO in control tasks, short-horizon math, and long-horizon terminal agents. Across more than 1,200 training runs, including 96 with LLMs, we find that PPO generally leads at small batch sizes, while GRPO often overtakes heavily bootstrapped PPO and reaches parity with lightly bootstrapped PPO at larger batches. At longer horizons and in more complex domains, GRPO needs larger batches to catch up to PPO’s performance. Accounting for training compute, critics pay off at small budgets, but their returns diminish at larger budgets.