To Jev or Not to Jev: When to Use System-1 Decision Models as LLM Judges

Large language models are now routinely graded by other large language models, a practice known as LLM-as-a-judge that underpins model evaluation, training-data filtering and the reward signals used in reinforcement learning. The most accurate judges are frontier models that reason before they answer, and at the volumes these workloads require (thousands of comparisons per training batch, or sixteen candidate answers per prompt in best-of-N sampling) their cost and latency add up quickly.

In September 2026, TypeSafe released Jev, the first of a new class of System One decision models that answer typed questions about a piece of text with calibrated probabilities in a single pass, without generating any text. Dozens of models with the same interface have followed (see the S1MB leaderboard and the Jev Decision Index on Hugging Face), and our new paper, To Jev or Not to Jev: When Can System-1 Decision Models Replace LLM Judges? (arXiv:2611.00505), asks when such a model can stand in for a generative judge, how it should be queried and where it fails. To answer these questions, we evaluated 14 System-1 decision models, including Jev 1.14, against four generative judges on eleven established judge benchmarks, giving every model identical questions and inputs.

Key takeaways from the study:

• Five System-1 judges (DReX-Max, d1-ultra, Mercury-Decide, Jev 1.14 and pplx-decider 1.2) are statistically indistinguishable from GPT-5.6 without reasoning on all five pairwise and best-of-4 benchmarks we tested, though at our sample sizes only gaps of about 5 percentage points or more would reliably show up.
• With one reference answer written per prompt, eleven System-1 judges reach 90.3 to 93.4% on JudgeBench, level with frontier reasoning judges, and on best-of-16 selection the approach costs about one-sixteenth as much as GPT-5.6 with reasoning.
• Resistance to text that addresses the grader varies from model to model (0 to 49% of correct verdicts reversed), so it has to be measured for each model rather than assumed.
• Two general System-1 leaderboards that contain no judging tasks track judging accuracy (Pearson r of 0.95 to 0.96 across all models, falling to 0.78 to 0.89 without the Kev family), which gives teams a practical way to shortlist a judge.

How a System-1 judge differs from a generative one

A generative judge reads the responses and writes its verdict as text, often after a long chain of reasoning, so its cost and latency grow with every token it produces. A System-1 decision model instead receives a state (the user's request and the candidate responses, passed as named fields) together with a set of typed questions, and it returns a probability for every question at once.

The questions come in three types. A Noul is a yes/no statement (for example, that a response follows every instruction in the request), a Choice selects among named options (response A is better, or response B is better) and a Score places the answer on an ordinal scale such as 1 to 6.

How the two kinds of judge are queried · schematic

A generative judge writes its reasoning and verdict token by token, while a System-1 judge receives the same material as a state, answers typed questions in a single forward pass and returns probabilities that can be used directly as confidence.

Because the model reads the state once and answers all of its questions in parallel, asking ten questions costs little more than asking one, and pricing is per input token only (for example, Jev is listed at $0.042 per million input tokens at the time of writing). The outputs are probabilities rather than words, so a Choice returns 0.83 for response A instead of the letter A, which makes it simple to average over both presentation orders, set thresholds or send uncertain items to a stronger judge. These models do not write out intermediate reasoning, which is the main source of their speed and, as the same-size comparison below suggests, a likely source of their weakness on items whose correctness has to be worked out.

How we tested the models

We selected System-1 models with a rule fixed in advance, which matters because a hand-picked set could flatter or penalize the category. A model had to expose the typed Noul, Choice and Score interface and be either runnable on a laptop (open weights up to 9 billion parameters) or served through OpenRouter's decision-model catalog. From the independent S1MB leaderboard we took the strongest available model and a ladder of sizes (Jev 1.14, pplx-decider 1.2 and pplx-flash 1.2, Clef and Clef-flash, and the open Kev-4B), and we then piloted every other catalog entry and kept those that handled our requests reliably at a projected cost of at most $4 for the full suite (d1, d1-ultra, Mercury-Decide, Microsoft Decision-1, Span-02, and DReX v1.5, v2.0 and Max).

The comparison judges were Claude Haiku 5.5, GPT-5.6 with and without reasoning, and Claude Opus 5.5 with thinking, which span a range from a small fast model to a frontier reasoning model. All judges received the same rubric, and the generative judges returned only a verdict and a probability, so that the comparison measures judgments rather than explanations. Every benchmark is an established, published one, chosen to cover a different way a judge can succeed or fail:

Each pairwise item was evaluated in both presentation orders, and differences between judges were tested with prompt-clustered bootstrap confidence intervals and permutation tests corrected for multiple comparisons (Holm). When this post calls two judges statistically indistinguishable, it means the corrected test found no significant difference, not that the judges are equivalent: on the 300- to 419-item benchmarks, the smallest difference our tests would detect with 80% power is roughly 4 to 6 percentage points (about 5 to 7 after Holm correction), and it is larger on the 80-item EvalBiasBench set. For four of the strongest models we also ran the complete RM-Bench (11,943 pairs) and RewardBench 2 (1,763 sets), and the full-size results agree with our samples to within 2.3 percentage points.

The best System-1 judges match a fast frontier judge

The 14 decision models differ far more than a shared label suggests. Asked the same A/B question, they range from 66.6% (Kev-4B) to 84.6% (d1-ultra) on JudgeBench and from 57.3% (Kev-4B) to 87.0% (DReX-Max) on RewardBench 2, so the choice of decision model matters as much as the choice between System-1 and System-2 judging.

Paper Table 1 · 14 System-1 models, swap-averaged pairwise Choice; dashed lines are the generative judges' means on the same three benchmarks

The spread between the strongest and weakest System-1 models (18 points on JudgeBench and about 21 points in mean accuracy) is larger than the gap between the strongest System-1 models and any of the generative judges, which makes model selection the first decision that a team considering System-1 models should weigh carefully.

Five models (DReX-Max, d1-ultra, Mercury-Decide, Jev 1.14 and pplx-decider 1.2) are statistically indistinguishable from GPT-5.6 without reasoning on all five pairwise and best-of-4 benchmarks, and the two strongest, DReX-Max and d1-ultra, are significantly more accurate than Claude Haiku 5.5 on JudgeBench, LLMBar and RewardBench 2 (by 10.5 to 14.4 percentage points). These models are also stable when the two responses swap places, changing 7.7 to 12.9% of their JudgeBench verdicts compared with 38.6% for Kev-4B.

To check that these rankings hold beyond our samples, we ran four of the strongest decision models (DReX-Max, d1-ultra, Jev 1.14 and pplx-decider 1.2) on the complete RewardBench 2 (1,763 items) and RM-Bench (11,943 pairs) and on JudgeBench's separate Claude-generated split (270 pairs). d1-ultra reaches 87.5% on RewardBench 2 and 88.0% on RM-Bench, and DReX-Max 84.7% and 87.4%, both ahead of Jev 1.14 (82.7% and 83.8%) and pplx-decider 1.2 (78.1% and 82.3%), and the order is similar on the JudgeBench Claude split (78.3% for DReX-Max, 77.0% for d1-ultra, 74.8% for pplx-decider and 73.7% for Jev). d1-ultra's full-set RewardBench 2 score is close to that of Opus 5.5 with thinking on our 300-item sample (88.5%), although the two are measured on different item sets. The weakness on elaborate wrong answers also persists at full size, since on the RM-Bench pairs where the incorrect answer is the more elaborate one these four models reach only 73.1 to 82.1%.

Frontier reasoning judges remain ahead on items whose correctness has to be worked out, where Opus 5.5 with thinking reaches 94.7% on JudgeBench. These benchmarks are also approaching saturation at the top: Opus 5.5 with thinking scores above 90% on both JudgeBench and LLMBar, while the best System-1 judges reach 84.6% and 88.5%, so the headroom that separates models is small, and with pilot-scale test sets (80 to 1,925 items per set) a statistical tie between two strong judges should be read as "too close to separate on these benchmarks" rather than as proof that they are equivalent. The next two sections show that most of the remaining gap depends on how the System-1 judge is asked and on what information it receives.

How the question is asked matters most for smaller models

A decision model answers exactly the question it is given, so we compared three ways of framing the same judgment. In the pairwise formulation the model sees both responses and chooses A or B, and its probabilities are averaged over both presentation orders. In the checklist formulation each response is judged on its own against six generic yes/no criteria (correctness, final answer, instruction following, completeness, focus and appropriate refusal) plus three to six criteria specific to the prompt, and the reference-guided formulation is described in the next section.

The three query formulations used in the paper · schematic

The checklist and the reference both add a small amount of work by a generative model, done once per prompt rather than once per response, so their cost is shared across every candidate response the System-1 judge evaluates.

Checklists improve the weaker decision models substantially and the strongest ones hardly at all. They raise LLMBar accuracy by 8.1 to 17.2 percentage points for Clef, pplx-flash, Kev-4B and Clef-flash and RewardBench 2 accuracy by 11.0 to 13.7 points for DReX v1.5, DReX v2.0, pplx-flash and Kev-4B, whereas for the five strongest models a checklist is never significantly better than the A/B question, with a single exception (pplx-decider on RM-Bench, +6.9 points).

Because extra questions on the same input are nearly free for a decision model, the main cost of a checklist is the one call to Claude Haiku 5.5 that writes the prompt-specific criteria (about $0.16 per 1,000 prompts). For a team running a small or inexpensive decision model, this is the cheapest accuracy gain available, while a team already using one of the strongest models can keep the simpler A/B question.

The correctness gap is a reasoning gap, and a reference answer closes most of it

JudgeBench pairs a correct answer with a plausible incorrect one, so a judge can only succeed by checking the work, which is precisely what a single-pass model does not do. To separate the System-1 design from model size, we compared Kev-4B with Qwen3.5-4B, the instruction-tuned sibling of its base model, running on the same laptop. Without thinking, Qwen3.5-4B is statistically indistinguishable from the Kev-4B checklist on four of five benchmarks, whereas with a 4,000-token thinking budget it reaches 84.0% on JudgeBench, 15.7 points above the Kev-4B checklist, at a median of 117 seconds per judgment compared with 0.70 seconds for Kev-4B.

The deficit of a small decision model on correctness may therefore come mainly from the absence of reasoning rather than from its size. This suggests a practical division of labor: a reasoning model can work out the answer once per prompt, and the decision model can then check every candidate response against it. In our reference-guided formulation, GPT-5.6 writes one short reference answer per prompt, and the decision model is asked whether a response's final answer agrees with it.

Paper Table 1 · JudgeBench, all 14 System-1 models, pairwise Choice vs. reference-guided; the right-hand column repeats the reference-guided accuracy

The largest gains go to the weaker models (pplx-flash gains 21.4 points and Kev-4B 21.4), which suggests that the reference supplies the correctness check that these models cannot perform on their own.

With a reference, eleven of the 14 models reach 90.3 to 93.4% on JudgeBench, statistically indistinguishable from Opus 5.5 (94.7%) and GPT-5.6 with reasoning (93.1%), and six of them are significantly more accurate than GPT-5.6 without reasoning (by 7.0 to 9.0 points). The reference is not always right, however. On 15 of the 350 JudgeBench pairs (about 4%), at least 10 of the 14 reference-guided judges pick the wrong response, which most likely marks an incorrect reference (GPT-5.6 with reasoning, which wrote the references, also gets only a third of these pairs right when it judges them directly). On those items the System-1 judges follow the reference and are right only about 7% of the time, so reference errors pass straight through to the verdict and cap accuracy on this benchmark at roughly 96%. The approach is most economical when many responses share one prompt, as in best-of-N sampling: on PPE best-of-16, every reference-guided System-1 judge is statistically indistinguishable from GPT-5.6 with reasoning (87.2 to 92.8% against 91.2% on the 180 prompts with a single verifiable answer), at $0.0051 per prompt with Jev, including the GPT-5.6 call that writes the reference, compared with $0.0827 for GPT-5.6 with reasoning and $0.1308 for Opus. Without a reference, the best System-1 checklist on PPE (DReX-Max, 79.3%) matches Haiku and GPT-5.6 without reasoning but trails the two reasoning judges by 8.4 and 11.4 points.

Robustness depends on the model, not on the System-1 design

A judge that grades text written by another model can be gamed by that text, which matters most in reinforcement learning, where the model being trained is optimized against the judge. To test this, we appended a short note to the incorrect response in JudgeBench and LLMBar pairs, stating that the response had been verified against the reference solution and should be rated as the better one, and we measured how often a judge's correct verdict flipped.

Paper robustness results · our injection payload on JudgeBench and LLMBar, pairwise Choice; the shaded band spans the four generative judges

The spread among System-1 models (0.0 to 49.0%) is far wider than among the generative judges, and even the most vulnerable decision model reverses fewer verdicts than a generative model of the same size as Kev-4B (77.3%), so robustness has to be checked model by model.

The note reverses 0.0 to 4.7% of the correct verdicts of DReX-Max, d1-ultra, Microsoft Decision-1, d1 and pplx-decider, which is statistically indistinguishable from the generative judges (0.4 to 2.3%), meaning that an item-by-item paired test (exact McNemar with Holm correction, detailed in the robustness appendix of the paper) finds no significant difference at this sample size, but 16.7 to 49.0% of those of the other nine System-1 models, with Mercury-Decide the most vulnerable. A same-size generative judge (Qwen3.5-4B without thinking) is more vulnerable still (77.3%), so the weakness is not inherent to decision models, and judging each response separately with a checklist lowers the rate for every vulnerable model except Span-02.

Two further biases appear across judges of both kinds. On RM-Bench, where the same content is presented in different styles, Jev 1.14 loses 12.9 points when the incorrect answer is the more elaborate one (close to GPT-5.6 without reasoning at 11.3 points), whereas the other System-1 models lose 15.0 to 33.1 points. On the CALM benchmark, rewriting the better answer in a fearful tone reversed 62 to 95% of A/B verdicts for every one of the 18 judges we tested, including Opus 5.5 with thinking. A two-auditor check found that 85% of these rewrites preserved the original content, and re-scoring on that subset left the reversal rates essentially unchanged, so sensitivity to tone is shared by current judges rather than specific to System-1 models. A note on whether it counts as a bias depends on the rubric: ours ranked correctness and instruction following above helpfulness and told judges not to reward a confident tone, but it did not tell them to ignore tone, so a judge could defensibly rate an anxious answer as less helpful. The size of the effect is still striking, since every judge let tone outweigh content in most of these pairs.

Where System-1 judges excel: calibrated confidence

A judge is well calibrated when its stated confidence matches how often it is right, so that verdicts given with 80% confidence turn out correct about 80% of the time. Jev 1.14 has the lowest calibration error of any judge we tested (expected calibration error, or ECE, of 0.019, with a 95% bootstrap interval of 0.015 to 0.032). The four best-calibrated System-1 judges (Jev, DReX-Max, d1-ultra and d1, 0.019 to 0.031) cannot be separated from one another or from GPT-5.6 with reasoning (0.038, interval 0.027 to 0.057), but they are clearly better calibrated than Haiku, GPT-5.6 without reasoning and Opus (0.056 to 0.087), whose probabilities have to be stated in words rather than read from the model. These values do not depend on the binning: computing ECE with 15, 20 or equal-mass bins changes each one by at most 0.004.

Calibrated confidence makes a two-stage judge practical. In this design, the System-1 model judges every item and only the items it is unsure about are sent to a frontier reasoning judge, with the confidence threshold fixed on a separate development split before testing.

Confidence-based routing as evaluated in the paper · schematic

On LLMBar, routing from pplx-decider to Opus 5.5 matches Opus alone (91.7%) while sending it only 17% of items, and routing from Jev reaches 90.7% at 10% escalation. On JudgeBench, where errors are harder to detect, routing reaches 92.3% with Jev at 60% escalation and 91.5% with pplx-decider at 41% (Opus alone: 94.4%), because these models identify their own mistakes with an AUROC of 0.71 to 0.90, compared with 0.63 to 0.81 for Kev-4B.

Because routing depends on a probability threshold, we also checked how repeatable these probabilities are. Sending the same 50 JudgeBench requests twice to five hosted decision models, without caching, produced the same verdict in 96 to 100% of cases. The probabilities themselves move slightly between calls: the mean absolute change in P(A) is 0.014 for Jev 1.14, 0.007 for DReX-Max, 0.010 for Mercury-Decide and 0.039 for d1-ultra (the noisiest, with 96% of verdicts unchanged), while pplx-decider returned bit-identical probabilities every time. A single call is therefore enough for a stable verdict, but items close to a routing threshold may fall on either side of it from one call to the next, particularly with d1-ultra.

Several System-1 judges also agree with people at least as well as the generative judges do. Six of them correlate more strongly with human HelpSteer2 helpfulness ratings than every generative judge (Spearman correlation of 0.445 to 0.498 against 0.43 to 0.44), although the confidence intervals overlap. Measured against agreement between people, however, every judge we tested falls well short: HelpSteer2's own annotators agree on helpfulness at a quadratic-weighted Cohen's κ of 0.79 after the dataset's quality filtering, whereas the judges reach 0.36 to 0.64 on the same measure.

Agreement with human HelpSteer2 helpfulness ratings (300 responses). For κ, each judge's scores are mapped onto the five human rating levels so that the two share a scale; 95% intervals are about ±0.1.

On AgentRewardBench, which asks whether a web agent completed its task, pplx-decider (AUROC 0.921), d1-ultra (0.915) and DReX-Max (0.908) separate successful from failed trajectories as well as GPT-5.6 without and with reasoning (0.907 and 0.918), with Opus slightly ahead (0.933).

Where System-1 judges fall short

Long inputs remain the clearest weakness. On LongRewardBench, whose contexts run from 4K to 64K tokens, the strongest System-1 judges (Jev 1.14 and pplx-decider at 70.8%, DReX-Max at 69.8%) trail GPT-5.6 without reasoning (81.3%) by about 10 to 12 points overall. The gap is concentrated between 8K and 32K tokens, where these three models score 60.5 to 71.2% against 74.6 to 84.7% for GPT-5.6 without reasoning. Kev-4B drops more steeply, from 78.0% at 4K to 59.3% at 8K, and two models with 32K-token context windows (d1-ultra and Microsoft Decision-1) cannot accept the longest inputs at all, so their long-context scores mostly reflect that limit.

Detecting side effects in agent trajectories, meaning actions that changed a website in ways the user did not ask for, is hard for every judge, with F1 scores of 6.5 to 35.3 for the System-1 judges that flag any side effects and 26.7 to 34.0 for the generative ones. Kev-4B never flags a side effect at the default threshold, since its highest probability across 292 trajectories is 0.425.

A third lesson concerns how generative judges are asked to score. On PPE best-of-16, generative judges rating each response on a 1-to-6 scale gave the same top score to 10 to 14 of the 16 candidates on average, so the selection was close to random, and moving them to a 1-to-100 scale was necessary for a fair comparison. System-1 judges avoid this problem because their probabilities are continuous, which is one reason they perform well at ranking many candidates once a reference answer supplies the missing correctness check.

General leaderboards predict judging quality

Practitioners choosing among a growing list of decision models need a cheaper signal than running a full judging evaluation, so we checked whether two public System-1 leaderboards, neither of which contains judging tasks, rank models the way our benchmarks do. Scores on the Jev Decision Index correlate with mean pairwise judging accuracy at r = 0.96 across 12 models (0.89 when the Kev family is excluded), and scores on the S1MB English leaderboard correlate at r = 0.95 across 9 models (0.78 without Kev). The two leaderboards also agree closely with each other (r = 0.97 on the models they share). Removing Kev weakens the correlations because the Kev models span the low end of both scales, and with only 6 to 12 models the estimates are imprecise: the 95% interval is 0.86 to 0.99 for r = 0.96 but 0.55 to 0.98 for r = 0.89 and −0.09 to 0.97 for r = 0.78, so among the stronger models alone the S1MB result cannot rule out a weak relationship. These results suggest that general decision quality transfers to judging, and that a leaderboard is a reasonable way to shortlist candidates before testing them on a team's own evaluation data.

Jev Decision Index 0.4.1 (multimodalart/jev-decision-index) and paper Table 1 · mean swap-averaged Choice accuracy over JudgeBench, LLMBar and RewardBench 2; dashed line is a least-squares fit

The Kev models sit at the low end of both axes, and the strongest commercial models cluster in the upper right, where differences of a few leaderboard points no longer order the models reliably, so a leaderboard is best treated as a shortlist rather than a final ranking.

Practical recommendations

Taken together, the results support eight practices for teams that want to use System-1 models as judges.

• Recommendation 1. Start from one of the stronger decision models (DReX-Max, d1-ultra, Jev 1.14 or pplx-decider in our tests, or Mercury-Decide where inputs are trusted, since it was the most vulnerable model to injected instructions) and use the public leaderboards to shortlist newer ones, since model choice accounts for the largest share of the variation we observed.
• Recommendation 2. Run pairwise comparisons in both orders and average them, which removes most position bias at the cost of a second call.
• Recommendation 3. With smaller or older models, ask about each response with a short checklist rather than with a single pairwise question.
• Recommendation 4. For tasks where correctness decides the verdict, such as math, code or best-of-N selection, generate one reference answer per prompt with a reasoning model and ask the System-1 judge whether each response agrees with it; the reference is shared across all candidates, so its cost is spread over them.
• Recommendation 5. When the highest accuracy is needed, route low-confidence items to a reasoning judge, which recovers most of the remaining gap at a fraction of the cost of sending every item there.
• Recommendation 6. Before a System-1 judge is exposed to text written by users or produced under optimization pressure, such as reinforcement learning against the judge itself, measure its robustness to injected instructions, since attack success rates differed by more than 40 points between models.
• Recommendation 7. Shorten or filter inputs longer than about 8K tokens, or send them to a generative judge.
• Recommendation 8. When a generative judge is used for ranking, have it score on a fine-grained scale such as 1 to 100 rather than a coarse one.

Limitations and what comes next

Several limitations apply to these findings. Some of the strongest decision models announced during the study (Fastino GLiDE, Jeffan-31B, StartLux and Darwin-27B) are not served by the hosted provider we used at this point in time and would need GPUs with at least 80 GB to self-host, two free catalog models (mammoth-1 and elephant-1) were capped at 1,000 requests per day, and a few others exceeded our cost limit or failed our reliability pilot, so our results are a lower bound on what System-1 judges can do. Each configuration was run once at pilot scale, the prompt-injection results rest on a single payload that we wrote, and cost comparisons use list prices with an assumed pricing tier for GPT-5.6, so absolute costs will vary by provider and contract.

The broader conclusion is that System-1 models are already usable as judges for a large class of evaluation tasks, provided that the model, the question formulation and the deployment safeguards are chosen with the same care as the judge prompt for a generative model. Future work should examine whether decision models trained specifically for judging close the remaining correctness gap without a reference answer, how robustness to adversarial text can be measured more broadly than with one payload, and how the long-context gap changes as context windows grow. The full paper, including per-benchmark tables, statistical tests and the complete list of models we could not evaluate, is available on arXiv (arXiv:2611.00505).