CliniCARE-Bench: Clinical AI Agents Can Be Right for the Wrong Reasons

We evaluated 16 agentic systems on 25 clinical care scenarios over 750 real patient cases. Every single system committed to an answer more often than the evidence allowed, and up to one in five correct verdicts rested on an investigation the case authors had explicitly prohibited.

Foundation models can match or beat expert scores on medical knowledge exams. But a production clinical agent has to do something different and much harder than answer a question from a clean prompt. It may need to search a longitudinal record, reconcile structured data with free-text notes, apply the right clinical or regulatory standard, decide whether the available evidence is sufficient, and produce a conclusion that another reviewer can audit.

For a healthcare organization, that gap is the whole question. A model that aces medical exams is not one you can point at your patients' charts. Value comes down to two things: which tasks the agent can do reliably, and how carefully you wire the agent into your workflow, with governed data access, your own protocols enforced, evidence requirements, and knowing when to defer to a clinician. That is an evaluation problem before it is a deployment one. Test a model against your workflows, find the work it can own, and put it there to augment clinicians, not replace them.

CliniCARE-Bench (Clinical Calibrated Audit of Medical Reasoning in EHR) is a benchmark for that kind of work. It has 25 clinician-authored scenarios, each turned into 30 patient-specific cases from real MIMIC-IV records, for 750 cases total. An agent investigates each one inside a governed, fully logged tool environment and returns one of four verdicts: Yes, No, Indeterminate: Lack of Data, or Indeterminate: Medically Ambiguous.

Results: Across 16 agentic systems, four-way accuracy tops out at 76.1%, but raw accuracy gives the models too much credit. When we credit a verdict only when it's correct and the agent broke none of the prohibited shortcuts our expert Clinical Board defined, scores drop by 4.8 to 14.8 percentage points. The leaderboard reorders too. A right answer, on its own, is not evidence that a defensible investigation produced it.

A confident answer can hide an investigation that never happened

Take one case. A patient with cirrhosis is admitted with hepatic encephalopathy, the confusion that sets in when a failing liver lets toxins build up. It almost always has a trigger, and treating the trigger is how you treat the episode. So the question has two halves: is a trigger present in the record, and were the candidates this presentation calls for actually checked?

The systems found a trigger. The patient had stopped taking the medication that clears the toxins, and arrived with deranged electrolytes and injured kidneys. Real evidence, correctly retrieved.

But this patient also has ascites, fluid pooled in the abdomen. In a cirrhotic patient who turns confused, that fluid has to be sampled with a needle to check it for infection, the most dangerous trigger on the list. No one ever drew the sample, and the admission is closed. That candidate can never be ruled in or out, so the precipitant was never established, and the defensible verdict is Indeterminate: Lack of Data.

Thirteen of sixteen systems commit to a definitive label anyway, and twelve of them commit to "Yes", at confidences as high as 100%. Seven ruled out infection on the strength of blood cultures, urine cultures and a chest X-ray and let that stand in for the one infection nobody had tested for. The verdict looks decisive. The work behind it is not.

A thin investigation and a wrong answer is the obvious kind of failure. A subtler failure, just as common, is when the system takes the same shortcut and lands on the right verdict anyway. Grade only the answer, and a lucky guess scores exactly like a careful work-up. CliniCARE-Bench grades both. Every retrieval, computation, and policy lookup is logged and replayable, so we can inspect the investigation itself, not just the final report. Once you score the process, the leaderboard moves.

How it works

We don't hand agents a tidy, pre-assembled context. They get a reproducible runtime built on MIMIC-IV, covering 364,627 patients and about 900 million structured rows. What keeps it a genuine investigation rather than a database lookup is how the environment is built.

Patient-scoped tools, not raw data. Agents pull one patient's chart through bounded, clinician-verified tools built around clinical entities like labs, vitals, notes, medications, and orders. When those tools fall short, a capped, read-only SQL fallback fills the gap, logged separately so we can tell when an agent drops beneath the intended interface. A sandboxed shell (pandas, numpy, scipy) is there for computation, turning retrieved values into the derived quantities adjudication needs, like a severity score or a trend.

Heterogeneity is kept intact. The data isn't pre-joined or pre-summarized. The agent has to reconcile structured events against free-text notes and rebuild the timeline itself.

Policy is a part of the judgment. Many cases hinge on an external standard like KDIGO or CMS SEP-1. Agents consult a fixed policy corpus through an interface that keeps provenance, so any policy claim traces back to its source. An agent can get the patient data exactly right, apply the wrong rule, and still land on the wrong verdict.

The two Indeterminate classes carry weight. They split missing evidence from genuine ambiguity, where the evidence is all present but still supports more than one defensible read. Abstention lives inside the standard, not in a threshold bolted on afterward.

Reference verdicts come from an ensemble of three frontier harnesses from different families, calibrated against blinded Clinical Board review. On the calibration sample, the reference matched independent clinician judgment on 91% of defined-label cases (κ = 0.87), landing closer to each clinician than the two clinicians landed to each other.

What we found

Nobody clears 76.1%. The top system still gets almost a quarter of its cases wrong, and the field runs down to 65.3%. Macro-F1 lands below accuracy for every system, because the two Indeterminate classes are where they all struggle most.

Every system cuts corners, and it reorders the board. Require a clean investigation and each system loses ground, 4.8 to 14.8 points, which means each one reaches some correct verdicts through prohibited shortcuts. The drops aren't uniform, so the ranking changes too. Gemini-3.1-Pro posts 72.7% raw, right with the leaders, then falls to 57.9% once shortcut answers come out, below every Claude and Codex system but one. On the worst-hit system, roughly one correct verdict in five was reached by breaking a rule.

Every system under-abstains. Over-commitment, giving a firm verdict when the record calls for deferral, runs 21.3% to 55.3%. Over-abstention runs only 7.3% to 16.5%. It holds for all sixteen systems and repeats on a rebalanced set, so it's a property of the models, not the label mix. And when systems do abstain, they often abstain on cases the record settles cleanly.

Confidence separates them better than accuracy does. Calibration error spans 0.046 to 0.262, almost sixfold, and it doesn't rank systems the way accuracy does. For a clinician who can't cheaply redo the work, a confidence that doesn't track correctness is worse than a couple of lost points.

Right answer, wrong receipts. Systems reach the correct verdict without citing the evidence for it. Citation precision tracks the harness, not the model, and barely moves from a small model to a frontier one. Whether a conclusion is traceable is set by the scaffold. Whether it's correct is set by the model.

More compute buys neither. A tenfold spread in cost per run maps to about a ten-point accuracy range, and pushing reasoning effort from low to high moves accuracy 0.7 percentage points, inside the noise. The bottleneck is finding the right evidence, not thinking longer.

Re-running won't catch the error. Over three attempts, pass^k lands 8.3 to 12.6 percentage points below avg@k, mostly because a system keeps agreeing with its own wrong answer. These models are stably wrong, not randomly wrong.

Why it matters

A correct final answer doesn't tell you that an agent ran a defensible investigation to get there. In clinical work, where a medical professional has to review what the AI did, the investigation is the product. A documented, grounded, hedged work-up can be audited. A bare verdict can't.

CliniCARE-Bench scores all of it in one framework: investigation, evidence grounding, policy use, process, and calibrated abstention. The recipe isn't specific to medicine. Governed tool access, explicit evidence requirements, and scoring process apart from outcome are what any auditable high-stakes agent needs, in finance and law as much as the clinic.

Code available here: https://github.com/scaleapi/clinicare
The patient-linked artifacts will be hosted soon by Pacific AI as a part of MedHELM.

For details on the system, framework, and full benchmark results, see our paper on arXiv: https://arxiv.org/abs/2608.07796