SWE Atlas

Update November 2026

SWE Atlas v1.1 is live. It combines the three category leaderboards into a single, easy-to-run index covering the same 284 tasks as v1.0. Every model was re-run with a new judge (GPT-6.1 Sol, majority of 3 votes), refined grading criteria and restricted network access, so v1.1 scores aren’t directly comparable to v1.0.

Overview

SWE Atlas evaluates coding agents on professional software engineering work beyond issue resolution, across three complementary categories. Tasks come from 15 production repositories in 6 languages (Go, Python, TypeScript, C, C++ and JavaScript), and the agent works in each repository inside a Docker container with standard shell tools, so it can build, run and test the code. Every task is single-turn: the prompt states the goal and leaves the approach to the agent. For the full methodology, see the paper: SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution.

Score vs. cost

Results by Category

Acknowledgements

Thanks to Artificial Analysis and Inductive ML for their contributions to the v1.1 release. Artificial Analysis’s SWE-Atlas-QnA-AA dataset provided wording for several updated Codebase QnA rubric criteria, and Inductive ML’s reward-hacking feedback further hardened the task environments.