[Papers]

Research papers and publications from Scale Labs covering AI evaluation, safety, benchmarking, and frontier model analysis.

Date Title Category

Date Title

9/4/2026READY or Not: Reliable Enterprise Agent DeploymentVeronica Chatrath, Bryan Zhu, Jingxuan Fan, George Pu, Soham Dinesh Tiwari, Soham Dan, Ryan Young, Yuan (Christy) Li, Yuang Yao, Apaar Shanker, Minglai Yang , Daniel Yue Zhang, Yunzhong He, Ying Liu, Chenguang Wang, Zhijun Yin, Yuan (Emily) XueAgents, Enterprise

8/20/2026CliniCARE-Bench: Clinical Calibrated Audit of Medical Reasoning in EHRVeronica Chatrath, Bryan Zhu, George Pu, Jingxuan Fan, Apaar Shanker, Varun Ursekar, Anahita Sharma, Jason Qin, Keqi Han, Soham Dinesh Tiwari, Soham Dan, Vijay Kalmath, Yuan (Christy) Li, Daniel Yue Zhang, Chenguang Wang, Zainab Doctor, Zhijun Yin, Nigam H. Shah, Yuan XueEvaluation and Alignment

8/6/2026HarnessOpt-Bench: Evaluating LLMs at Harness OptimizationVarun Ursekar, Apaar Shanker, Yash Maurya, Shehab Yasser, Vijay Kalmath, Veronica Chatrath, Yuan (Emily) XueAgents, Enterprise, Evaluation and Alignment

6/30/2026DrugDiscoveryBench: Can Coding Agents Assist Early-Stage Drug Discovery?Afra Feyza Akyürek, Xinming Tu, Alec Gutmanstein, Jason Qin, Divyansh Agarwal, Sofia Monasdotter, Sergey Chekhov, Brenda Hernandez Villegas, Kirill Chugunov, Judah Engel, Veronica Chatrath, Oscar Kavanagh, Geobio Boo, Ernesto Hernandez, Ying Liu, Yuan (Emily) Xue, Aakash Sabharwal, Daniel Yue Zhang, Zainab Doctor, Yuanhao Qu, Yunzhong He, Sami HassaanAgents, Enterprise, Evaluation and Alignment

6/29/2026SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding SessionsMohit Raghavendra, Anisha Gunjal, Aakash Sabharwal, Yunzhong HeAgents, Evaluation and Alignment

6/19/2026ChainWorld: Composing Long-Horizon Desktop Workloads from Atomic OSWorld TasksVincent Siu, Manasi Sharma, Dawn Song, Daniel Yue Zhang, Chenguang WangAgents, Evaluation and Alignment

6/10/2026Rubric-Guided Self-Distillation: Post-Training Without Rubric VerifiersMohammadHossein Rezaei, Anas Mahmoud, Zihao Wang, Utkarsh Tyagi, Advait Gosai, Razvan-Gabriel Dumitru, Aakash Sabharwal, Bing Liu, Yunzhong HePost-Training

6/9/2026PSEBench: A Controllable and Verifiable Benchmark for Evaluating LLMs in Patient Safety Event TriageKeqi Han, Ryan Young, Annabel Strauss, Lindsey Hughes, Katharine M. Nesbitt, Nicole Schueler, Che Ngufor, Carl Yang, Yuan (Emily) Xue, Zhijun YinEvaluation and Alignment, Agents, Enterprise

6/4/2026Insights Generator: Systematic Corpus-Level Trace Diagnostics for LLM AgentsAkshay Manglik, Apaar Shanker, Kaustubh Deshpande, Jason Qin, Yash Maurya, Veronica Chatrath, Vijay Kalmath, Levi Lentz, Yuan (Emily) XueAgents, Evaluation and Alignment, Enterprise

5/29/2026ARCA: Adapter-Residual Credit Assignment When Token Signals DegenerateRodney Lafuente-MercadoPost-Training, Reasoning

5/19/2026Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVRUtkarsh Tyagi, Xingang Guo, MohammadHossein Rezaei, Daniel George, Anas Mahmoud, Jackson Lee, Bing Liu, Yunzhong HePost-Training, Reasoning

5/17/2026ASPI: Seeking Ambiguity Clarification Amplifies Prompt Injection Vulnerability in LLM AgentsUdari Madhushani Sehwag, Zhengyang Shan, Heming Liu, Dileepa Lakshan, Joseph Brandifino, Max FenkellSafety & Oversight, Agents, Evaluation and Alignment

5/13/2026ROK-FORTRESS: Measuring the Effect of Geopolitical Transcreation for National Security and Public SafetyMichael S. Lee, Yash Maurya, Drew Rein, Bert Herring, Jonathan Nguyen, Kyungho Song, Udari Madhushani Sehwag, Jiyeon Cho, Kaustubh Deshpande, Yeongkyun Jang, Joo Jiyeon, Minn Seok Choi, Evi Fuelle, Christina Q. Knight, Joseph Brandifino, Max Fenkell

5/12/2026Reward Hacking in Rubric-Based Reinforcement LearningAnas Mahmoud, MohammadHossein Rezaei, Zihao Wang, Anisha Gunjal, Bing Liu, Yunzhong HeSafety & Oversight, Post-Training

5/7/2026SWE Atlas: Benchmarking Coding Agents Beyond Issue ResolutionMohit Raghavendra, Soham Dan, Miguel Romero Calvo, Yannis Yiming He, Johannes Mols, Gautam Anand, Cole McCollum, Edgar Arakelyan, Vijay Bharadwaj, Andrew Park, Jeff Da, MohammadHossein Rezaei, Bing Liu, Brad Kenstler, Yunzhong HeEvaluation and Alignment, Agents

4/22/2026Coverage, Not Averages: Semantic Stratification for Trustworthy Retrieval EvaluationAndrew Klearman, Radu Revutchi, Rohin Garg, Rishav Chakravarti, Sam Denton, Yuan (Emily) XueEvaluation and Alignment, Science of Data, Enterprise

4/13/2026HiL-BENCH (Human-in-Loop Benchmark)Mohamed Elfeki, Tu Trinh, Kelvin Luu, Guangze Luo, Nathan Hunt, Ernesto Hernandez, Nandan Marwaha, Yannis Yiming He, Charles Wang, Fernando Carabedo, Alessa Castillo, Bing LiuEvaluation and Alignment, Safety & Oversight

3/12/2026Defensive Refusal Bias: How Safety Alignment Fails Cyber DefendersDavid Campbell, Neil Kale, Udari Madhushani Sehwag, Bert Herring, Nick Price, Dan Borges, Alex Levinson, Christina Q. KnightSafety & Oversight

2/26/2026LLM Novice Uplift on Dual-Use, In Silico Biology TasksChen Bo Calvin Zhang, Christina Q. Knight, Nicholas Kruus, Jason Hausenloy, Pedro Medeiros, Nathaniel Li, Aiden Kim, Yury Orlovskiy, Coleman Breen, Bryce Cai, Jasper Götting, Andrew Bo Liu, Samira Nedungadi, Paula Rodriguez, Yannis Yiming He, Mohamed Shaaban, Zifan Wang, Seth Donoughe, Julian MichaelSafety & Oversight

2/25/2026VeRO: An Evaluation Harness for Agents to Optimize AgentsVarun Ursekar, Apaar Shanker, Veronica Chatrath, Yuan (Emily) Xue, Sam DentonAgents, Post-Training, Evaluation and Alignment

2/12/2026LHAW: Controllable Underspecification for Long-Horizon TasksGeorge Pu, Michael S. Lee, Udari Madhushani Sehwag, David Lee, Bryan Zhu, Yash Maurya, Mohit Raghavendra, Yuan (Emily) Xue, Sam DentonAgents, Safety & Oversight, Evaluation and Alignment

1/15/2026SciPredict: Can LLMs Predict the Outcomes of Research Experiments in Natural Sciences?Udari Madhushani Sehwag, Elaine Lau, Haniyeh Ehsani Oskouie, Shayan Shabihi, Erich Liang, Andrea Toledo, Guillermo Mangialardi, Sergio Fonrouge, Ed-Yeremai Hernández Cardona, Paula Vergara, Utkarsh Tyagi, Chen Bo Calvin Zhang, Pavi Bhatter, Nicholas Johnson, Furong Huang, Ernesto Gabriel Hernández Montoya, Bing LiuSafety & Oversight, Evaluation and Alignment

1/6/2026Agentic Rubrics as Contextual Verifiers for SWE AgentsMohit Raghavendra, Anisha Gunjal, Bing Liu, Yunzhong HeAgents, Safety & Oversight, Evaluation and Alignment

12/22/2025MoReBench: Evaluating Procedural and Pluralistic Moral Reasoning in Language Models, More than OutcomesYu Ying Chiu, Michael S. Lee, Rachel Calcott, Brandon Handoko, Paul de Font-Reaulx, Paula Rodriguez, Chen Bo Calvin Zhang, Ziwen Han, Udari Madhushani Sehwag, Yash Maurya, Christina Q. Knight, Harry R. Lloyd, Florence Bacus, Mantas Mazeika, Bing Liu, Yejin Choi, Mitchell L Gordon, Sydney LevineReasoning, Safety & Oversight, Evaluation and Alignment

12/18/2025MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP ServersChaithanya Bandi, Razvan-Gabriel Dumitru, Ben Hertzberg, Divyansh Agarwal, Geobio Boo, Tejas Polakam, Sami Hassaan, Jeff Da, HiJae Kim, Vipul Gupta, Manasi Sharma, Andrew Park, Martin Dimakis, Ernesto Gabriel Hernández Montoya, Dan Rambado, Ivan Salazar, Rafael Cruz, MohammadHossein Rezaei, Chetan Rane, Ben Levin, Daniel Yue Zhang, Brad Kenstler, Bing LiuAgents, Reasoning, Safety & Oversight, Evaluation and Alignment

12/17/2025Audio MultiChallengeAdvait Gosai, Tyler Vuong, Utkarsh Tyagi, Steven Li, Wenjia You, Miheer Bavare, Arda Uçar, Zhongwang Fang, Brian Jang, Bing Liu, Yunzhong HeMultimodal, Safety & Oversight, Evaluation and Alignment

12/16/2025Imitation Learning for Multi-turn LM Agents via On-Policy Expert CorrectionsNiklas Lauffer, Xiang Deng, Srivatsa Kundurthy, Brad Kenstler, Jeff DaPost-Training, Agents

11/25/2025PropensityBench: Evaluating Latent Safety Risks in Large Language Models via an Agentic ApproachUdari Madhushani Sehwag, Shayan Shabihi, Alex McAvoy, Vikash Sehwag, Yuancheng Xu, Dalton Towers, Furong HuangSafety & Oversight, Evaluation and Alignment

11/13/2025Professional Reasoning BenchAfra Feyza Akyürek, Advait Gosai, Chen Bo Calvin Zhang, Vipul Gupta, Jaehwan Jeong, Anisha Gunjal, Tahseen Rabbani, Maria Mazzone, David Randolph, Mohammad Mahmoudi Meymand, Gurshaan Chattha, Paula Rodriguez, Diego Mares, Pavit Singh, Michael Liu, Subodh Chawla, Pete Cline, Lucy Ogaz, Ernesto Hernandez, Zihao Wang, Pavi Bhatter, Marcos Ayestaran, Bing Liu, Yunzhong HeSafety & Oversight, Evaluation and Alignment, Reasoning

11/10/2025ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research AgentsManasi Sharma, Chen Bo Calvin Zhang, Chaithanya Bandi, Clinton J. Wang, Ankit Aich, Huy Nghiem, Tahseen Rabbani, Ye Htet, Brian Jang, Sumana Basu, Aishwarya Balwani, Denis Peskoff, Marcos Ayestaran, Sean M. Hendryx, Brad Kenstler, Bing LiuReasoning, Safety & Oversight, Evaluation and Alignment

11/5/2025Best Practices for Biorisk Evaluations on Open-Weight Bio-Foundation ModelsBoyi Wei, Zora Che, Nathaniel Li, Udari Madhushani Sehwag, Jasper Götting, Samira Nedungadi, Julian Michael, Summer Yue, Dan Hendrycks, Peter Henderson, Zifan Wang, Seth Donoughe, Mantas MazeikaSafety & Oversight, Evaluation and Alignment

10/28/2025Remote Labor Index: Measuring AI Automation of Remote WorkMantas Mazeika, Alice Gatti, Cristina Menghini, Udari Madhushani Sehwag, Shivam Singhal, Yury Orlovskiy, Steven Basart, Manasi Sharma, Denis Peskoff, Elaine Lau, Sumana Basu, Jaehyuk Lim, Lachlan Carroll, Alice Blair, Vinaya Sivakumar, Brad Kenstler, Yuntao Ma, Julian Michael, Xiaoke Li, Oliver Ingebretsen, Aditya Mehta, Jean Mottola, John Teichmann, Kevin Yu, Zaina Shaik, Adam Khoja, Richard Ren, Jason Hausenloy, Long Phan, Connor Smith, Ye Htet, Ankit Aich, Tahseen Rabbani, Vivswan Shah, Andriy Novykov, Felix Binder, Kirill Chugunov, Luis Ramirez, Matias Geralnik, Hernán Mesura, Dean Lee, Ed-Yeremai Hernández Cardona, Annette DiamondAgents, Safety & Oversight, Evaluation and Alignment, Reasoning

10/20/2025REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable RewardsZafir Stojanovski, Oliver Stanley, Joe Sharratt, Richard Jones, Abdulhakeem Adefioye, Jean Kaddour, Andreas KöpfReasoning, Agents, Safety & Oversight, Evaluation and Alignment

10/15/2025Beyond Seeing: Evaluating Multimodal LLMs On Tool-enabled Image Perception, Transformation, and ReasoningXingang Guo, Utkarsh Tyagi, Advait Gosai, Paula Vergara, Ernesto Gabriel Hernández Montoya, Chen Bo Calvin Zhang, Bin Hu, Yunzhong He, Bing Liu, Rakshith Sharma SrinivasaSafety & Oversight, Evaluation and Alignment, Reasoning, Multimodal

10/8/2025Online Rubrics Elicitation from Pairwise ComparisonsMohammadHossein Rezaei, Robert Vacareanu, Zihao Wang, Clinton J. Wang, Bing Liu, Yunzhong He, Afra Feyza AkyürekSafety & Oversight, Evaluation and Alignment, Post-Training

9/25/2025Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-TrainingJunkai Zhang, Zihao Wang, Lin Gui, Swarnashree Mysore Sathyendra, Jaehwan Jeong, Victor Veitch, Wei Wang, Yunzhong He, Bing Liu, Lifeng JinPost-Training, Science of Data

9/23/2025Progress over Points: Reframing LM Benchmarks Around Scientific ObjectivesAlwin Jin, Sean M. Hendryx, Vaskar NathSafety & Oversight, Evaluation and Alignment

9/19/2025SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, Karmini Sampath, Maya Krishnan, Srivatsa Kundurthy, Sean M. Hendryx, Zifan Wang, Chen Bo Calvin Zhang, Noah Jacobson, Bing Liu, Brad KenstlerAgents, Safety & Oversight, Evaluation and Alignment

9/11/2025TutorBench: A Benchmark To Assess Tutoring Capabilities Of Large Language ModelsRakshith Sharma Srinivasa, Zora Che, Chen Bo Calvin Zhang, Diego Mares, Ernesto Hernandez, Jayeon Park, Dean Lee, Guillermo Mangialardi, Charmaine Ng, Ed-Yeremai Hernández Cardona, Anisha Gunjal, Yunzhong He, Bing Liu, Chen XingSafety & Oversight, Evaluation and Alignment

8/26/2025Reliable Weak-to-Strong Monitoring of LLM AgentsNeil Kale, Chen Bo Calvin Zhang, Kevin Zhu, Ankit Aich, Paula Rodriguez, Scale Red Team, Christina Q. Knight, Zifan WangSafety & Oversight, Evaluation and Alignment

8/13/2025Search-Time Data ContaminationZiwen Han, Meher Mankikar, Julian Michael, Zifan WangSafety & Oversight, Evaluation and Alignment

7/23/2025MultiNRC: A Challenging and Native Multilingual Reasoning Evaluation Benchmark for LLMsAlexander R. Fabbri, Diego Mares, Jorge Flores, Meher Mankikar, Ernesto Hernandez, Dean Lee, Bing Liu, Chen XingReasoning, Safety & Oversight, Evaluation and Alignment

7/23/2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable DomainsAnisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath, Bing Liu, Sean M. HendryxScience of Data, Post-Training

7/21/2025WebGuard: Building a Generalizable Guardrail for Web AgentsBoyuan Zheng, Zeyi Liao, Scott Salisbury, Zeyuan Liu, Michael Lin, Qinyuan Zheng, Zifan Wang, Xiang Deng, Dawn Song, Huan Sun, Yu SuAgents, Safety & Oversight, Evaluation and Alignment

7/15/2025Chain of Thought Monitorability: A New and Fragile Opportunity for AI SafetyTomek Korbak, Mikita Balesni, Elizabeth Barnes, Yoshua Bengio, Joe Benton, Joseph Bloom, Mark Chen, Alan Cooney, Allan Dafoe, Anca Dragan, Scott Emmons, Owain Evans, David Farhi, Ryan Greenblatt, Dan Hendrycks, Marius Hobbhahn, Evan Hubinger, Geoffrey Irving, Erik Jenner, Daniel Kokotajlo, Victoria Krakovna, Shane Legg, David Lindner, David Luan, Aleksander Mądry, Julian Michael, Neel Nanda, Dave Orr, Jakub Pachocki, Ethan Perez, Mary Phuong, Fabien Roger, Joshua Saxe, Buck Shlegeris, Martín Soto, Eric Steinberger, Jasmine Wang, Wojciech Zaremba, Bowen Baker, Rohin Shah, Vlad MikulikReasoning, Safety & Oversight, Evaluation and Alignment

6/28/2025Teaching Models to Verbalize Reward Hacking in Chain-of-Thought ReasoningMiles Turpin, Andy Arditi, Marvin Li, Joe Benton, Julian MichaelPost-Training, Reasoning

6/18/2025FORTRESS: Frontier Risk Evaluation for National Security and Public SafetyChristina Q. Knight, Kaustubh Deshpande, Ved Sirdeshmukh, Meher Mankikar, Scale Red Team, SEAL Research Team, Julian MichaelSafety & Oversight, Evaluation and Alignment

6/16/2025Adaptive Guidance Accelerates Reinforcement Learning of Reasoning ModelsVaskar Nath, Elaine Lau, Anisha Gunjal, Manasi Sharma, Nikhil Baharte, Sean M. HendryxReasoning

6/13/2025Agent-RLVR: Training Software Engineering Agents via Guidance and Environment RewardsJeff Da, Clinton J. Wang, Xiang Deng, Yuntao Ma, Nikhil Barhate, Sean M. HendryxAgents, Post-Training, Reasoning

6/5/2025A Red Teaming Roadmap Towards System-Level SafetyZifan Wang, Christina Q. Knight, Jeremy Kritz, Willow E. Primack, Julian MichaelSafety & Oversight, Evaluation and Alignment

5/9/2025Assessing Robustness to Spurious Correlations in Post-Training Language ModelsJulia Shuieh, Prasann Singhal, Apaar Shanker, John Heyer, George Pu, Sam DentonPost-Training, Science of Data

3/14/2025Relevance Isn't All You Need: Scaling RAG Systems With Inference-Time Compute Via Multi-Criteria RerankingWill LeVine, Bijan VarjavandReasoning

3/8/2025Critical Foreign Policy Decisions (CFPD)-Benchmark: Measuring Diplomatic Preferences in Large Language ModelsBenjamin Jensen, Ian Reynolds, Yasir Atalan, Michael Garcia, Austin Woo, Anthony Chen, Trevor HowarthSafety & Oversight, Evaluation and Alignment

3/5/2025The MASK Benchmark: Disentangling Honesty From Accuracy in AI SystemsRichard Ren, Arunim Agarwal, Mantas Mazeika, Cristina MenghiniSafety & Oversight, Evaluation and Alignment

2/13/2025ENIGMAEVAL: A Benchmark of Long Multimodal Reasoning ChallengesClinton J. Wang, Dean Lee, Cristina Menghini, Johannes Mols, Jack Doughty, Adam Khoja, Jayson Lynch, Sean M. Hendryx, Summer Yue, Dan HendrycksReasoning, Safety & Oversight, Evaluation and Alignment

2/11/2025J2: Jailbreaking to JailbreakJeremy Kritz, Vaughn Robinson, Robert Vacareanu, Bijan Varjavand, Michael Choi, Bobby Gogov, Scale Red Team, Summer Yue, Willow E. Primack, Zifan WangSafety & Oversight, Evaluation and Alignment

2/10/2025ProjectTest: A Project-level LLM Unit Test Generation Benchmark and Impact of Error Fixing MechanismsYibo Wang, Congying Xia, Wenting Zhao, Jiangshu Du, Chunyu Miao, Zhongfen Deng, Philip S. Yu, Chen XingSafety & Oversight, Evaluation and Alignment

1/29/2025MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMsVed Sirdeshmukh, Kaustubh Deshpande, Johannes Mols, Lifeng Jin, Ed-Yeremai Hernández Cardona, Dean Lee, Jeremy Kritz, Willow E. Primack, Summer Yue, Chen XingSafety & Oversight, Evaluation and Alignment, Reasoning

1/23/2025Humanity's Last ExamLong Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Sean Shi, Michael Choi, Anish Agrawal, Arnav Chopra, Adam Khoja, Ryan Kim, Richard Ren, Jason Hausenloy, Oliver Zhang, Mantas Mazeika, Summer Yue, Alexandr Wang, Dan HendrycksSafety & Oversight, Evaluation and Alignment, Reasoning

1/2/2025ToolComp: A Multi-Tool Reasoning & Process Supervision BenchmarkVaskar Nath, Pranav Raja, Claire Yoon, Sean M. HendryxSafety & Oversight, Evaluation and Alignment, Reasoning

10/11/2024Refusal-Trained LLMs Are Easily Jailbroken As Browser AgentsPriyanshu Kumar, Elaine Lau, Saranya Vijayakumar, Tu Trinh, Scale Red Team, Elaine Chang, Vaughn Robinson, Sean M. Hendryx, Shuyan Zhou, Matt Fredrikson, Summer Yue, Zifan WangSafety & Oversight, Evaluation and Alignment

9/29/2024Balancing Cost and Effectiveness of Synthetic Data Generation Strategies for LLMsYung-Chieh Chan, George Pu, Apaar Shanker, Parth Suresh, Penn Jenks, John Heyer, Sam DentonPost-Training, Science of Data

9/27/2024Revisiting the Superficial Alignment HypothesisMohit Raghavendra, Vaskar Nath, Sean M. HendryxPost-Training

9/5/2024Planning In Natural Language Improves LLM Search For Code GenerationEvan Wang, Federico Cassano, Catherine Wu, Yunfeng Bai, Will Song, Vaskar Nath, Ziwen Han, Sean M. Hendryx, Summer Yue, Hugh ZhangPost-Training

8/30/2024Pre-Training Multimodal Hallucination Detectors with Corrupted Grounding DataSpencer Whitehead, Jacob Phillips, Sean M. HendryxSafety & Oversight, Evaluation and Alignment, Multimodal, Science of Data

8/27/2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks YetNathaniel Li, Ziwen Han, Ian Steneker, Willow E. Primack, Riley Goodside, Hugh Zhang, Zifan Wang, Cristina Menghini, Summer YueSafety & Oversight, Evaluation and Alignment

7/18/2024Learning Goal-Conditioned Representations for Language Reward ModelsVaskar Nath, Dylan Slack, Jeff Da, Yuntao Ma, Hugh Zhang, Spencer Whitehead, Sean M. HendryxPost-Training

5/1/2024A Careful Examination of Large Language Model Performance on Grade School ArithmeticHugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, Tiffany Zhao, Pranav Raja, Dylan Slack, Qin Lyu, Sean M. Hendryx, Russell Kaplan, Michele (Mike) Lunati, Summer YueSafety & Oversight, Evaluation and Alignment

3/5/2024The WMDP Benchmark: Measuring and Reducing Malicious Use With UnlearningNathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D. Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, Gabriel Mukobi, Nathan Helm-Burger, Rassin Lababidi, Lennart Justen, Andrew B. Liu, Michael Chen, Isabelle Barrass, Oliver Zhang, Xiaoyuan Zhu, Rishub Tamirisa, Bhrugu Bharathi, Adam Khoja, Zhenqi Zhao, Ariel Herbert-Voss, Cort B. Breuer, Samuel Marks, Oam Patel, Andy Zou, Mantas Mazeika, Zifan Wang, Palash Oswal, Weiran Lin, Adam A. Hunt, Justin Tienken-Harder, Kevin Y. Shih, Kemper Talley, John Guan, Russell Kaplan, Ian Steneker, David Campbell, Brad Jokubaitis, Alex Levinson, Jean Wang, William Qian, Kallol Krishna Karmakar, Steven Basart, Stephen Fitz, Mindy Levine, Ponnurangam Kumaraguru, Uday Tupakula, Vijay Varadharajan, Ruoyu Wang, Yan Shoshitaishvili, Jimmy Ba, Kevin M. Esvelt, Alexandr Wang, Dan HendrycksSafety & Oversight, Evaluation and Alignment, Post-Training

1/22/2024Out-of-Distribution Detection & Applications With Ablated Learned Temperature EnergyWill LeVine, Benjamin Pikus, Jacob Phillips, Berk Norman, Fernando Amat Gil, Sean M. HendryxComputer Vision

11/21/2023A Baseline Analysis of Reward Models’ Ability To Accurately Analyze Foundation Models Under Distribution ShiftWill LeVine, Benjamin Pikus, Anthony Chen, Sean M. HendryxPost-Training

10/5/2023A Holistic Approach For Test And Evaluation Of Large Language ModelsDylan Slack, Jean Wang, Denis Semenenko, Kate Park, Sean M. HendryxSafety & Oversight, Evaluation and Alignment

10/4/2023On the Performance of Multimodal Language ModelsUtsav Garg, Erhan BasMultimodal, Post-Training

4/28/2023Empirical Analysis of the Strengths and Weaknesses of PEFT Techniques for LLMsGeorge Pu, Anirudh Jain, Jihan Yin, Russell KaplanPost-Training

4/11/2023Detecting and Preventing Hallucinations in Large Vision Language ModelsAnisha Gunjal, Jihan Yin, Erhan BasComputer Vision

3/11/2023Enabling Calibration In The Zero-shot Inference Of Large Vision-Language ModelsWill LeVine, Benjamin Pikus, Pranav Raja, Fernando Amat GilComputer Vision

1/29/2023Improving the Accuracy-Robustness Trade-Off of Classifiers via Adaptive SmoothingYatong Bai, Brendon G. Anderson, Aerin Kim, Somayeh SojoudiSafety & Oversight, Evaluation and Alignment

3/7/2022GlideNet: Global, Local and Intrinsic based Dense Embedding NETwork for Multi-category Attributes PredictionKareem Metwaly, Aerin Kim, Elliot Branson, Vishal MongaComputer Vision

11/16/2021CAR – Cityscapes Attributes Recognition A Multi-category Attributes Dataset for Autonomous VehiclesKareem Metwaly, Aerin Kim, Elliot Branson, Vishal MongaComputer Vision

11/7/2021Natural Adversarial ObjectsFelix Lau, Nishant Subramani, Sasha Harrison, Aerin Kim, Elliot Branson, Rosanne LiuComputer Vision

10/11/2021DEBAGREEMENT: A comment-reply dataset for (dis)agreement detection in online debatesJohn Pougué-Biyong, Valentina Semenova, Alexandre Matton, Rachel Han, Aerin Kim, Renaud Lambiotte, J. Doyne FarmerSafety & Oversight, Evaluation and Alignment

7/31/2021On The State of Data In Computer Vision: Human Annotations Remain Indispensable for Developing Deep Learning ModelsZeyad Emam, Andrew Kondrich, Sasha Harrison, Felix Lau, Yushi Wang, Aerin Kim, Elliot BransonComputer Vision, Science of Data

4/20/2021Evaluating Deep Neural Networks Trained on Clinical Images in Dermatology with the Fitzpatrick 17k DatasetMatthew Groh, Caleb Harris, Luis Soenksen, Felix Lau, Rachel Han, Aerin Kim, Arash Koochek, Omar BadriComputer Vision

11/27/2020A Survey of Deep Learning Approaches for OCR and Document UnderstandingNishant Subramani, Alexandre Matton, Malcolm Greaves, Adrian LamComputer Vision

9/4/2026

READY or Not: Reliable Enterprise Agent DeploymentAgents, Enterprise

8/20/2026

CliniCARE-Bench: Clinical Calibrated Audit of Medical Reasoning in EHREvaluation and Alignment

8/6/2026

HarnessOpt-Bench: Evaluating LLMs at Harness OptimizationAgents, Enterprise, Evaluation and Alignment

6/30/2026

DrugDiscoveryBench: Can Coding Agents Assist Early-Stage Drug Discovery?Agents, Enterprise, Evaluation and Alignment

6/29/2026

SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding SessionsAgents, Evaluation and Alignment

6/19/2026

ChainWorld: Composing Long-Horizon Desktop Workloads from Atomic OSWorld TasksAgents, Evaluation and Alignment

6/10/2026

Rubric-Guided Self-Distillation: Post-Training Without Rubric VerifiersPost-Training

6/9/2026

PSEBench: A Controllable and Verifiable Benchmark for Evaluating LLMs in Patient Safety Event TriageEvaluation and Alignment, Agents, Enterprise

6/4/2026

Insights Generator: Systematic Corpus-Level Trace Diagnostics for LLM AgentsAgents, Evaluation and Alignment, Enterprise

5/29/2026

ARCA: Adapter-Residual Credit Assignment When Token Signals DegeneratePost-Training, Reasoning

5/19/2026

Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVRPost-Training, Reasoning

5/17/2026

ASPI: Seeking Ambiguity Clarification Amplifies Prompt Injection Vulnerability in LLM AgentsSafety & Oversight, Agents, Evaluation and Alignment

5/13/2026

ROK-FORTRESS: Measuring the Effect of Geopolitical Transcreation for National Security and Public Safety

5/12/2026

Reward Hacking in Rubric-Based Reinforcement LearningSafety & Oversight, Post-Training

5/7/2026

SWE Atlas: Benchmarking Coding Agents Beyond Issue ResolutionEvaluation and Alignment, Agents

4/22/2026

Coverage, Not Averages: Semantic Stratification for Trustworthy Retrieval EvaluationEvaluation and Alignment, Science of Data, Enterprise

4/13/2026

HiL-BENCH (Human-in-Loop Benchmark)Evaluation and Alignment, Safety & Oversight

3/12/2026

Defensive Refusal Bias: How Safety Alignment Fails Cyber DefendersSafety & Oversight

2/26/2026

LLM Novice Uplift on Dual-Use, In Silico Biology TasksSafety & Oversight

2/25/2026

VeRO: An Evaluation Harness for Agents to Optimize AgentsAgents, Post-Training, Evaluation and Alignment

2/12/2026

LHAW: Controllable Underspecification for Long-Horizon TasksAgents, Safety & Oversight, Evaluation and Alignment

1/15/2026

SciPredict: Can LLMs Predict the Outcomes of Research Experiments in Natural Sciences?Safety & Oversight, Evaluation and Alignment

1/6/2026

Agentic Rubrics as Contextual Verifiers for SWE AgentsAgents, Safety & Oversight, Evaluation and Alignment

12/22/2025

MoReBench: Evaluating Procedural and Pluralistic Moral Reasoning in Language Models, More than OutcomesReasoning, Safety & Oversight, Evaluation and Alignment

12/18/2025

MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP ServersAgents, Reasoning, Safety & Oversight, Evaluation and Alignment

12/17/2025

Audio MultiChallengeMultimodal, Safety & Oversight, Evaluation and Alignment

12/16/2025

Imitation Learning for Multi-turn LM Agents via On-Policy Expert CorrectionsPost-Training, Agents

11/25/2025

PropensityBench: Evaluating Latent Safety Risks in Large Language Models via an Agentic ApproachSafety & Oversight, Evaluation and Alignment

11/13/2025

Professional Reasoning BenchSafety & Oversight, Evaluation and Alignment, Reasoning

11/10/2025

ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research AgentsReasoning, Safety & Oversight, Evaluation and Alignment

11/5/2025

Best Practices for Biorisk Evaluations on Open-Weight Bio-Foundation ModelsSafety & Oversight, Evaluation and Alignment

10/28/2025

Remote Labor Index: Measuring AI Automation of Remote WorkAgents, Safety & Oversight, Evaluation and Alignment, Reasoning

10/20/2025

REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable RewardsReasoning, Agents, Safety & Oversight, Evaluation and Alignment

10/15/2025

Beyond Seeing: Evaluating Multimodal LLMs On Tool-enabled Image Perception, Transformation, and ReasoningSafety & Oversight, Evaluation and Alignment, Reasoning, Multimodal

10/8/2025

Online Rubrics Elicitation from Pairwise ComparisonsSafety & Oversight, Evaluation and Alignment, Post-Training

9/25/2025

Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-TrainingPost-Training, Science of Data

9/23/2025

Progress over Points: Reframing LM Benchmarks Around Scientific ObjectivesSafety & Oversight, Evaluation and Alignment

9/19/2025

SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?Agents, Safety & Oversight, Evaluation and Alignment

9/11/2025

TutorBench: A Benchmark To Assess Tutoring Capabilities Of Large Language ModelsSafety & Oversight, Evaluation and Alignment

8/26/2025

Reliable Weak-to-Strong Monitoring of LLM AgentsSafety & Oversight, Evaluation and Alignment

8/13/2025

Search-Time Data ContaminationSafety & Oversight, Evaluation and Alignment

7/23/2025

MultiNRC: A Challenging and Native Multilingual Reasoning Evaluation Benchmark for LLMsReasoning, Safety & Oversight, Evaluation and Alignment

7/23/2025

Rubrics as Rewards: Reinforcement Learning Beyond Verifiable DomainsScience of Data, Post-Training

7/21/2025

WebGuard: Building a Generalizable Guardrail for Web AgentsAgents, Safety & Oversight, Evaluation and Alignment

7/15/2025

Chain of Thought Monitorability: A New and Fragile Opportunity for AI SafetyReasoning, Safety & Oversight, Evaluation and Alignment

6/28/2025

Teaching Models to Verbalize Reward Hacking in Chain-of-Thought ReasoningPost-Training, Reasoning

6/18/2025

FORTRESS: Frontier Risk Evaluation for National Security and Public SafetySafety & Oversight, Evaluation and Alignment

6/16/2025

Adaptive Guidance Accelerates Reinforcement Learning of Reasoning ModelsReasoning

6/13/2025

Agent-RLVR: Training Software Engineering Agents via Guidance and Environment RewardsAgents, Post-Training, Reasoning

6/5/2025

A Red Teaming Roadmap Towards System-Level SafetySafety & Oversight, Evaluation and Alignment

5/9/2025

Assessing Robustness to Spurious Correlations in Post-Training Language ModelsPost-Training, Science of Data

3/14/2025

Relevance Isn't All You Need: Scaling RAG Systems With Inference-Time Compute Via Multi-Criteria RerankingReasoning

3/8/2025

Critical Foreign Policy Decisions (CFPD)-Benchmark: Measuring Diplomatic Preferences in Large Language ModelsSafety & Oversight, Evaluation and Alignment

3/5/2025

The MASK Benchmark: Disentangling Honesty From Accuracy in AI SystemsSafety & Oversight, Evaluation and Alignment

2/13/2025

ENIGMAEVAL: A Benchmark of Long Multimodal Reasoning ChallengesReasoning, Safety & Oversight, Evaluation and Alignment

2/11/2025

J2: Jailbreaking to JailbreakSafety & Oversight, Evaluation and Alignment

2/10/2025

ProjectTest: A Project-level LLM Unit Test Generation Benchmark and Impact of Error Fixing MechanismsSafety & Oversight, Evaluation and Alignment

1/29/2025

MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMsSafety & Oversight, Evaluation and Alignment, Reasoning

1/23/2025

Humanity's Last ExamSafety & Oversight, Evaluation and Alignment, Reasoning

1/2/2025

ToolComp: A Multi-Tool Reasoning & Process Supervision BenchmarkSafety & Oversight, Evaluation and Alignment, Reasoning

10/11/2024

Refusal-Trained LLMs Are Easily Jailbroken As Browser AgentsSafety & Oversight, Evaluation and Alignment

9/29/2024

Balancing Cost and Effectiveness of Synthetic Data Generation Strategies for LLMsPost-Training, Science of Data

9/27/2024

Revisiting the Superficial Alignment HypothesisPost-Training

9/5/2024

Planning In Natural Language Improves LLM Search For Code GenerationPost-Training

8/30/2024

Pre-Training Multimodal Hallucination Detectors with Corrupted Grounding DataSafety & Oversight, Evaluation and Alignment, Multimodal, Science of Data

8/27/2024

LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks YetSafety & Oversight, Evaluation and Alignment

7/18/2024

Learning Goal-Conditioned Representations for Language Reward ModelsPost-Training

5/1/2024

A Careful Examination of Large Language Model Performance on Grade School ArithmeticSafety & Oversight, Evaluation and Alignment

3/5/2024

The WMDP Benchmark: Measuring and Reducing Malicious Use With UnlearningSafety & Oversight, Evaluation and Alignment, Post-Training

1/22/2024

Out-of-Distribution Detection & Applications With Ablated Learned Temperature EnergyComputer Vision

11/21/2023

A Baseline Analysis of Reward Models’ Ability To Accurately Analyze Foundation Models Under Distribution ShiftPost-Training

10/5/2023

A Holistic Approach For Test And Evaluation Of Large Language ModelsSafety & Oversight, Evaluation and Alignment

10/4/2023

On the Performance of Multimodal Language ModelsMultimodal, Post-Training

4/28/2023

Empirical Analysis of the Strengths and Weaknesses of PEFT Techniques for LLMsPost-Training

4/11/2023

Detecting and Preventing Hallucinations in Large Vision Language ModelsComputer Vision

3/11/2023

Enabling Calibration In The Zero-shot Inference Of Large Vision-Language ModelsComputer Vision

1/29/2023

Improving the Accuracy-Robustness Trade-Off of Classifiers via Adaptive SmoothingSafety & Oversight, Evaluation and Alignment

3/7/2022

GlideNet: Global, Local and Intrinsic based Dense Embedding NETwork for Multi-category Attributes PredictionComputer Vision

11/16/2021

CAR – Cityscapes Attributes Recognition A Multi-category Attributes Dataset for Autonomous VehiclesComputer Vision

11/7/2021

Natural Adversarial ObjectsComputer Vision

10/11/2021

DEBAGREEMENT: A comment-reply dataset for (dis)agreement detection in online debatesSafety & Oversight, Evaluation and Alignment

7/31/2021

On The State of Data In Computer Vision: Human Annotations Remain Indispensable for Developing Deep Learning ModelsComputer Vision, Science of Data

4/20/2021

Evaluating Deep Neural Networks Trained on Clinical Images in Dermatology with the Fitzpatrick 17k DatasetComputer Vision

11/27/2020

A Survey of Deep Learning Approaches for OCR and Document UnderstandingComputer Vision

READY or Not: Reliable Enterprise Agent Deployment