FineART: Scale AI’s First Open-Source Robotics Dataset and Model

In multi-step tasks, each action depends on the last. Missing one step can cause failure in the task.

Breaking down a high-level task into step-by-step subtasks is intuitive for humans: open the box with a box cutter, unfold the flaps, remove the items. Today’s open robot models are often trained on demonstrations labeled with a single task-level instruction: unpack the box.

To close that gap, Scale AI and Hugging Face have released FineART, the largest open robot dataset with subtask labels, alongside FineART-VLA, the first open System 1-System 2 model in LeRobot that can reason over step-by-step actions. Scale AI’s data engine, together with Hugging Face’s LeRobot ecosystem, enables collection, training, and evaluation from end to end.

Data Quality and Diversity Are Essential for Robotics

AI has progressed because there is a wealth of data for language and vision. Open datasets for robotics remain sparse by comparison.

Volume alone will not be sufficient. Data for robotics must also be high-quality and diverse. Scale AI brings years of experience delivering results for labs, enterprises, and governments, which we extend to the physical frontier.

What’s Inside the FineART Dataset

FineART is a collection of 40,543 recorded episodes, where each episode is a complete attempt at a task, captured from start to finish. Together, the episodes represent 1,718 hours of footage across 151 different tasks.

Each episode also includes subtask annotations. A subtask annotation pairs a timestamped action clip, marked by beginning and end, with a descriptive natural-language caption. Across the full dataset, there are 533,913 labeled steps, or approximately 13 per episode.

Three characteristics set FineART apart:

• Contact-rich long-tail: 77% of the data covers pick-and-place and sorting while the remaining 23% covers complex manipulation: textiles, cables, tool use, and precision insertion.
• Two-level data annotation: Every episode is labeled with task-level and temporally segmented subtask annotations.
• Three-stage data pipeline: One team collected episodes, a second annotated each subtask, and a third validated quality.

FineART-VLA: A Model That Plans Before It Acts

A vision-language-action (VLA) model is a system that takes a camera frame, a language instruction, and the robot state as input and outputs an action.

FineART-VLA is a VLA that plans before it acts. Unifying high-level planning and low-level motor control, it is LeRobot’s first open System 1-System 2 VLA available on Hugging Face:

• System 2 (Planning): Reasons over the scene to predict subtasks.
• System 1 (Control): Translates predicted subtasks into actions.

3,400 Real-World Evaluations

The model’s results show that FineART improves real-world performance. Specifically, FineART’s diverse coverage and subtask annotations unlock three core capabilities:

• Spatial instruction following: Understands relative positions between left, right, and middle.
• Long-horizon execution: Supports step-by-step guidance through natural-language prompting.
• Cross-embodiment transfer: Generalizes to multi-stage tasks on new hardware.

How to Get Started

Robot learning accelerates with open data, open models, and accessible hardware. FineART and FineART-VLA are natively integrated into Hugging Face’s LeRobot ecosystem.

Get started: 

1. Build with 500K subtask annotations: Load lerobot/FineART via LeRobotDataset.
2. Annotate your episodes: Run lerobot-annotate to segment subtasks on any LeRobot dataset. For larger-scale and higher-quality annotations, reach [email protected].
3. Train with FineART-VLA: Run lerobot-train to fine-tune FineART-VLA on your subtask annotations.

Important Links

• Dataset: huggingface.co/datasets/lerobot/FineART
• Model: huggingface.co/lerobot/FineART-VLA-subtask-midtrained
• Training code: huggingface.co/docs/lerobot/main/fineart_vla
• Paper: arxiv.org/pdf/2609.36416

FineART is a collaboration between Scale AI and Hugging Face's LeRobot team, with co-authors Pepijn Kooijmans and Thomas Wolf. We thank the operators who collected the data, annotated the data, and conducted real-world evaluations. Full acknowledgments are in the paper.