tl;dr: Human demonstrations allow for diverse data collection in the real world, but pose a difficult learning problem due to the large embodiment gap between robots and humans. Being able to effectively learn from high volumes of human interaction data could unlock the future of robotics.
We investigate the value of introducing human demonstrations in a low-data regime. Through our co-training experiment, we show that adding a limited amount of human demonstrations to robot datasets can nearly double performance on unseen environments while only sacrificing a nominal amount of in-distribution performance. When it comes to real-world deployments, including human data alongside robot data can provide the generalization necessary for models to be robust when encountering unstructured, real world settings.
Robotics Has a Data Problem.
Data that closely resembles a target robot or embodiment is easier to learn from, but more challenging to collect at high volumes. Robot teleoperation[1] is the most direct source of training data, as it is expressed in the same state space a robot policy will control. Unfortunately, hardware complexities associated with deployment and maintenance of teleoperation rigs in the real world make this collection paradigm both expensive and often not generalizable to new systems.
Other approaches – such as using portable collection devices designed to approximate a robot’s end-effector[2] occupy a middle ground. This results in a relatively small embodiment gap and supports more scalable data collection in diverse environments. With this said, transferring this data to different hardware and control systems remains challenging.
A third school of thought is to learn from human demonstrations directly. In accordance with the data collection tradeoff, this paradigm enables large scale collection and maximal diversity, but the large embodiment gap poses a difficult learning objective.
Our hands are not made up of actuators; the viewpoint, morphology, and action distribution all differ, and robot policies cannot consume human data the same way they consume robot data. Beyond this, there are open questions on alignment, data mixture, and actual learning objective, all of which need to be sufficiently answered before large scale usage can occur.
As each data source offers a distinct set of benefits, an approach called co-training uses multiple sources together during model training. The goal is to capture the strengths of each source while mitigating their individual limitations. For example, human demonstrations can expose a robot policy to a wider range of objects, scenes, and task strategies, while robot data provides the embodiment-specific dynamics needed to translate that knowledge into actuator-level control.
Academia has shown the ability to produce robot policies that perform well in a lab, but making robots actually work in the real world will require policies to be performant in unseen environments and situations. This robustness to unseen scenes is what we set out to investigate. In this study, we asked whether a small amount of egocentric human data could make a robot policy less dependent on evaluation scenes matching training scenes.
Experiments
In our study, we asked a narrow question: can the value of co-training be detected in a low-data regime? To investigate this, we conducted an experiment studying what effect adding a small amount of task-aligned human data has on task performance.
Our experimental setup consists of fine-tuning a baseline policy on five hours of on-embodiment robot demonstrations. We compared this against a policy fine-tuned on the same robot data plus an additional five hours of task-aligned egocentric human data. We then evaluated these policies across three manipulation tasks, varying in difficulty, on a single robot embodiment consisting of two 6-DOF Trossen WidowX stationary arms and two-finger gripper end-effectors.
The first task; picking up a variety of objects and placing them into a container; and the second, picking up a cup and placing it on a saucer.
Data Collection
Robot data was collected using a leader-follower teleoperation setup on the same robot that was used for policy evaluation. Human data was collected with a custom egocentric collection rig, which consists of a head-mounted camera along with motion sensors. We paired this data with our custom 3D hand-pose tracking to generate usable human proprioception. To minimize the visual gap between the two sources, we placed the robot's top-down camera in an orientation that aligns with the egocentric human camera's viewpoint in both height and angle.
Model Architecture
We chose the π0.5 vision-language-action model[3] for our experiments as it is a frontier open source robotic model and a good starting point given its large scale pretraining. Similar work has been done on non-VLM backed architectures[4], but we were curious whether co-training benefits would generalize to this family of models.
Action Representation
Robot policies often operate in joint space, meaning they directly predict the state of each actuator in the robot arm, but this poses a problem since human data does not have a clear parallel. In order to align both data sources, we choose to represent spatial actions in end-effector (EEF) space, meaning the model learns to predict the position of the hand/end-effector rather than the joint angles that would produce such a position. This adds some processing overhead in order to connect the model and low-level robot controller.
The final data flow looks like this: the robot emits state in joint space → we run forward kinematics to convert it to EEF space → the model ingests state and emits an action in EEF space → we run inverse kinematics to convert the action back to joint space → the robot controller takes in the action signal in joint space.
Evaluation Methodology
Each task is decomposed into a small set of sequential subgoals and for each we track three separate binary signals.
Setup: did the policy transition into the state required to attempt the subgoal? This is a proxy for whether the policy showed an understanding of, at a high level, what the subgoal is supposed to be and that it is attempting it.
Execution: was the subgoal completed successfully?
First-attempt success: was the subgoal completed on the first attempt, without a retry?In the results section, we report the rate for each metric, aka the percentage of subgoals, across all rollouts, for which that metric was satisfied.
We evaluate each policy both in-distribution (ID), on the objects and scenes represented in the training data, and out-of-distribution (OOD), on scenes the policy has never encountered before. OOD states include but are not limited to object color, texture, and instance, as well as placement and orientation.
Results
The results are conclusive: our co-training recipe reliably increased a policy’s robustness to novel scenes.
Out-of-distribution, the policies trained on both human and robot data outperformed the robot only baseline in all metrics.
For object-in-container, subgoal success doubled from 20% to 40%. First-attempt success rose from 12.5% to 32.5%, meaning the co-trained policy was substantially more likely to get the behavior right without first making a failed attempt. Overall task success also doubled, from 20% to 40%. Cup-on-saucer showed the same general pattern. Out-of-distribution subgoal success increased from 42.5% to 55%, while overall task success rose from 30% to 45%.
The co-trained policy was exposed to a broader range of object appearances and configurations, along with multiple execution strategies for each task. Although human demonstrations do not provide exact robot actions, they preserve important task-level structure: which objects matter and where to move. When the evaluation scene looked unfamiliar, the policy was therefore less dependent on the narrow visual and behavioral regularities present in the robot data. The human demonstrations may have helped it recognize the same underlying task in more forms, while the robot demonstrations taught it how to execute that task with its particular embodiment.
While subgoal setup depends on skills such as perception and control, it is less sensitive to low-level execution thus can serve as a proxy for high-level task intent. Out-of-distribution setup improved by 10 percentage points on both tasks: from 72.5% to 82.5% for cup-on-saucer and from 60% to 70% for object-in-container.
In-distribution, the two policies remained relatively close but the robot data only policy was marginally better. Subgoal success fell from 72.5% to 65% on cup-on-saucer and from 75% to 70% on object-in-container. Overall task success declined by ten percentage points on both tasks: from 65% to 55% for cup-on-saucer and from 70% to 60% for object-in-container.
This is not entirely unexpected since this is both a low data regime and adding human data increases the domain gap to data seen at evaluation time. Although we made an attempt to align viewpoints and represent both in end-effector space, fundamental differences still remained. The policy was encouraged to learn behavior that transferred across appearances and embodiments, but slightly reduced its precision on the narrow distribution represented by robot demonstrations. External work has shown that this ID performance drop can get smaller, or even disappear, as the overall dataset size grows.[5]
The simplest way to analyze the result is through the generalization gap: the drop-off from ID to OOD task success. For cup-on-saucer, that gap narrowed from 35 percentage points with robot-only training to 10 points with co-training. For object-in-container, it narrowed from 50 points to 20. The co-trained policy was substantially less dependent on evaluation scenes being aligned with the training scenes. This behavior is quite important for real world robustness.
This was a case study on generalization in low volume co-training data regimes. The effects we show are a modest but consistent improvement in out-of-distribution robustness, but more work can be done to further solidify findings.
Model Caught “Cheating”
The cup-on-saucer training data involved a handoff, regrasping the cup from one arm to the other before placing it, though the language prompt was simply "place cup on saucer". We noticed that the policy would sometimes not attempt a handoff at all, and in cases where it did attempt one that did not go cleanly, it would give up and place the cup directly with the initial grasp arm. In our eyes this was not necessarily a policy failure due to how the prompt was worded. For this reason we decided not track it as a scored subgoal.
Implementation Details
We found that three choices materially affected task performance.
Initially our predictions were absolute EEF positions with respect to a fixed point (the top down camera), but this led to our policy exhibiting failures due to low precision. We eventually moved to relative prediction where the action is purely a directional signal of where to move the arm from its current position. Relative prediction resulted in noticeably better downstream task performance.
Fixing the robot camera to approximate the human egocentric viewpoint also reduced an unnecessary source of visual variation. One day after one too many hours of debugging concerningly poor eval results, we discovered that the camera mount had been bumped and the slight misalignment was the culprit.
Another issue we faced was with jerky EEF rotations, and that turned out to be a result of how we were encoding pose. Euler-angle representations contain numerical discontinuities between nearby physical orientations which turned out to be hard for the policy to learn. We instead used a 6D rotation representation: the model predicts two three-dimensional vectors, which are converted into a valid rotation matrix using Gram–Schmidt orthogonalization. This provides a smoother representation of orientation and improved training performance.
The Future
Recent work suggests that results may depend heavily on how much of both human and robot data is used. EgoScale5 pretrained a VLA model on more than 20,000 hours of human data and found that cross-embodiment learning improved as the amount of human data increased. It also found that large-scale human pretraining worked best when followed by a smaller midtraining stage using aligned human and robot data.
PI’s study Emergence of Human to Robot Transfer in VLAs[6] showed that training on human demonstrations produced roughly a twofold improvement across several generalization scenarios. More interestingly, the benefit of human data increased as the diversity of the robot data used during pretraining increased. In other words, more robot data made the policy better at learning from humans.
It would also be interesting to investigate the role of unaligned data. This data is harder to translate to robot actions, but on the other hand can contain far more behavioral and environmental diversity. Our hypothesis is a mix: aligned data to bridge the embodiment gap, unaligned data to introduce new skills that transfer to the robot. How that mix should shift across pre-training, mid-training, and post-training remains open.
We anticipate this being an active field of research for a while as solving the human transfer problem will accelerate the entire industry’s learning rate.
Thanks to our Friends
Special thanks to our partners from the RL2 lab at Georgia Tech who inspired much of this work. We collaborated with them on the EgoVerse[7] paper which was the first large scale multi-institution study on the effects of human data, and an accompanying open-source dataset. This work has helped shape how we think about egocentric human data and its role in robot learning. We encourage readers interested in this direction to take a look at the paper, which was presented at RSS 2026 in Sydney, Australia.
[1] Zhao et al., “Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware,” 2023.
[2] Chi et al., “Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots,” 2024.
[3] Physical Intelligence et al., “π0.5: a Vision-Language-Action Model with Open-World Generalization,” 2025.
[4] Kareer et al., “EgoMimic: Scaling Imitation Learning via Egocentric Video,” 2024.
[5] Zheng et al., “EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data,” 2026.
[6] Kareer et al., “Emergence of Human to Robot Transfer in Vision-Language-Action Models,” RSS 2026.
[7] Punamiya et al., “EgoVerse: An Egocentric Human Dataset for Robot Learning from Around the World,” RSS 2026. See also the project and dataset and code.