BACK
Physical AI11/17/2026

Measuring Robot Data Diversity: Insights from 20,000 Hours of Real-World Data

By Yusuf Umut Ciftci, Chitrangada Devulapalli, Pragna Mannam

At Scale AI, we believe that building a high-quality robotics dataset is more than just collecting more hours. We emphasize diversity on four axes of manipulation: task, object, action, and environment. Each of these axes has its own long-tail distribution: tasks are the most diverse by effective number, while the objects being manipulated have the longest tail. Through a detailed analysis of text and visual embeddings on our 20,000-hour dataset, we find that vision-only models’ embedding spaces mostly capture the scene and place little emphasis on the action. The vision-language-action model we tested keeps the action distinct, though it also receives the verb in its language input. Understanding what models see in robot data is a critical step toward building robotics foundation models.

Breaking down the data

The 20,000-hour slice of subtask-annotated teleoperation data we analyze consists of Mobile Trossen robot data, which has two arms on a wheeled base (Mobile ALOHA). Here is the dataset at a glance.

The dataset in numbers
Recorded videos344,625 recordings
Subtasks9,924,297
Tasks797
Places231 sites across 11 scene types

Each recording contains a single task, such as “Put supplies away”, split into several subtasks: short video clips, each with a one-line annotation (“Place the measuring cups in the container”). Each recording was also made at one site, a particular room, which is labeled with its scene type (kitchen, office, etc.) and its lighting.

To make sense of such a large corpus of data, we built a visualizer that lays out every subtask on a model’s map, where similar video clips sit close together, and lets us search based on annotation or browse from a verb to the object it acts on.

How do we define diversity of our dataset?

We look at our dataset along four axes: tasks, verbs, objects and scene types. For each one we count its variations and check whether these are densely represented or have a long-tail distribution.

To capture the spread in a single metric we use the effective number: how many equal-sized groups would give the same variety (Hill, 1973; Jost, 2006). If every group within an axis held the same number of hours, the effective number would be the number of groups. When a few groups dominate, the effective number gets smaller. Figure 2 below shows the shape of the data distribution across each axis.

Figure 2: The dataset at a glance, with long-tail distributions on three axes. (A) hours per verb, with the verbs under 0.1% of the hours pooled in grey. (B) hours by kind of object; “Other” is the 5.9% of the hours that list no kind, and a recording that lists two kinds counts under both, so the shares sum past 100%. (C) hours (left) and distinct sites (right) per scene type, with each type’s recordings in brackets.
DimensionCountLargestEffective number
Tasks797Zip bag, 5.7%194
Verbs86put, 19%17
Objects3,696towel, 6.2%162
Scene types11kitchen, 33%—

We show long-tail distributions across verbs, objects, and sites in Figure 2. More than 1 million subtask phrases are used only once. Figure 2A shows our long-tail distribution within verb labels. Some sites have a larger representation in the dataset: bedrooms hold 5% of the hours at just 4 sites (Figure 2C).

Objects, mentioned in the task annotations, have the longest tail. Their 3,696 labels behave like about 162 equally common ones: about as much variety as tasks (194) and far more than verbs (17), but spread over a much longer list, since thousands of object labels appear only a handful of times. In the next section, we show how varied the common objects are.

How the data looks to a person

Volume tells us how much of each task there is, but not how different recordings may be from each other. A thousand towel folding episodes could have the same exact white towel on the same table. We looked at what the camera saw per task, and asked three questions. Which objects show up, and how varied are they? How varied are the rooms they were recorded in? And how do tasks compare to each other on these two criteria?

To study the visual diversity of our dataset we run an object detector on the first frame of every recording and count the descriptive words the annotations use.

At first glance, the first frame

Analyzing the first frame of every recording, we found 1,372 unique objects across our dataset, and each frame typically holds 7 objects. Tables are the most common and are present in 71% of the first frames, then plants (42%) followed by bottles (33%).

Figure 3: Detector labels in the first frames, sized by the number of recordings whose first frame shows them (people are left out). Orange words are acted on in at least 10 subtasks somewhere in the data; blue words are seen but almost never handled.

As our dataset is composed of loco-manipulation data, the first frame doesn’t capture every object in the episode. Only 39% of the total objects were found in the first frames. Here are the most common objects, next to the ones that appear rarely.

Figure 4: (A) Most frequent: objects with a usable crop, each from a different recording that handles it. A dagger (†) denotes the 2 crops that use the bounding box of the head noun (“bag” for a laptop bag). (B) Rare: objects each named in 6 subtasks or fewer.

The same object, many forms

The most common objects come in many forms. The text annotations label between 83 and 506 different modifier words to each of the 12 most frequent objects. These modifiers include color, material and state of the objects. Each of these 12 objects appears across 39 or more of the 231 sites.

Figure 5: Six objects: each at eight different sites.

What the scenes look like

To compare how varied the backgrounds are in each scene type in our dataset, we took the first 60 frames from each and counted the different background objects in them. Kitchens had the most variety, with 82 different objects, and industrial rooms the least, with just 14.

Figure 6: (A) Most common: the 30 most common background objects, across first frames. (B) Distinct labels per scene type: distinct background labels per scene type, over all first frames, with each scene type’s recordings in brackets and the rows in order of hours.

Below are examples of sites in each scene type, to show how much rooms of the same kind can differ, along with examples of each of the four lighting labels: normal, dim, bright and variable.

Figure 7: Scene types, one frame from each of up to four sites per scene type.
Figure 8: Lighting, each label’s four frames from four different sites.

The most diverse task

So far we looked at our dataset as a whole. Here, we look at the diversity within each task. To make the comparison fair, we take the same number of recordings from every task, 200 picked at random, to remove bias towards tasks with a larger share of the data. Of the 270 tasks with at least 200 recordings, we score on seven axes from the number of sites and objects a task covers to how different its clips look (see Appendix). The last of these is the Vendi score measuring how many truly different clips a set contains, as judged by the VLM video embedding from the next section.

"Put supplies away" comes out on top, with a few other tasks close behind. We compare our most diverse task with the median task below:

MeasurePut supplies awayMedian task
Task objects242
Background objects127.559.7
Vendi score of its clips43.116.1
Figure 9: “Put supplies away” at 24 different sites, covering all 24 of its task objects; each frame is captioned with its scene type and up to three task objects. Picked to show new sites, then new objects.

The same task in three rooms:

“Put supplies away” in an office, a kitchen and an entryway.

Figure 10: Office. “Place the black book in the rightmost slot of the desk file organizer” (5.2 s): a dark desk with a potted plant.
Figure 11: Kitchen. “Place the measuring cups in the container” (5.1 s): a blue table set with colored bowls.
Figure 12: Entryway. “Move the green rubber band from the table to the stack of rubber bands” (7.2 s): a white table with a wicker bowl, plants and colored cups.

How the data looks to a model

A robot policy does not see our scene labels. It sees pixels, and often an annotation, through a learned encoder. So we also turn every subtask into a vector with five models, and ask what each one recognizes accurately:

  • VLM text annotation embedding (VLM text), a vision-language model (Qwen3-VL-Embedding), on the annotation alone.
  • WM, a video world model (V-JEPA 2.1), on 16 frames spread evenly over the clip.
  • VLM video embedding (VLM video), the same Qwen3-VL-Embedding model as VLM text, on the clip at one frame per second (4 to 64 frames).
  • DINO, an image model (DINOv3), on one image of the scene: the clip’s first frame.
  • π0.5, the vision-language part of the π0.5 robot policy, on that same first frame and the annotation together.

Every image comes from the robot’s top camera; the wrist cameras are not used. π0.5 is the only model we test that sees both the image and language annotation.

The VLM text annotation embedding

We start with the VLM text annotation embedding space, which turns each annotation into a vector, so texts with similar meanings get similar vectors, and similar texts land close together. (Figure 13A)

Some verbs have dense representations within the space. For example, “hand off” forms one long cluster through the middle of the embedding space: 94% of its subtasks sit in patches where it is the most common verb. Meanwhile, place, move, reposition, push and stack spread through mixed parts of the space. (Figure 13B)

When we group the annotations into 128 clusters (k-means), the groups follow the object more than the action. In each cluster we also identify the top three terms that appear in the text annotation (Figure 14). We use AMI scores to understand the correlation between groups. AMI scores how well two groupings agree: 0 is chance and 1 is a perfect match. 18 of the 20 largest groups have an object in their top terms, and the groups match the object in the text annotation (AMI 0.44) better than the verb (AMI 0.21).

Figure 13: (A) All annotations, the 2.51 million distinct annotations, each colored by its majority verb on a UMAP of the VLM text annotation embedding; the other verbs (12% of subtasks) are light grey, and a stronger color marks a text used by more subtasks. (B) One verb per panel, the 12 verbs with the most subtasks, one panel each, on the same map, with each verb’s share of subtasks.
Figure 14: The 20 largest groups of the 128 annotation groups, each named by its three most frequent verb and object words.

The visual embedding spaces

In WM and DINO each cluster is mainly associated with a room type. The VLM video embedding has one large region where kitchen and office scenes overlap, and π0.5 has some clusters that can be separated purely by scenes. The VLM text annotation embedding lacks correlation with scene types since “pick up the towel” is said in many kinds of scenes.

Figure 15: One column per model. (A) Scene types. Each model’s embedding space, colored by scene type (in the VLM text column, by each text’s most common one). Colors mix where scene types overlap, and brightness follows point density. (B) The six largest tasks, where each sits on the same map, in orange over all the clips in grey, with its share of hours in brackets (in the VLM text column, each text counts under its subtasks’ most common task).

In the vision-only embeddings, each of the largest tasks is a well-separated cluster. The embedding space is collapsed into a 2D visualization here and can be misleading, so the nearest neighbor analysis below comes from the vectors themselves.

Nearest neighbor analysis

In order to understand what an embedding space represents, we took a recording's 10 closest neighbors and checked if they share its label: the room type, the task, the object or the verb. We score this as κ: 0 is what random neighbors would give, and 1 means all ten always match. We take the neighbors from other sites only, so that a match cannot come from the same task recorded again in the same room.

Figure 16: κ with neighbors from other sites, on 1 million sampled subtasks. The VLM text annotation embedding is left out here, since it sees only the annotation and no images.

In the vision-only models, the scene type matches best among nearest neighbors (κ 0.512 to 0.642). The task (0.103 to 0.317) and the main object manipulated (0.144 to 0.423) match far less.

What each embedding space represents well

In Figure 16, we see how the embedding spaces compare when nearest neighbors are matched by scene, task, verb, and object. Some embeddings are better at representing specific labels.

  • The VLM video embedding represents tasks (0.317) and objects the best (0.423) across sites.
  • DINO represents the scene type the best (0.642), from a single frame.
  • π0.5 represents verbs the best (0.763) compared to vision-only models with at most 0.141.

In a site analysis, WM tells sites apart the best. The site explains 0.37 of the variation which is twice as much as the VLM video embedding (0.17) or DINO (0.19). Yet WM embedding space has the weakest representation of the room type, the task, the object and the verb of the three vision-only models.

A caveat. The task annotation input to π0.5 contains the verb. Therefore its embedding’s ability to represent the verb shows that it retains that information, and not necessarily that it infers it from the input images.

Conclusion

Hours are the simplest number to report about a robot dataset, but they say little about what a robot actually sees and how it interacts with the world. How the hours are spread, how varied the objects and scenes are, and what a model's embedding space separates tell us much more.

In embedding space, vision-only models mostly capture where the robot is rather than what it is doing: nearest neighbors match on scene type far more than on the verb. VLM text annotation embeddings do separate actions, though they group annotations more by object. The VLA we tested keeps the verb far more distinct than any vision-only model. However, it also receives the verb in its language input, so this shows it retains the action, not necessarily that it infers it from pixels.

Every axis of our data has a long tail. Most long-tail studies, including our own prior work, focus on tasks, yet verbs, objects and scenes are just as unevenly distributed. Whether their long tails help policies the way the task tail does remains an open question. As we continue to grow our high-diversity robotics operations at Scale, we will continue to study the effect of these diversity axes and long-tail distributions on policy performance and generalization.

Appendix

  • Names. Spelling variants share one label (zipper bag → zip bag). Verbs are kept as annotated.
  • Effective number. The exponential of the Shannon entropy of the shares, exp(−Σ pᵢ ln pᵢ), known as the Hill number of order 1, N1 (Hill, 1973; Jost, 2006). It equals the plain count when every group has the same share and falls as a few groups dominate. The Vendi score (Friedman & Dieng, 2023) applies the same idea to pairwise similarities instead of label counts.
  • AMI. Adjusted mutual information between two groupings, corrected for chance. The verb and object labels are not equally distributed: the 50 most common verbs cover 99.8% of the annotations, the 50 most common objects only 56.0%.
  • Vectors and maps. In every model, the vectors are centered, reduced to 128 dimensions with PCA, scaled to unit length and compared by cosine. π0.5’s are also whitened (rescaled to the same variance in every direction), because a few directions dominate its raw vectors. The maps are drawn with UMAP.
  • κ (neighbor agreement). The share of a recording’s 10 nearest neighbors that share its label, rescaled so that random neighbors score 0, computed per subtask.
  • The most diverse task. To compare tasks of different sizes fairly, we score each task on equal-sized random samples. From 200 of its recordings we count across six axes: distinct sites, scene types, lighting conditions, task objects, background objects and verbs. A seventh measure is the Vendi score of 2,000 of its clips in the VLM video embedding. Each count is averaged over 100 random draws and the Vendi score over 5. We then rank every task on each of the seven measures as a percentile, and a task's diversity score is the mean of its seven percentiles. The Vendi score (Friedman & Dieng, 2023) is the effective number of distinct items in a set, computed from the similarity between every pair of items. Its values depend on the embedding model, so it compares sets within one model, not across models.