Before a robot demonstration video becomes useful training data, someone has to decide whether it's trustworthy enough to learn from. That decision is the Quality Check (QC) problem, and at scale, no human can make it alone. The goal of an automated QC system is to surface clips that are broken, ambiguous, or risky enough to need human review before they enter the dataset.
Small scale quality control is a human task: open the video, watch carefully, and mark whether the data should be used. Large scale QC determines what the model is allowed to learn from.
We have been studying this problem across two data streams:
- Robot-mounted manipulation videos, where the task is to decide whether a demonstration is valid before annotation
- Ego-camera collector videos, where the task is to catch first-person recording failures, motion failures, environment violations, and incomplete tasks
The surprising lesson is that "just ask a VLM if the video is good" is the wrong recipe. The most useful systems work more like measurement pipelines, they make the evidence explicit, break the rules down into smaller parts, calibrate for each type of failure, and send uncertain clips to humans along with the exact visual evidence that triggered the model.
The Problem With “Pass” and “Reject”
The QC rulebook starts with what sounds like a simple question: should this clip pass?
But a rejected clip can be rejected for many different reasons. The camera might turn green. The task might stop early. An object might fall out of reach. Two actions might overlap. A human hand might enter the scene. A camera might be missing. The clip might contain private information.
Those are very different problems. They need different evidence, and they create different risks if the model gets them wrong.
Data Distribution
| Local robot QC snapshot | Count |
|---|---|
| Labeled demonstrations | 4,088 |
| Passed QC demonstrations | 3,796 |
| Failed QC demonstrations | 292 |
| QC failure rate | 7.1% |
Most clips pass. In one sample robot QC snapshot, only a small share of labeled demonstrations were rejected. A model can look decent by mostly predicting “passed,” while still missing the rare failures we care about. The opposite is also bad: a model that predicts “rejected” too often floods reviewers with false alarms and removes useful clips from the dataset.
So we cannot judge the model by accuracy alone. We need to ask three simpler questions:
- How many rejected clips does it catch?
- How many passed clips does it preserve?
- And when it makes a mistake, is that mistake acceptable for the review workflow?
The task worked well because it was narrow and concrete. The VLM was not trying to decide whether the entire clip was good or bad. It was helping reviewers flag one specific, visible camera issue.
The Broad VLM Judge
We started with the simplest version of the idea: give a VLM the rulebook, examples, and the video, then ask it to decide whether the clip should pass or be rejected.
The benchmark was controlled across runs. We used the same task mix, left-camera streams only, the full QC rules in the prompt, and compact contact sheets with rule examples. We also tried several ways of giving the model evidence: full videos, low-fps transcodes, split videos, sampled frames, and a second verification pass using the same model.
The result was not a clean win; each setup failed in a different way. Whole-video inputs caught more rejected clips, but they also rejected too many clips that had passed. Sparse-frame inputs preserved more passed clips, but missed most of the failures. The verifier made the model more conservative, but not more useful.
So the problem was not simply “the model cannot see,” but that we were asking one model to answer too broad a question.
Breaking QC into Small Checks
Instead of asking, “Should this whole video pass?” we moved to focused checks. Each check looks for one failure mode and gets the evidence needed to judge that failure.
- Is there a sensor issue?
- Did the demonstration fail to complete?
- Did an object leave the workspace in an unrecoverable way?
- Do two actions overlap when they should be separate?
This is closer to how a human reviewer thinks. A reviewer doesn't just decide that a clip "feels bad"; they look for a reason: the camera failed, the task stopped early, the object fell out of reach, the action labels overlap. Each reason needs different evidence.
For the VLM checks, we decode the video, sample frames across time, add explicit final-tail frames, and pack everything into labeled contact sheets. That changed the output from a vague pass/reject decision into a specific claim tied to visible evidence.
When Failure Only Appears at the End
Sensor Issue was the first place where focused checks with explicit evidence clearly helped.
In the human labels, many rejected clips were marked for camera artifacts: green frames, pixelation, blur, or other visual corruption. One pattern showed up repeatedly: most of the video looked fine, but the final frame, or final few frames, turned green.
A broad video judge can easily miss that. If the model samples the video internally, it may never see the final bad frame. And if the prompt says not to reject a clip for one isolated artifact, it may ignore the exact frame that explains the label.
The policy needed to be specific: an isolated green tail frame does not always mean the entire demonstration should be rejected, but it should be surfaced as Sensor Issue evidence on the affected frames or segment.
Once we made those tail frames explicit in the contact sheet and aligned the prompt with the policy, the task became much easier. The model was no longer guessing from the whole video. It was checking the exact evidence the label depended on.
See video for animation.
The First Useful Slice
The first useful target was not all robot QC. It was narrower: green frames, pixelation, and color corruption in the camera stream.
This is the kind of visible issue that can be useful in a review pipeline. The VLM contact-sheet linter preserved more passed clips than the CV-only checker, a simpler computer vision script that only looks for visual artifacts, while still catching most green, pixel, and color failures. The CV-only checker caught slightly more rejected clips, but produced more false positives.
The broader Sensor Issue category was harder. Blur and smudge are difficult to separate from normal motion, lighting, and viewpoint changes. When we widened the rule too much, both approaches became less useful.
The takeaway is not that VLMs solve QC on their own. When the failure mode is specific, the evidence is visible, and the policy is clear, a VLM can help reviewers find the right clips faster.
Other Robot Linters
We also built focused linters for incomplete demonstrations, unrecoverable mistakes, and action overlap.
These are harder than green-frame detection because they require temporal understanding. For incomplete demonstrations, the model has to compare the task goal to the final state and decide whether the demonstration actually finished. For unrecoverable mistakes, it has to notice when an object is dropped, broken, lost from the workspace, or otherwise makes the task impossible to continue.
The robot-mounted work gave us a simple recipe: match each failure to the evidence that makes it easy to spot.
Ego-camera QC followed the same idea, but the evidence was different: motion, inactivity, blur, duration, and recording consistency. That made cheap video features a better starting point for review triage.
Ego QC: When Cheap Features Beat VLMs
The ego-camera QC track looked different from robot-mounted manipulation. These were first-person collector videos, and the rules covered sensor issues, idle data, movement, scene compliance, ergonomics, and task completion.
The dataset is larger and the best model is not a VLM:
| Ego QC item | Value |
|---|---|
| Labeled clips | 9,145 |
| Latest held-out test clips | 1,777 |
| Current best model | LightGBM |
| Final features | 120 |
| Out-of-time AUC | 0.8003 |
We started with a large feature inventory: 6,124 candidate signals.
The most useful families were not the fanciest encoders. They were brightness-calibrated blur statistics, cheap temporal features, pose and hand trajectories, EfficientNet centroid/kNN distances, and category-specific anchor distances.
| Feature family | What helped |
|---|---|
| Clip-level cheap CV | Duration, frame differences, optical flow, pause runs, scene-change cadence. Duration alone solved Pace Too Slow at AUC 0.96. |
| Brightness-calibrated blur | Sharpness by brightness bin, centroid drift, and blob persistence. This was the top family by LightGBM gain. |
| MediaPipe pose and hands | Keypoint visibility, hand positions, trajectories, and end-of-clip pose snapshots. |
| EfficientNet-B0 embeddings | Frozen ImageNet features, especially fold-safe kNN distances to good/bad centroids. |
| Cheap-anchor distances | Distances from each clip's CV stats to per-category good/bad anchors. |
Several plausible upgrades did not help:
| Attempt | Outcome |
|---|---|
| DINOv2 and OpenCLIP as stronger image encoders | No improvement over EfficientNet centroid features; slower extraction. |
| VLM prompts over tail and uniform frames | Delta AUC only +0.0008; redundant with cheap features. |
| Collector-history features | Strong alone, harmful combined; likely shortcut learning. |
| Stacked meta-features and learned latents | Within noise or worse out-of-time. |
The lesson was the same as in robot-mounted QC, but the winning evidence was different. For ego-camera videos, many failures are about motion cadence, blur, hands, duration, and frame stability. Cheap measurements see those directly.
The best QC model is not always the most general model. It is the model whose inputs match the failure mode.
Some Failures are much harder to catch than others.
A single accuracy number hides the most useful thing about a QC model: the failure modes are not equally hard. They line up on a ladder, from failures that are basically a number, to failures where you have to watch and understand the whole video.
Suppose we let a model auto-approve the clips it is most confident are good, sending them straight downstream with no human review, and route only the rest to QC. Call the share it auto-approves the coverage. Higher coverage frees more reviewer time (QC is the pipeline's bottleneck, nothing moves until a clip is cleared), but it also lets more bad clips ride along into the first approved pile. The question is which ones slip through.
At the bottom are failures that are really just a number. "Nothing is happening" means too few distinct frames, so count them; "pace too slow" means the demonstration is simply too slow for the task. Cheap frame-difference and pixel-variation checks settle these, and the model catches essentially all of them even when we auto-approve aggressively.
One step up is hands position, is the collector's hand where it should be? A raw pixel statistic won't tell you, but mature hand-tracking models will, so this stays easy: only 4% missed at half coverage.
Body support sits in the middle: flagging when a collector braces or leans on something for support. That takes some sense of what the body is doing across the clip, more than a pixel statistic, far less than judging a whole task. The model is right more often than not, but misses climb fast as we automate harder (17% at half coverage, 52% at 70%).
At the top is the illogical demonstration, deciding the clip doesn't make sense as a task at all. There's no shortcut: you have to watch and understand the entire video. A collector who sprays a knife instead of sharpening it produces perfectly normal motion, lighting, and duration, nothing in the pixels is wrong.
So coverage is a dial, not a free lunch. The more we auto-approve, the more of the harder categories leak through and need to be rejected afterwards.
Working Signals for Faster Human Review
The takeaway is not “let the model reject clips on its own.” It is that the model can surface specific, evidence-backed issues so humans can review the right clips faster.
For robot-mounted manipulation data, the strongest current signal is the green/pixel/color Sensor Issue linter. It is a good candidate for human-review flagging because it catches many visible camera artifacts while preserving most good clips. The incomplete-demonstration and unrecoverable-mistake VLM linters are also useful, but in a narrower way: they can build high-precision review queues when they fire, not replace human QC. None of these checks should automatically reject clips without human confirmation.
For ego-camera QC, the LightGBM model is best viewed as a ranked triage layer. Cheap video features provide the first pass, drift monitoring helps catch shortcut learning, and category-specific review handles ambiguous cases. Humans still adjudicate the long tail. Across both data streams, the pattern is the same:
Across both data streams, the pattern is the same:
- Make the evidence explicit.
- Break broad QC into concrete failure modes.
- Tune thresholds per rule.
- Preserve good data aggressively.
- Route uncertain or high-impact decisions to humans.
- Audit for shortcuts before trusting the curve.
Why This Matters
Robot learning is usually discussed in terms of models and data scale. But "more data" is only helpful if the data is usable. If bad demonstrations enter training, they do not just add noise. They teach the wrong invariants: that green final frames are normal, that incomplete tasks are acceptable, that a failed grasp can be part of the policy, that camera artifacts are part of the world.
QC is therefore not a cleanup step after data collection. It is part of the learning system.
The most encouraging result from these experiments is not that one model solved quality control. It did not. The encouraging result is that QC becomes tractable when we stop treating it as a vague judgment and start treating it as a set of observable physical failure modes.
For physical intelligence, that is the pattern we expect to keep seeing. The hard part is rarely the label name. The hard part is building the measurement that makes the label real.
Interested in collaboration or using Scale’s Physical AI data engine? Contact us at [email protected].
If you use or build on our work, please cite: Choghari, Jade, Sansone, Agustin, McCullough, Don, Sivapurapu, Mouli. (2026). From Video Review to Measurement: Building Better Robot Data QC.