The Fifth Elephant 2026 Winter Edition

Talk/session proposal - 30-40 mins

Description

In a robot-learning dataset built from first-person video, idle footage is the most expensive footage there is: it costs the same to store, move and annotate as footage of real work, and it teaches a policy nothing. So we set out to remove it. A vision-language model described each 2–5 second window of head-camera video — which hands were visible and what they were doing — and deterministic rules over those descriptions split the timeline into working and not-working. Auditable, cheap, and supposedly the easy part. The idle labels came back confident and wrong, and rounds of prompt revision did not converge. The prompts were not the problem. The label was not identifiable from the input: we were asking a question whose answer was not in the window the model was shown.

Idle is not a property of a moment; it is a relation between a moment and a job. Holding food is idle on a toy assembly line and work in a kitchen. Holding a phone is idle on a production line and work at a phone-repair bench. A model can read the hands perfectly and still be unable to say whether what they are doing counts. We therefore added a second layer that reads the whole clip, infers the primary task first, and only then labels departures from it. That fixed the food and the phone. This has alimitation: the second layer never sees pixels, only the first layer’s sentences. Where a caption is thin or wrong, the judgement built on it is wrong in a new way, with full confidence. The two layers now disagree on more than a quarter of the timeline, and we have nothing to arbitrate between them. This talk is about where that line falls, how we found it, and what we still cannot measure.

1–2 takeaways:

1. A locality test for any label you ask a model to produce. Write the label’s definition using only what is inside the window the model sees. If you cannot — if the definition needs words like “relevant to”, “beyond what is needed” or “part of the task” — no prompt will fix it, because the information is not in the input. That is an architecture decision about context, not a prompt-engineering one. We will show exactly where this line fell in our pipeline and what it cost to find.
2. A judgement layer is capped by the description layer beneath it. Our second pass can never contain more information than the captions it reads. The counter-intuitive consequence: the fastest way to improve the hard problem (what is idle) was to improve the layer we had already declared finished (how hands are described). Chunk boundaries, blank captions and vague object names all showed up first as idle-detection errors, long before anyone called them captioning errors.

Which audiences will benefit most:
Data and ML engineers who build structured labels on top of a generative model — especially anyone who splits a long input (video, audio, logs, transcripts) into chunks and asks a question per chunk. Chunking is a silent decision about what context the model is allowed to have, and some labels cannot survive it. Also useful to anyone writing an annotation guideline: the trap opens the moment a definition contains the word “relevant”. No computer vision background is needed.

Draft slides
Not yet. I will share a draft with comments enabled ahead of the review session.

What I don’t know yet / want help with

1. How do you estimate the accuracy of a label with no ground truth? Our working/idle split has never been checked against human annotation. We have the windows where rules and model disagree, and adjudicating those is far cheaper than labelling a random sample. But disagreement sampling only finds where two estimators differ; it is blind to the errors they share, which are exactly the errors a shared caption layer would produce. My current plan is a small stratified random audit alongside the disagreement set. I would like to be argued with on whether that is enough.
2. What does reproducibility mean when the annotator is stochastic? Two runs of the same prompt on the same video agree on only 86% of seconds. That is a test-retest reliability problem: any change smaller than the run-to-run disagreement cannot be called a regression or an improvement. I have not found a satisfying way to version a dataset whose source moves under it, and I would like to hear from anyone who has.

Matchmaking tags
I can help with: experience · technique · critique
I need help with: ideas · critique · experience
I’d like to meet: practitioners · researchers · dataset owners

Topic/domain tags
computer vision · vision-language models · annotation · evaluation · data quality

Availability: I am available before the event for light-touch matching/facilitation

Dr. Mai Diab
Bio
Mai Diab is Director of Applied Research & Physical AI at Humyn Labs, where she leads the team building first-person video datasets and their annotation for robotics and vision-language-action models. She holds a PhD in computer vision, and has spent more than ten years in research and university teaching; she is a visiting lecturer at the Faculty of Computers and Artificial Intelligence, Benha University. Her current work is mostly on building models to solve problems that only come from in the wild datasets.