Jan 2027
18 Mon
19 Tue
20 Wed
21 Thu
22 Fri 09:00 AM – 06:00 PM IST
23 Sat 09:00 AM – 06:00 PM IST
24 Sun
Mai Diab
Submitted Oct 10, 2026
Talk/session proposal - 30-40 mins
Description
In a robot-learning dataset built from first-person video, idle footage is the most expensive footage there is: it costs the same to store, move and annotate as footage of real work, and it teaches a policy nothing. So we set out to remove it. A vision-language model described each 2–5 second window of head-camera video — which hands were visible and what they were doing — and deterministic rules over those descriptions split the timeline into working and not-working. Auditable, cheap, and supposedly the easy part. The idle labels came back confident and wrong, and rounds of prompt revision did not converge. The prompts were not the problem. The label was not identifiable from the input: we were asking a question whose answer was not in the window the model was shown.
Idle is not a property of a moment; it is a relation between a moment and a job. Holding food is idle on a toy assembly line and work in a kitchen. Holding a phone is idle on a production line and work at a phone-repair bench. A model can read the hands perfectly and still be unable to say whether what they are doing counts. We therefore added a second layer that reads the whole clip, infers the primary task first, and only then labels departures from it. That fixed the food and the phone. This has alimitation: the second layer never sees pixels, only the first layer’s sentences. Where a caption is thin or wrong, the judgement built on it is wrong in a new way, with full confidence. The two layers now disagree on more than a quarter of the timeline, and we have nothing to arbitrate between them. This talk is about where that line falls, how we found it, and what we still cannot measure.
1–2 takeaways:
Which audiences will benefit most:
Data and ML engineers who build structured labels on top of a generative model — especially anyone who splits a long input (video, audio, logs, transcripts) into chunks and asks a question per chunk. Chunking is a silent decision about what context the model is allowed to have, and some labels cannot survive it. Also useful to anyone writing an annotation guideline: the trap opens the moment a definition contains the word “relevant”. No computer vision background is needed.
Draft slides
Not yet. I will share a draft with comments enabled ahead of the review session.
What I don’t know yet / want help with
Matchmaking tags
I can help with: experience · technique · critique
I need help with: ideas · critique · experience
I’d like to meet: practitioners · researchers · dataset owners
Topic/domain tags
computer vision · vision-language models · annotation · evaluation · data quality
Availability: I am available before the event for light-touch matching/facilitation
Dr. Mai Diab
Bio
Mai Diab is Director of Applied Research & Physical AI at Humyn Labs, where she leads the team building first-person video datasets and their annotation for robotics and vision-language-action models. She holds a PhD in computer vision, and has spent more than ten years in research and university teaching; she is a visiting lecturer at the Faculty of Computers and Artificial Intelligence, Benha University. Her current work is mostly on building models to solve problems that only come from in the wild datasets.
Hosted by
{{ gettext('Login to leave a comment') }}
{{ gettext('Post a comment…') }}{{ errorMsg }}
{{ gettext('No comments posted yet') }}