Jan 2027
18 Mon
19 Tue
20 Wed
21 Thu
22 Fri 09:00 AM – 06:00 PM IST
23 Sat 09:00 AM – 06:00 PM IST
24 Sun
Submitted Oct 11, 2026
{Submission type - choose one:
If talk/session proposal, showcase, or poster
{Describe your session in 2 paragraphs.}
We built two open datasets from Indian court judgments with 35 volunteer law students and no budget: one marking legal entities, one marking what each sentence does. Same team, same year, same domain. One needed a single annotation pass and yielded 46,545 entities. The other needed three annotators and an adjudicator, yielded 354 documents, and finished at an inter-annotator agreement of 0.59. We chose both setups by instinct and only understood afterwards what had actually decided them. This talk is about what did.
Four questions determine the shape of any labelling operation, and getting them wrong is expensive in ways that surface a year later as what looks like a model bug. I will go through those questions, nine specific things that went wrong for us and what each cost, and the handful of measurements that would have caught them early — including one that costs nothing and almost nobody does. Along the way, how Wikipedia, eBird, Common Voice, AI4Bharat and Karya answer the same questions differently. I will end on the problem I have not solved: every one of those projects has a way to be corrected after publication, and a released dataset has none. Useful to anyone who labels data, commissions labelled data, or trains on someone else’s.
{Mention 1-2 takeaways from your session.}
{Which audiences will benefit most from your session?}
{Add your bio: who you are, where you work, and any relevant context.}
I am working as Lead Applied Research Scientist at Thomson Reuters. I have worked in open source AI solultions in Indian Legal domain.
{Add a link to draft slides, if available. PDF/PPT preferred. Please make sure comments are enabled.}
{Topic/domain tags - choose 2-3 broad tags that describe your session or problem space.}
data for good, legal
Hosted by
{{ gettext('Login to leave a comment') }}
{{ gettext('Post a comment…') }}{{ errorMsg }}
{{ gettext('No comments posted yet') }}