The Fifth Elephant
The Fifth Elephant

The Fifth Elephant 2026 Winter Edition

Call for Data Problems and Proposals

All submissions
Prathamesh Kalamkar

Prathamesh Kalamkar

@prathameshkalamkar

Crowdsourcing: Disagreement is the data

Submitted Oct 11, 2026

🎤 I am submitting: To speak I have a submission for: A talk - 30-40 mins

{Submission type - choose one:

  • Talk/session proposal - 30-40 mins}

If talk/session proposal, showcase, or poster
{Describe your session in 2 paragraphs.}
We built two open datasets from Indian court judgments with 35 volunteer law students and no budget: one marking legal entities, one marking what each sentence does. Same team, same year, same domain. One needed a single annotation pass and yielded 46,545 entities. The other needed three annotators and an adjudicator, yielded 354 documents, and finished at an inter-annotator agreement of 0.59. We chose both setups by instinct and only understood afterwards what had actually decided them. This talk is about what did.
Four questions determine the shape of any labelling operation, and getting them wrong is expensive in ways that surface a year later as what looks like a model bug. I will go through those questions, nine specific things that went wrong for us and what each cost, and the handful of measurements that would have caught them early — including one that costs nothing and almost nobody does. Along the way, how Wikipedia, eBird, Common Voice, AI4Bharat and Karya answer the same questions differently. I will end on the problem I have not solved: every one of those projects has a way to be corrected after publication, and a released dataset has none. Useful to anyone who labels data, commissions labelled data, or trains on someone else’s.

{Mention 1-2 takeaways from your session.}

  1. Measure disagreement per label before you train anything. The labels your annotators argue about are the labels your model will get wrong. Agreement isn’t a quality score to report at the end; it’s an early forecast of where the model will fail.
  2. Making a confusing label more fine-grained rarely makes it less confusing. If people disagree about the concept itself, extra categories and more calibration won’t converge it. Change what you’re asking annotators for, or publish the disagreement instead of averaging it away.

{Which audiences will benefit most from your session?}

{Add your bio: who you are, where you work, and any relevant context.}
I am working as Lead Applied Research Scientist at Thomson Reuters. I have worked in open source AI solultions in Indian Legal domain.

{Add a link to draft slides, if available. PDF/PPT preferred. Please make sure comments are enabled.}

https://docs.google.com/presentation/d/1pY-Q-SidZdsWcLsjwZzj9CpfQUe3D3E7/edit?usp=sharing&ouid=104996849920819615404&rtpof=true&sd=true

{Topic/domain tags - choose 2-3 broad tags that describe your session or problem space.}
data for good, legal

All submissions

Comments

{{ gettext('Login to leave a comment') }}

{{ gettext('Post a comment…') }}
{{ gettext('New comment') }}
{{ formTitle }}

{{ errorMsg }}

{{ gettext('No comments posted yet') }}

Hosted by

The Fifth Elephant
The Fifth Elephant
Jumpstart better data engineering and AI futures