Jan 2027
18 Mon
19 Tue
20 Wed
21 Thu
22 Fri 09:00 AM – 06:00 PM IST
23 Sat 09:00 AM – 06:00 PM IST
24 Sun
Abhishek Nair
@nairabhishek73
Submitted Oct 11, 2026
Submission type:
If talk/session proposal, showcase, or poster
Session Description:
Digital Green’s FarmerChat, an AI advisory app, has received about 8 million messages from farmers. The app answers each question as it comes in. Research and program teams need something else: a view of what farmers ask across crops, regions and seasons, and whether the service meets those needs. Dalgo, A project tech4dev initiaitive and Digital Green set out to tag messages against a research taxonomy covering domain, category, intent and crop or livestock. Rather than sending everything to an LLM, we tested a deterministic-first approach on two samples of 100,000 and 64,000 messages. Each message passes through existing source tags, then keyword matching against a controlled vocabulary, then a sentence transformer with strict thresholds, then rule filters. Only what remains goes to an LLM. Before scaling to the full dataset, we set three criteria: accuracy, cost and coverage. About 93% of messages were handled without an LLM, and LLM spend was about $28 for roughly 12,000 messages, including test runs.
This talk is about what surprised us, and what still doesn’t work. The LLM was not reliably more accurate: on crop and livestock names it over-captured, forcing in names outside the reference list, and a curated keyword list often did better. Where methods disagreed on task labels, the cause was usually a gap in the taxonomy, not the model. The easiest fix was usually the taxonomy sheet, not a bigger model. The transformer earned its place as a filter rather than a classifier, and its task and entity thresholds (0.25 vs 0.60) had to be set very differently. We will also be open about the tail: the residue only an LLM could label, messages dropped as out of context, non-English text that was translated or stripped at clean-up, and the error rate inside the 93% that we cannot yet fully verify. We close on why this recipe fits any development-sector service sitting on message logs, such as health hotlines, education chatbots and grievance systems, at a fraction of the assumed cost.
Key Takeawayss:
Target Audience:
Analysts and data teams at NGOs, social enterprises and government programs sitting on large volumes of free text, such as chatbot and helpline logs, open survey responses or grievance tickets, with a small budget. Also practitioners working out how to evaluate text classification when there is no labelled ground truth.
Bios (To be confirmed)
Abhishek Nair is a senior product manager at Dalgo, Project Tech4Dev’s open-source data platform used by NGOs to consolidate, transform and visualise program data.
Pradeep Kaushik is the Director of Engineering at Dalgo and a Fractional CxO with Project Tech4dev. He also serves as Partner and Grants Committee Member at AVPN.
This work was led by Anusha Ganapati (Tech4Dev) with Jona Repishti and Eric (Digital Green).
Draft Slides: https://docs.google.com/presentation/d/1buZSHT4LCQsDNZlqNK5kaKR8xgdH0ocsr2ZM0eZih_0/edit?usp=sharing
What don’t you know yet, or what do you want help with? This can be an open question, unresolved challenge, missing dataset, technique you are unsure about, implementation blocker, or a place where you want critique.
Match making tags -
Topic/domain tags - data for good, evaluation, agriculture
Hosted by
{{ gettext('Login to leave a comment') }}
{{ gettext('Post a comment…') }}{{ errorMsg }}
{{ gettext('No comments posted yet') }}