The Fifth Elephant 2026 Winter Edition

Submission type:

• Talk/session proposal - 30-40 mins
• Showcase - new ideas - 20 mins

If talk/session proposal, showcase, or poster

Session Description:
Digital Green’s FarmerChat, an AI advisory app, has received about 8 million messages from farmers. The app answers each question as it comes in. Research and program teams need something else: a view of what farmers ask across crops, regions and seasons, and whether the service meets those needs. Dalgo, A project tech4dev initiaitive and Digital Green set out to tag messages against a research taxonomy covering domain, category, intent and crop or livestock. Rather than sending everything to an LLM, we tested a deterministic-first approach on two samples of 100,000 and 64,000 messages. Each message passes through existing source tags, then keyword matching against a controlled vocabulary, then a sentence transformer with strict thresholds, then rule filters. Only what remains goes to an LLM. Before scaling to the full dataset, we set three criteria: accuracy, cost and coverage. About 93% of messages were handled without an LLM, and LLM spend was about $28 for roughly 12,000 messages, including test runs.

This talk is about what surprised us, and what still doesn’t work. The LLM was not reliably more accurate: on crop and livestock names it over-captured, forcing in names outside the reference list, and a curated keyword list often did better. Where methods disagreed on task labels, the cause was usually a gap in the taxonomy, not the model. The easiest fix was usually the taxonomy sheet, not a bigger model. The transformer earned its place as a filter rather than a classifier, and its task and entity thresholds (0.25 vs 0.60) had to be set very differently. We will also be open about the tail: the residue only an LLM could label, messages dropped as out of context, non-English text that was translated or stripped at clean-up, and the error rate inside the 93% that we cannot yet fully verify. We close on why this recipe fits any development-sector service sitting on message logs, such as health hotlines, education chatbots and grievance systems, at a fraction of the assumed cost.

Key Takeawayss:

• Start with the cheapest, most transparent method and measure coverage stage by stage. Send only the residue to an LLM. Against a controlled vocabulary, keyword matching can beat an LLM on accuracy as well as cost.
• When two methods disagree, look at your taxonomy before your model. Disagreements are a free audit of your categories.

Target Audience:
Analysts and data teams at NGOs, social enterprises and government programs sitting on large volumes of free text, such as chatbot and helpline logs, open survey responses or grievance tickets, with a small budget. Also practitioners working out how to evaluate text classification when there is no labelled ground truth.

Bios (To be confirmed)
Abhishek Nair is a senior product manager at Dalgo, Project Tech4Dev’s open-source data platform used by NGOs to consolidate, transform and visualise program data.
Pradeep Kaushik is the Director of Engineering at Dalgo and a Fractional CxO with Project Tech4dev. He also serves as Partner and Grants Committee Member at AVPN.

This work was led by Anusha Ganapati (Tech4Dev) with Jona Repishti and Eric (Digital Green).

Draft Slides: https://docs.google.com/presentation/d/1buZSHT4LCQsDNZlqNK5kaKR8xgdH0ocsr2ZM0eZih_0/edit?usp=sharing

What don’t you know yet, or what do you want help with? This can be an open question, unresolved challenge, missing dataset, technique you are unsure about, implementation blocker, or a place where you want critique.

• Accuracy at scale. We can measure coverage precisely, but accuracy only through spot checks and agreement between methods. What’s the cheapest credible way to estimate error across the 93% without hand-labelling thousands of messages? Can method disagreement serve as an error signal?
• Languages. Messages arrive in many languages. We classify translated text, and some non-English text is stripped at clean-up . What are we losing, and is matching in the original language worth the effort?
• Keeping it current. New crops, seasons and question types keep appearing. How do we keep the taxonomy and thresholds current without re-tuning by hand?
• What gets dropped. Messages with more than one intent, and messages dropped as “out of context”: how much real signal sits there?

Match making tags -

• I can help with: experience, ideas
• I need help with: technique, critique
• I’d like to meet: researchers / practitioners / domain experts / tool builders / dataset owners

Topic/domain tags - data for good, evaluation, agriculture