The Fifth Elephant
The Fifth Elephant

The Fifth Elephant 2026 Winter Edition

Call for Data Problems and Proposals

All submissions

Kartik

@kwatrakartik

Before You Trust That Survey, Trace the Data-Generating Process

Submitted Oct 25, 2026

🎤 I am submitting: To speak I have a submission for: A talk - 30-40 mins

We often analyse data that is already generated, but the data generating process plays a huge role in how reliable the conclusions from the data are. This talk is about the issues in conducting surveys (a data-generating process) and some of the ways you can pre-empt them or address them post facto.

Every stage of a survey, from questionnaire design and sampling to fieldwork, shapes data quality, which shows up in two ways: the precision of the estimates and their bias. I will focus on fieldwork, where many problems arise that the design never accounts for, and that are easily missed unless you look for them. Some of these can be spotted in paradata (data about the process of collecting each response): how the questionnaire was canvassed (e.g. the flow of questions), the time of day of interviews, interview durations, location patterns, interviewer and team-level patterns, and deviations from the prescribed interviewing methodology. Others show up in the response data itself, such as suspiciously low variation within an interviewer’s work, answers that contradict each other, or age heaping (ages clustering on round numbers). For each, I’ll show how to spot it and what,if anything, can be done about it, both during fieldwork and after. The methods range from statistical measures of reliability and consistency to patterns in paradata and responses that I’ve learned to watch for in my survey work.

I will be using synthetic data that mimics issues I’ve found in real fieldwork to illustrate the points.

Takeaways:
For practitioners commissioning surveys:

  1. How to pre-empt data issues in in-person surveys, and how to detect them after the fact.
  2. Ways to address these issues before, during and after fieldwork using paradata and the response data, including writing paradata requirements into the survey contract and monitoring fieldwork as it happens.

For consumers of survey data:

  1. What paradata to request before analysing survey data, where available.
  2. How to develop a critical eye for survey data.

About me:
Kartik Kwatra
Data Scientist
I’ve been doing end-to-end data work at How India Lives for over 8 years, with a focus on public data. I have worked on pan-India surveys, rural and urban, across every stage: questionnaire design, sampling, overseeing fieldwork, and analysis and reporting.
Linkedin

I’d like to meet: researchers / practitioners / domain experts

Topic/domain tags -
data quality, survey data, paradata

Availability before the event: yes

All submissions

Comments

{{ gettext('Login to leave a comment') }}

{{ gettext('Post a comment…') }}
{{ gettext('New comment') }}
{{ formTitle }}

{{ errorMsg }}

{{ gettext('No comments posted yet') }}

Hosted by

The Fifth Elephant
The Fifth Elephant
Jumpstart better data engineering and AI futures