Anthill Inside 2019

A common problem when using deep neural network models for classification problems is handling out of distribution data. In such scenarios, the classifiers tend to assign the new data point to one of the known classes with high probability, which can lead to unintended and potentially harmful consequences. At MakeMyTrip (MMT), we use deep learning based NLP classifier for understanding intent of utterances received by MMT’s chatbot Myra. Here we discuss how we handle user utterances of intents for which Myra has not been trained.

Outline

• Introduction to in-sample and out-of-sample distributions when using ML models.
• Example scenario:
  • text is misclassified where true class was not present in the training data.
  • Possible approaches - creating an ‘other’ class requires a lot of training data - not feasible
• Proposed approach:
  • learn a distributed representation of the target (embeddings) instead of discrete classes.
  • compare the predicted distribution of in-sample versus out-of-sample examples.
• Create a 2-step classifier which first decides to classify the example or not.
• Present final results about how such examples are handled in Myra and how it eases the discovery of new intents.

Speaker bio

I am Akhil Lohia, data scientist at MakeMyTrip, where we’re developing Myra, MakeMyTrip’s task bot for assisting millions of our customers with post sale issues such as cancelling & modifying bookings, enquiring about flight status, baggage limits, refund status etc. Here I am presenting a very commonly faced problem in machine learning based classifiers, where such models can give unexpected results for out of sample data, and one approach to deal with this problem based on recent research. I obtained my BS in Economics from IIT Kanpur and MS in Data Science from Barcelona Graduate School of Economics. I have worked on research projects involving RCTs and estimation of structural models related to Indian demographics before joining MakeMyTrip.

Links

• www.linkedin.com/in/akhil-lohia

Slides

https://docs.google.com/presentation/d/1669vbEH_2bAldARdA7vt0UyqPSTp6PkEAKLAJQ_s5UI/edit?usp=sharing