Anthill Inside 2019

A conference on AI and Deep Learning


Bandit algorithms to Reduce Cognitive Load on Customer Care Agents (Paper accepted for the demo track at SIGIR-2019)

Submitted by Hrishi Ganu (@blah) on Tuesday, 26 March 2019

Preview video

Session type: Full talk of 40 mins


We will describe a human-in-the loop system, AgentBuddy that is helping Intuit improve the quality of search it offers to internal Customer Care Agents (CCAs). AgentBuddy aims to reduce the cognitive effort on part of the CCAs while at the same time boosting the quality of our legacy federated search system. It addresses two key pain points 1)Given several candidate query answering mechanisms, how to select the right mechanism given a question and 2)Having retrieved a set of lengthy documents how to help the agent zoom in on the content most important for the question at hand. We address #1 using an elegant approach for principled exploration based on bandit algorithms and for #2 we have several models based on supervised and unsupervised learning. Since this is a real world system deployed on AWS we will also discuss practical challenges in scaling and how we overcame them.


  • Business Problem
  • Approach
  • Formulation
  • ML algorithms used
  • Architectural challenges
  • Scaling up
  • Latency issues and sane fall-backs
  • Validation
  • Ground truth labels
  • Merics and measurement
  • Business Impact
  • Takeaways for Data Science practice

Speaker bio

Hrishi has been a regular speaker at Anthill and delivered a full-length session on “Building and driving adoption for a robust semantic search system”
Hrishi did his Master’s from Indian Institute of Science (IISc), Bangalore in 2005 where he worked on Computer Vision for studying atomization in Cryogenic Rocket Engines. Post that he did a full-time PGDM from IIM-Kozhikode. He has been working in the ML/Analytics space for over 11 years and has had long stints at Amazon Core ML and at Mu Sigma before joining Intuit’s IAT team. At Intuit, he’s working on NLP with a focus on creating algorithms that are robust to noise in user input.
Aside of work he spends time playing with his 4-year-old daughter and in solving puzzles.


Preview video


  • Anwesha Sarkar (@anweshaalt) 5 months ago

    Thank you for submitting the proposal. Submit your slides and preview video by 20th April (latest) it helps us to close the review process.

  • Zainab Bawa (@zainabbawa) Reviewer 4 months ago


    We’ll need to see draft slides – by 27 May – which help us understand your thinking to assess the fit of your proposal for The Fifth Elephant. Since this proposal has been submitted a while ago with no further updates, you also have to let us know if your plans have changed and if you want us to move your proposal to future editions of Rootconf and/or The Fifth Elephant.

    Other comments for your slides, when you draft them:

    1. Explain why the decision to go with Bandit algorithms? which other options were considered? Or why does Bandit algorithm serve your use case and nothing else? what other similar cases does use of Bandit algorithms make sense? What are the limitations?
    2. The concepts used Cognitiva Load, CCA, etc which will have to be explained enough for the audience to get the context. Else, participants will be searching for definitions online.
    • Hrishi Ganu (@blah) Proposer 3 months ago

      @zainab I’ve updated the submission bhy including 1)link to demo video and 2)added a slide deck

  • Abhishek Balaji (@booleanbalaji) Reviewer 3 months ago

    We’re considering this for Anthil Inside, since the talk is theory driven. Anthill Inside is currently postponed hence we’re parking this for now. We’ll get back to you when we have dates confirmed.

  • Mithun Ghosh (@mithunghosh) 3 months ago (edited 3 months ago)

    Hi @booleanbalaji this has enough depth on practical side of implementation hence it is suitable for fifth elephant too.
    This is a system working in production, designed from scratch. We can work together on slides to make it suitable for fifth elephant by adding more of system side of implementation details. Let me know your opinion.

    • Abhishek Balaji (@booleanbalaji) Reviewer 2 months ago

      Hi Mithun,

      Here’s some more feedback:

      • the deck does not look cohesive. the initial motivation of diversity of customers etc does not clearly relate to the later parts of the talks.

      • in general, there is too much text on the slides.

      • supervised learning two slides can be combined into one with a simple message.

      • technical merit: contextual bandit setting has its own limitation of when and how it converges. the author does not seem to talk about it.

      • no need to stress too much sigir demo publication. if it were a full paper, it holds some merit to show in the slides.

      • burying the head: the actual problem and solution of generating a response and extracting the right part from the right article is coming too late.

      • not much attention seems to be paid to describe this problem and explain the solution

      • later slides have too much text. not good.

      • the author is suddenly talking about LDA where as contextual bandit has taken back seat. the overall solution presented in the slides lacks clarity.

Login with Twitter or Google to leave a comment