The Fifth Elephant 2019

Gathering of 1000+ practitioners from the data ecosystem

Tickets

Developing a bot that can answer support queries and aid in decision making with analytics

Submitted by Varun Nathan (@varunn) on Monday, 15 April 2019


Preview video

Session type: Discussion Session type: Full talk of 40 mins

Abstract

Responding to repetitive queries from customers can overload the support team. Developing the capability to handle such repetitive queries can significantly enhance the productivity of support agents and they can utilise their time in resolving problems that are more challenging and involved.
This talk will focus on the modelling approach that we at Freshworks took to develop a bot that has the ability to understand natural language, respond to customer messages and aid agents in understanding the kind of content that needs to be created to enable self service. In addition, we’ll also throw light on deployment and challenges involved in scaling a system such as this.

Outline

  1. Motivation - Why should this problem be solved
  2. Problem Statement - Definition of the end objective
  3. Modelling Methodology - An overview of the available data, ML algorithms and validation metrics that define success
  4. Deployment in Production - An overview of the pipelines for data extraction, training models, serving predictions and feedback consumption
  5. Challenges - scaling and dealing with streaming data

Requirements

No Requirements

Speaker bio

I joined the data science team of Freshworks in May 2018 and since then have been working on problems in the NLP space. Prior to this, I have experience of solving problems including risk modelling, churn modelling, life time customer value prediction, campaign analytics, creating networks from social data to decipher relationships and market mix modelling.

Slides

https://www.slideshare.net/VarunNathan/ml-framework-for-autoresponding-to-customer-support-queries-150165427

Preview video

https://www.youtube.com/watch?v=4T1oW6kAbgA&feature=youtu.be

Comments

  • Anwesha Sarkar (@anweshaalt) Reviewer 5 months ago (edited 5 months ago)

    Thank you for your submission. Submit your preview video and slides by 22nd March(latest). It helps us to provide a fair evaluation to the proposal and close the review process.

  • Zainab Bawa (@zainabbawa) Reviewer 5 months ago

    The focus of this proposal is unclear – what is it that the talk is trying to do and say? Who is the intended audience? We need draft slides and a preview video to complete full evaluation.

    • Varun Nathan (@varunn) Proposer 5 months ago

      The talk will focus on how a chatbot was modelled to respond to customer queries. The intended audience for this talk include people who are either applying data science techniques to solve problems involving text or are interested in learning about the same.
      I’m working on the draft slides and will turn it in by the end of this week.

  • Varun Nathan (@varunn) Proposer 4 months ago

    @Zainab,@Anwesha - I have shared the link to the presentation deck. Hope this aids in the evaluation.

    • Zainab Bawa (@zainabbawa) Reviewer 4 months ago

      The preview video hasn’t been uploaded yet. This is also required for the evaluation.

    • Zainab Bawa (@zainabbawa) Reviewer 4 months ago

      Hello Varun,

      We got this feedback from the reviewers of your proposal:

      1. This is a good application for big data and data science. There are commercial companies in this space to prove that the application is relevant to the industry.
      2. The data engineering architecture and model deployment parts of the proposal are not relevant. Rather, chatbots is an area searching for killer apps and commercial adoption. So this part of the proposal is interesting and novel.
      3. The abstract is clear but the slides are not. Slides need more work but the raw material is good.
      4. There is focus on engineering implementation. Participants at The Fifth Elephant will find it interesting to hear about the missteps and learnings in finding, cleaning and processing the data sources, scaling challenges and adoption by the consumers. If the focus is moved to “lessons learnt and battle scars”, then only will the proposal be interesting.

      Based on the above feedback, you have to submit revised and detailed slides, along with a preview video where you pitch the talk, within 7 days. This will help us make a final or a near-final decision.

  • Venkata Pingali (@pingali) 4 months ago (edited 4 months ago)

    Hi! Varun,

    I would echo previous review that emphasized the learning -
    non-obvious aspects that audience who didnt go through the
    experience wouldnt know. This will help audience understand
    what it will take to do this. In particular:

    1. It will help to understand the behavior of the work product.
      How many of the responded requests were valid/useful? How is
      that related to the “Helpful” metric? Any thoughts on elements
      that the approach was missing that resulted in unrelated/unhelpful
      suggestions?

    2. Surprised to find no mention of labeling or addition of context.
      Was that a conscious choice?

    3. Support questions keep changing over time. How do you keep the
      model relevant over time? How much effort would that incur?

    Thanks!

    • Varun Nathan (@varunn) Proposer 4 months ago

      Thanks, Zainab and Venkata, for your comments. I’m working on the deck to accomodate the thought process of the reviewers. I will have the updated deck and the preview video uploaded by today EOD and then drop a note. Thanks!

      • Varun Nathan (@varunn) Proposer 4 months ago

        @zainab,@Venkata - The deck has been updated based on your comments. I have also uploaded a preview video for the talk. Thanks!

        • Zainab Bawa (@zainabbawa) Reviewer 4 months ago

          The preview video link is not working. It says that the video is not available.

          • Varun Nathan (@varunn) Proposer 4 months ago

            Hi Zainab, I have changed the video settings. You should be able to view it now. Please check and let me know. Thanks.

  • Zainab Bawa (@zainabbawa) Reviewer 4 months ago

    Thanks for updating the slides. They definitely contain more context and information. We will evaluate this and update you on the decision, here.

  • Venkata Pingali (@pingali) 3 months ago

    Hi! Varun,

    Had a chance to look at the updated slides. You dont appear to have
    addressed my feedback. Please do look at it.

    I see two further challenges - mainly with the presentation style. The
    audience will benefit from the understanding you have gathered in the
    process:

    1. “Linear” narrative style (verbose, detail oriented at each step).
      It can get tedious. Suggest using the slides with the modeling
      flow and show the ‘growth’ and thinking across multiple slides.

    2. A number of questions remain unanswered in my mind. For example,
      approximately 50% of the user queries could be addressed (what do we
      know about the rest?). Of them 15-20% were marked helpful. Rest were either
      not helpful or there were no responses (why? what do we know about them?
      How did it impact the experience?) Can you make the results more
      interpretable?

    -Venkata

    • Varun Nathan (@varunn) Proposer 3 months ago (edited 3 months ago)

      Hi Venkata,

      Thanks for your comments.
      1. I’ll take a look at the deck to see as to how your 1st point can be accomodated.
      2. There’s a slide in appendix on “why some suggestions are not helpful to the user” which talks about your 2nd point. I guess if the style is modified, this concern will be addressed.

      Let me take a closer look and then get back.

      Thanks,
      Varun

      • Varun Nathan (@varunn) Proposer 3 months ago

        Hi Venkata,

        Once again, thanks for your comments. I have changed the ordering of slides in the deck in addition to adding some points to answer your 2nd question. Please take a look and let me know your thoughts.
        Thanks!

  • Zainab Bawa (@zainabbawa) Reviewer 3 months ago

    Varun, here are some more comments from review:

    1. The slides are all over the place (perhaps because the story is yet to develop). The focus is missing. Moreover, the focus has to be moved to “lessons learnt and battle scars”. Currently, the slides don’t reflect this.
    2. The slide also need more diagrams and numbers.
    3. There are some extra details which are confusing because of lack of context. For example, why mention kafka streams data in challenges if there is no prelude to it? Why mention incremental improvements in lessons if there is no backstory in previous slides?
    4. Slide 4 should be tied with slide 14 by mentioning, “these were our objectives, and in quantitative terms this is how we are attempting it.” Currently, slide 14 doesn’t make an impression.
  • Zainab Bawa (@zainabbawa) Reviewer 3 months ago

    The other way to structure this talk is breaking it down into three sections:

    1. Business value of problem
    2. Simple solution
    3. Solution metrics

    With this structure, your proposal can be a potentially good beginner level talk for anyone who wants to gets started and not use complicated solutions. It is also important to stress on the value proposition for participants since this gets missed out. This is what matters for businesses: can we answer most of the T1 tickets and unblock the agents?

    • Varun Nathan (@varunn) Proposer 3 months ago

      Thanks, Zainab. I have updated the deck in accordance with the comments given by reviewers. Please have a look and let me know your feedback.
      Thanks!

  • Abhishek Balaji (@booleanbalaji) Reviewer 3 months ago

    Hi Varun, I had a chance to look through your updated slides. Here’s the feedback:

    1) The slides still dont explain the problem statement and the solutions possible. It jumps stright to the model training part. Why would someone be interested in looking your solution? Where can they apply this approach in their line or work?

    2) It currently reads like product documentation and does not set the narrative.

    3) Can you use an example or case study of how the bot helped your team? You can do this by running through each step in the flow and then doing deep into each step.

    4) The metrics/business impact slide is very hard to understand as someone in the audience. Make this into a graph to show the trend rather than making people compare values in a table.

    5) Similarly, when you show the online/offline processing model, try to split the steps into multiple slides or use animations to reveal step by step. Otherwise such a large flow chart is overwhelming and the audience will find it hard to follow.

  • Abhishek Balaji (@booleanbalaji) Reviewer 3 months ago

    Varun, did you get a chance to work on this?

    • Varun Nathan (@varunn) Proposer 3 months ago

      Hi Abhishek,

      Apologies for my delayed response. Yes, I have made changes to the deck on the basis of your comments. Please check the updated deck and share your thoughts.
      Thanks!

      • Abhishek Balaji (@booleanbalaji) Reviewer 2 months ago

        Thanks Varun,

        The proposal is still too specific and would not provide key takeaways for the entire breadth of audience at The Fifth Elephant. We will not be able to accept this proposal for The Fifth Elephant, but will keep in mind for future events, where the audience matches the talk. Will keep you updated.

Login with Twitter or Google to leave a comment