The Fifth Elephant 2019

The eighth edition of India's best data conference

Participate

ADAM - Bootstrapping a Deep NN based Sequence Labeling Model with minimal labeling

Submitted by Shashank Jaiswal (@shashank94) on Monday, 8 April 2019


Preview video

Session type: Lecture Section: Full talk Technical level: Intermediate Session type: Lecture Session type: Full talk of 40 mins

View proposal in schedule

Abstract

Deep Learning based models have achieved high accuracy on Named Entity Recognition tasks for natural language datasets. However, their efficacy on practical domain-specific data, like product titles, is often subpar due to several challenges - 1) labeled data is scarce or unavailable; 2) noise in the form of spelling errors, missing tokens, abbreviations etc.; 3) variance in structure (as it is not a natural language, hence no grammar); 4) manual labelling is costly. In this talk, I will talk about how at Clustr, we leveraged an existing sparse Knowledge Graph to generate a set of weak labeled seed data and used it to bootstrap a deep Recurrent Neural Network-based sequence labelling model. Further, we build upon the concepts of Active Learning to iteratively train our model with minimal amount of manual labelling. The key takeaways of the talk would be 1) how to deal with similar problems with availability of training data (even with different category of data e.g. images, sensor data etc.), 2) understanding why Deep neural network architecture can generalise very easily if used correctly; and 3) we would describe how active learning is a promising paradigm in building practical machine learning-based solutions to domain-specific problems riddled with scare labeled data

Outline

We would be presenting answers to the following…
Why just any generic approach would not have worked?
How our data source and structure left us with no previously adapted choices?
Why we the project was necessary to meet the end goals of the company?
And how did we tackle a number of problems on the way?

Company’s Goals:

Following are the primary product-goals of the company which are relevant to ADAM project.:
1)Universal Product Catalog
2)Aggregation and Market Analysis
3)Self evolving Knowledge Graph

Raw Dataset:

Introduction, Structure and The good, bad and the ugly of the data-set.

WHY ADAM (Automatic Detection and Annotation Module)… A deep NN model:

Definition,
Usecases::
1) Enrichment of Knowledge graph
2) For Analytics

Components of ADAM:

1)Smart Automatic Training Data Generation
2)State of the Art Sequence Tagging Model
3)Active Learning approach

Why the above architecture is chosen:

1)Zero ground truth and no training data available whatsoever.
2)Multi-Independent Source of data generation thus imagine the variance
3)Short representations and extremely noisy
4)Prone to Extreme human error (not bias but error!!!)

Finally details of the architecture and WHY they were necessary:

1)Smart Automatic Training Data Generation:

<>How we leveraged the structure of dataset (Stock-item and Stock-group)?
<>How we used the existing knowledge-base AKA (CREGS)?
<>How we improvised using the information from other sources like Amazon and GS1?

2)State of the Art Sequence Tagging Model:

<>Why we created our own word embeddings and how it helped us?
<>Why BiLSTM and CRF were used and why are they state of the art?
<>Why this specific architecture was needed and why anything else wouldn’t work
<>What was the accuracy and how well did the model performed?

3)Active Learning:

<>Why Active learning when we can generate labels automatically?
<>How we integrated and designed Manual annotation model ourselves?
<>How well did we reach maturity with the minimum data-points manually labelled?
<>Why extrinsic sampling or intrinsic sampling used?

Conclusion that we will showcase as per 5th Elephant:

1)How to tackle the noisy data problem in case of textual data?
2)Why a deep NN model plays an important role in generalisation?
3)Why Active Learning is a really important concept for dealing with the problem of no label data?

Speaker bio

IIT Roorkee Grad. (Batch 2017)
Data Scientist (Exp: 1.8 yrs at Clustr, Tally Analytics pvt. ltd.)
I have been a part of the Data Science team at Clustr.
I have worked on some innovative projects which involved skills on Deep-Learning, Machine Learning, complex data-structure and dynamic programming algorithms.
I have been the primary owner of ADAM project and have successfully converted it from a problem statement to working solution. I brain-stromed, coded and tackled all the problems faced while the journey of ADAM.

Links

Slides

https://www.slideshare.net/secret/Jx8awxewNlaK5n

Preview video

https://youtu.be/--Gc3bcIx2A

Comments

  • Anwesha Sarkar (@anweshaalt) Reviewer 2 months ago

    Thank you for submitting the proposal. Submit your preview video by 20th April (latest) it helps us to close the review process.

    • Shashank Jaiswal (@shashank94) Proposer 2 months ago

      Hey, I have uploaded the video too, please inform me if I need to make any modification somewhere, I will do the needful.

  • Zainab Bawa (@zainabbawa) Reviewer a month ago

    The feedback on your slides is as follows, Shashank:

    1. The problem statement outlined is very specific to Clustr. Audiences at The Fifth Elephant want to learn about problem statements that are general. Hence, your problem statement has to be something that has wider implications/applicability, beyond your company?
    2. What are the tried and tested ways to solve this problem? Why is your approach, with ADAM, novel? Explain this in the slides.
    3. Why build ADAM? Why not use tools that are already available out there to solve your problems?
    4. Did you compare your approach/solutions with other approaches/solutions? If yes, share details of comparison and why ADAM was a better choice?
    5. The slides don’t have conclusion and takeaways for the audience. What happens after explaining the architecture?
    6. Show before-after comparison in terms of changes and improvements (if any) – your problem before ADAM, and how things have turned out different after ADAM.
    7. How did your team adjust to this new solution? How has this been integrated in production? What challenges did you face?
    8. What can the audience learn from this experience?

    This proposal will be considered for Anthill Inside because the topics covered in this proposal fall in the realm of Anthill Inside.

    Next steps: submit revised slides by or before 21 May to make a final decision.

    • Shashank Jaiswal (@shashank94) Proposer a month ago

      I have made a small doc describing all the points that I want to convey during this session.
      https://docs.google.com/document/d/1fUsV_Puq8cL1thSCJ4qg2HWelaB0vosvUKme-Yu4fjA/edit?usp=sharing

      Please go through this once, I will update the slides shortly before deadline

      • Zainab Bawa (@zainabbawa) Reviewer 29 days ago (edited 29 days ago)

        This is useful, and needs to be put as a response here, instead of a Google doc.

        • Shashank Jaiswal (@shashank94) Proposer 29 days ago (edited 29 days ago)

          The problem statement outlined is very specific to Clustr. Audiences at The Fifth Elephant want to learn about problem statements that are general. Hence, your problem statement has to be something that has wider implications/applicability, beyond your company?
          1)Clustr was introduced to understand the problem we faced.
          2)The problem that we solved is a compex NER or Sequence Learning problem. These are very generic concepts in the field of NLP.
          3)Not only that we have shown some good work in the field on Active Learning. This is a concept which can be widely apply in any ML approach where the cost of Data Labelling Matters. And the approach we showed is also not specific to our problem. It can be applied anywhere.
          4)The E-commerce product-title type data that we are talking about is pretty generic and can be found in use already by many large and small organisations. And NER on them is pretty generic for ADAM to qualify here.
          5)We can generalise this to “an approach to entity extraction from product title like text in the absence of labeled data”. Keep the discussion as generic as possible and use Clustr reference only as an example to ground the discussion.
          6)I will update these information in the slides. Please, do tell me if u need more details regarding this point.

          What are the tried and tested ways to solve this problem? Why is your approach, with ADAM, novel? Explain this in the slides.
          1)There are 2 Crucial parts of ADAM.
          <>Deep Sequence Tagging Model. (There are few tried and tested way but cant work for us explained in the answer for 3rd point.)
          <>Active Learning. (Still Experimental, we would be one of a few organisation which are using this and real world demonstration would be worthy of learning)
          2)I will cover more on the existing literature on this topic and how ours is different (Will add in the slides)
          3)The approach is novel because it demonstrate a way to start with no label-data and move cautiously with Active learning. Thus using the resources optimally and provide a solution that can scale.

          Why build ADAM? Why not use tools that are already available out there to solve your problems?
          There are 3 different types of tools/Algorithm available
          1)One with no deep learning,hence hand crafted features (Didn’t work coz a lot of variance), like CRF-scuite, HMM, NLTK etc.
          2)The one with deep learning are really trained over different set of data Natural language, hence cannot be used, (The type of data that we have is different from what most models are tried and tested on. But the data that we have is very relevant to E-commerce industry. And Retail.)
          3)The state of the art architecture that already exist needs a lot of training data which can be costly. Hence we needed the concept of Active Learning to move forward.

          Did you compare your approach/solutions with other approaches/solutions? If yes, share details of comparison and why ADAM was a better choice?
          1)Yes we compared with CRFsuite and which was terrible cause it was unable to use any higher order correlations.
          2)The state of the art architecture that already exist needs a lot of training data which can be costly. Hence ADAM is a better choice.
          3)We need to understand that ADAM solves a problem which is pretty generic. But the approach with which ADAM was built is noteworthy.

          The slides don’t have conclusion and takeaways for the audience. What happens after explaining the architecture?
          1)That’s my bad, I will surely add them in the slides. (see point 8 for outline also)

          Show before-after comparison in terms of changes and improvements (if any) – your problem before ADAM, and how things have turned out different after ADAM.
          1)WIll update accordingly.

          How did your team adjust to this new solution? How has this been integrated in production? What challenges did you face?
          1)Since the innovation was in algorithm and training phase, the integration is never a problem. As said earlier,it was very generic problem that we were solving, But the journey is very different approach that we picked.
          2)This is a model which is consumed by a larger solution and briefly touch upon the catalogue
          What can the audience learn from this experience?
          1)How to tackle the noisy data problem in case of textual data.
          2)Why a deep NN model plays an important role in generalisation.
          3)Why Active Learning is a really important concept for dealing with the problem of no label data.

          I will surely try to update slides which will better describe these points.
          Please Inform me if these answer your doubts. I am happy to talk also if u need any clarification.

    • Shashank Jaiswal (@shashank94) Proposer a month ago

      Sure Let me update accordingly..

  • Zainab Bawa (@zainabbawa) Reviewer 29 days ago (edited 29 days ago)

    Will wait for revised slides. No questions yet.

    • Shashank Jaiswal (@shashank94) Proposer 28 days ago (edited 28 days ago)

      This is the final updated slides. Please go through it and Kindly inform me if the questions were answered. Thank you.
      Slides (1-4,8-12,33-35) is answering the questions that was pointed out.
      ALso the slides about architecture (15-26) has been updated kindly go through it

      https://www.slideshare.net/secret/3YWFP67xyCgspO

      • Zainab Bawa (@zainabbawa) Reviewer 28 days ago

        Sounds good! We’ll do a review and move to the next steps. Expect to hear from us by Friday, if not earlier.

        • Zainab Bawa (@zainabbawa) Reviewer 28 days ago

          Also, we have moved your proposal here.

  • simrat (@sims) 27 days ago

    Hi Shashank,

    Slide #11: Would it possible to show some results, both positive and negative before you go on to explain why the approach failed for you. How did you measure the performance etc. This will set the motivation for moving to the improved algorithm.

    Slide #16: Like wise here it would be great if you have a slide to show sample knowledge graph, extracted data from amazon , data set structure and how you combined them. Is it rule based? or something else? This will help the listners get an idea.

    Slide #24: explain with example what is active learning, how does it course correct any issue introduced during automated training data phase and how is the feedback passed back to the system.

    Slide #25: Kindly add one end to end example of synthetically generated data, tagged and refined.
    Also, detail how did you measure performance

    Right now we have no visibility to the data or results so it is like we have to believe whatever you are saying :)
    Adding these will make the information flow smooth for us.

    The objective of last few slides under heading “how does it fit in” is not coming out clearly

    • Shashank Jaiswal (@shashank94) Proposer 27 days ago (edited 26 days ago)

      @sims
      Hey Simrat,
      I understand what you are trying to say,
      Slide #11: Would it possible to show some results, both positive and negative before you go on to explain why the approach failed for you. How did you measure the performance etc. This will set the motivation for moving to the improved algorithm.
      most of the approaches that i describe in slide #10 were rejected directly on paper, but we did try 1 or 2 approaches I can add their results and inform why they were not good. This might take sometime as we would need to pull out some older work and make them run again. but surely can be done.

      Slide #16: Like wise here it would be great if you have a slide to show sample knowledge graph, extracted data from amazon , data set structure and how you combined them. Is it rule based? or something else? This will help the listners get an idea.
      Knowledge graph and Amazon data, sure I will add some examples no problem there,
      data-set example is present in Slide #17. Is there a need to add more clarification?

      Slide #24: explain with example what is active learning, how does it course correct any issue introduced during automated training data phase and how is the feedback passed back to the system.
      I will add a flowchart describing the feedback loop. Slide #26 does describe the pipeline though, but if specifically Active Learning part is not very clear, I will add them in the slides seperately

      Slide #25: Kindly add one end to end example of synthetically generated data, tagged and refined.
      Also, detail how did you measure performance

      I will incorporte this with the above flowchart aswell, basically I will use an example to descibe the flowchart.

      The objective of last few slides under heading “how does it fit in” is not coming out clearly
      Let me look into it. BTW these slides describe how do we incorporate ADAM for our use cases. I plan to talk about it more rather then just add them in the slides. Will that be alright or should i change the slides accordingly.

      Also, detail how did you measure performance
      I can add the details but exact numbers might take some time. It would definitely be before presentation dates. DO I need to add the numbers right now??

      Please inform me by when should i incorporate these changes.

      • Shashank Jaiswal (@shashank94) Proposer 19 days ago

        Hey I have made the changes as described above, see below comment to check the updated slide. Also, please inform me what are the next steps. Thank you

  • Shashank Jaiswal (@shashank94) Proposer 21 days ago

    I have made following changes in the slide uploaded here:
    https://www.slideshare.net/secret/Jx8awxewNlaK5n
    Slides 12-13 resolves issue with slide #11
    Slides 19-20 resloves issue with slide #16 (now slide #18)
    Slides 29-31 resloces issue with slide #24 #25 (now #27 #28)
    Slides 37-38 resolves “detail how did you measure performance” issue (The numbers are in the process of generation and will update here shortly)
    Slides 33-35 have been restructured (their heading to have a clear understanding)

    Please go through them once
    @zainabbawa and @sims Kindly inform if the issues still persist I will try to make them clear. Also I will format the slides a little to maintain a uniformity once the content is fixed.

  • Abhishek Balaji (@booleanbalaji) Reviewer a day ago

    Feedback from rehearsal:
    Time taken: 38 mins
    - Introduction has to be improved.
    - Use slides/figures to communicate quickly
    - It’s not mentioned what NER/POS is till the middle. Needs better explanation
    - The presentation structure is a bit all over the place. Try to practice a couple more times to nail it
    - Add references to the papers you’re referring to
    - Add information on what your contribution was
    - Create your own word embeddings
    - Use your knowledge graph as an input
    - Establish what Adam is and show future improvements
    - Active learning:
    - No metrics. How good is this?
    - Give positive and negative examples
    - Why 10k/20k wasnt useful? Why do you mention that you dont have training data/numbers
    - Not clear what each one does
    - Explain more on BILSTM
    - Why CRF?
    - How is a knowledge graph given to BiLSTM?

    • Misc
      • Bidirection: coorelation goes both ways; emperically shows better performance
      • 7k training sample data
      • Word vecotr data site, size of data used for Bi-LSTM training
      • Maake a strong pitch in Active learning

    Shashank, do work on the above feedback and get back to us with the revised slides by Jun 27, 2019. We’ll let you know the final decision on your talk after evaluating your revised content.

Login with Twitter or Google to leave a comment