Anthill Inside 2019

On infrastructure for AI and ML: from managing training data to data storage, cloud strategy and costs of developing ML models

Learning to Rank recommendation - Ranknet to LambdaMART to Groupwise scoring functions - experiments, introduction to Tensorflow Ranking

Submitted by narasimha m (@6544) on Monday, 29 April 2019


Preview video

Section: Full talk Technical level: Intermediate Session type: Lecture

Abstract

Search and product recommendations are typically served using CF, MF, FM techniques, content/context/sequence based methods or using learning to rank framework which is more generic than rest. Evolving from traditional classification and regression modeling methods, the loss functions, gradients, computation tricks have evolved to suit ranking problems (point wise to pair wise to list wise solutions) in this ambit. Learning to rank models such as LambdaRank and LambdaMart models (developed by Microsoft research group) are very competitive and have shown success in many instances (Kaggle, Yandex, Yahoo rankings challenges). More recently, multi-item group wise scoring functions were proposed by google research group (2018). This talk would focus on our team’s journey (what worked and what didn’t) in using these methods for ranking, importance of sampling, importance of measuring performance using multiple metrics (not just NDCG), importance of offline and online evaluations for ranking, and compare Ranknet, LambdaRank/LambdaMART, Listnet and multi-item group-wise scoring functions.

Outline

1. Introduction to learning to rank framework.
2. Metrics for evaluation. 
3. Description of the methods (Ranknet, LambdaRank, lamdaMART, multi-item group-wise scoring functions)
4. Integrating diverse feature types, feature engineering, transformations and sampling of items.
5. Results.
6. Summary
7. Future work

Speaker bio

Speaker: Arpit Katiyar: Arpit Katiyar is currently working as Lead Data Science Engineer at MakeMyTrip(MMT). He is part of user personalization team with focus on delivering relevant content to each MMT customer. He has a total work experience of 5 years in building machine learning based solution for biometric and large scale telecom data with companies Samsung and Mobileum, respectively. Arpit holds B. Tech and M. Tech degree in Computer Science from IIT Delhi with thesis work published in ACM - ICS(International Conference on Supercomputing).

Contributors:
Pulkit Bansal: Pulkit is currently Lead Data Scientist at MakeMyTrip. His focus at MakeMyTrip has been on problems relating to developing personalized hotel recommendations using ML models. He has also worked on feature store engineering and customer-loan propensity projects. He has total 5.5+ years of experience, and has previously worked on problems in quantitative finance and algorithmic trading at Goldman Sachs and WorldQuant. He has also worked on optimization problems in online advertising at Adobe. Pulkit has graduated with a BS-MS Dual Degree in Mathematics and Scientific Computing at IIT Kanpur, where he graduated at the top of his class.

Narasimha Medeme: Director, data science at MakeMyTrip. He oversees Hotel ranking, user personalization, multi LOB initiatives such as cross-sell, personalized notifications, and hotel dynamic pricing. He has ~15 years experience working with many fortune 500 company clients (previously at Kantar data science), multiple domains and business problems, solving diverse data science and optimization problems. While he is applying machine-learning/deep-learning models for personalization, he is also experimenting with deep reinforcement learning models/practices in off-line learning environments. Narasimha has MSc from University of Missouri, BTech from IIT Madras.

Links

Slides

https://docs.google.com/presentation/d/1KxUPdaybA0kSqBcKy9vpGMT_O8_qp8ML-q9G-Qh7lU8/edit#slide=id.p1

Preview video

https://drive.google.com/file/d/1rLRdiCjd0aX_xmEYVStSsjAs-rLi-muF/view?usp=sharing

Comments

  • Abhishek Balaji (@booleanbalaji) Reviewer a month ago

    Hi Arpit/Pulkit/Narasimha,

    Thank you for submitting a proposal. As per the policy, we only allow one presenter on stage per talk. Please make a decision on who would be presenting and update the same here. In addition, for us to evaluate your proposal, we need to see detailed slides and a preview video. Your slides must take the following points into consideration:

    • Problem statement/context, which the audience can relate to and understand. The problem statement has to be a problem (based on this context) that can be generalized for all.
    • What were the tools/options available in the market to solve this problem? How did you evaluate these, and what metrics did you use for the evaluation? Why did you decide to build your own ML model?
    • Why did you pick the option that you did?
    • Explain how the situation was before the solution you picked/built and how was the fraud/ghosting after implementing the solution you picked and built? Show before-after scenario comparisons & metrics.
    • What compromises/trade-offs did you have to make in this process?
    • What are the privacy, regulatory and ethical considerations when building this solution?
    • What is the one takeaway that you want participants to go back with at the end of this talk? What is it that participants should learn/be cautious about when solving similar problems?

    As next steps, we’d need to see the detailed and/or updated slides by 21 May, in order to close the decision on your proposal. If we dont receive an update by 21 May, we’d have to move the proposal for consideration for a future conference.

    • narasimha m (@6544) Proposer 29 days ago

      Sure Abhishek Balaji. We will upload the preview video of the speaker. And will upload details slides as well. I presume a couple of pages with final statistics can be updated post 21stMay session as well.

  • narasimha m (@6544) Proposer 27 days ago

    I’ve updated the slides. And, I updated speaker to one - my team member. We will upload the video too.

  • narasimha m (@6544) Proposer 11 days ago

    hi Abhishek Balaji, The preview video too has been updated on ~21/22nd May.

Login with Twitter or Google to leave a comment