BEGIN:VCALENDAR VERSION:2.0 PRODID:-//HasGeek//NONSGML Funnel//EN DESCRIPTION:On DataOps\, productionizing ML models\, and running experimen ts at scale. X-WR-CALDESC:On DataOps\, productionizing ML models\, and running experime nts at scale. NAME:MLOps Conference X-WR-CALNAME:MLOps Conference REFRESH-INTERVAL;VALUE=DURATION:PT12H SUMMARY:MLOps Conference TIMEZONE-ID:Asia/Kolkata X-PUBLISHED-TTL:PT12H X-WR-TIMEZONE:Asia/Kolkata BEGIN:VEVENT SUMMARY:Introduction to MLOps conference DTSTART:20210723T063000Z DTEND:20210723T064500Z DTSTAMP:20260421T161610Z UID:session/PDMHPCBcZvhF4cFtaTri1m@hasgeek.com SEQUENCE:0 CREATED:20210706T042410Z LAST-MODIFIED:20210715T123052Z LOCATION:Online ORGANIZER;CN="The Fifth Elephant":MAILTO:no-reply@hasgeek.com BEGIN:VALARM ACTION:display DESCRIPTION:Introduction to MLOps conference in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Theme - Machine Learning Operations DTSTART:20210723T064500Z DTEND:20210723T065500Z DTSTAMP:20260421T161610Z UID:session/NnLBwL2JbkeoTScZoHK5CG@hasgeek.com SEQUENCE:0 CREATED:20210314T140127Z LAST-MODIFIED:20210722T013256Z LOCATION:Online ORGANIZER;CN="The Fifth Elephant":MAILTO:no-reply@hasgeek.com BEGIN:VALARM ACTION:display DESCRIPTION:Theme - Machine Learning Operations in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Leveraging ML algorithms to deliver faster in a chaotic environmen t - Flipkart's case study DTSTART:20210723T065500Z DTEND:20210723T073000Z DTSTAMP:20260421T161610Z UID:session/FGKndcJWTZjduz2afSVatL@hasgeek.com SEQUENCE:1 CREATED:20210715T122118Z DESCRIPTION:Ecommerce in our country has come quite some distance and alon g with it has been successful in reshaping customer expectations and his/h er perception of hygiene & delight. Speedy delivery now has become a base expectation of today's customer. Add a certain set of categories \, the ne ed for quicker SLA just escalates higher.\n\nIn this battle to improve on the faster delivery aspirations \, eKart has leveraged a ML algorithm to c alculate SLA and simulate this for the future.\nThis talk is a sneak peak into the approach that we've taken on this problem statement. \n\nIn this talk \, you will learn about - \n\n- Importance of Customer Speed in the c ontext of ecommerce in India in 2021\n- The complexity of network run by a large scale ecommerce supply chain like eKart\n- How we are leveraging ML to solve for this problem\n- High level architecture overview of how the systems are designed \n- Choices we’ve taken on anomalies encountered in the ML output \, given the criticality of the output metric\n- “Sense & Response” how we’re powering an ability for the system & stakeholders to react to these changes\n\nAgenda - \n\n- Importance of SLA \, shift in buyer trend in India\n- An overview of the components of the ML Algorithm used for computing the SLA + the simulation algorithm \n- Architecture vi ew of the solution \n- Deep dive on the anomalous situations encountered & reactions taken by the system - \n - Look forward simulation \n - Infea sible inputs\n - Simulation result exceeding guardrails\n - Alerts on mo derate anomalies \n- Overall impact achieved \n\nWho should attend this - \n\n- Product Managers especially the ones in Supply Chain\n- Data Scienti sts + OR enthusiasts\n- People interested in applications of GraphDB\n\nbh aradwaaj.rajan@flipkart.com\; sandilya.konduri@flipkart.com LAST-MODIFIED:20230108T103046Z LOCATION:Online ORGANIZER;CN="The Fifth Elephant":MAILTO:no-reply@hasgeek.com URL:https://hasgeek.com/fifthelephant/mlops-conference-july-2021/schedule/ leveraging-ml-algorithms-to-deliver-faster-in-an-extremely-chaotic-environ ment-FGKndcJWTZjduz2afSVatL BEGIN:VALARM ACTION:display DESCRIPTION:Leveraging ML algorithms to deliver faster in a chaotic enviro nment - Flipkart's case study in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:End-to-end serverless transformers on AWS Lambda for NLP DTSTART:20210723T073000Z DTEND:20210723T075500Z DTSTAMP:20260421T161610Z UID:session/UofcGaEKsC4A9f9QrhTdCC@hasgeek.com SEQUENCE:2 CATEGORIES:Speaker to submit deck with full outline of the talk,30 min tal k,ML deployment workflows CREATED:20210715T121510Z DESCRIPTION:Transformers are everywhere! But how to serve them? How do you leverage serverless to get scalability without any worries? Isn't serverl ess used for light applications?. How to get the best latencies with your serverless? I will be sharing answers to these questions in my talk.\n\nSl ides - https://bit.ly/serverless-transformers\n\n### About Pratik\nA self- taught data scientist and open-source developer from India. He specialises in making Search & NLP solutions.\nHe runs a slack data science community http://maxpool.club and writes at https://pakodas.substack.com. \nYou can find his previous talks with PyData\, WiMLDS & DAIR at http://talks.prati k.ai\nPortfolio - http://pratik.ai\n\n## Agenda\n1. Paradigms of deploymen t\n - Live server\n - Batch processing\n - Serverless\n2. Benefit s of serverless\n3. Deploying transformer models on Lambda\n4. Exposing AP I\n5. Versioning lambdas\n6. CI/CD with GitHub actions\n7. Runtime limitat ions and consequences\n8. Multi-tenant design for lambdas\n9. Conclusion\n \n## Key takeaways\nLearn to deploy transformers in production\nServerless can be really good for many scenarios\nGet huge instant scalability with serverless\nTons of savings in cost and headache\n\n### Audience\nAny leve l of audience and whole ML community LAST-MODIFIED:20230810T072606Z LOCATION:Online ORGANIZER;CN="The Fifth Elephant":MAILTO:no-reply@hasgeek.com URL:https://hasgeek.com/fifthelephant/mlops-conference-july-2021/schedule/ end2end-serverless-transformers-on-aws-lambda-for-nlp-UofcGaEKsC4A9f9QrhTd CC BEGIN:VALARM ACTION:display DESCRIPTION:End-to-end serverless transformers on AWS Lambda for NLP in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Jupyter to Jupiter : scaling multi-tenant ML pipelines DTSTART:20210723T075500Z DTEND:20210723T083500Z DTSTAMP:20260421T161610Z UID:session/AxCsq1vewpg1U8A4nzGyVT@hasgeek.com SEQUENCE:2 CATEGORIES:TBD - to be decided,TBD - to be decided after the walkthrough., Organize walkthrough between speaker and editor CREATED:20210706T042645Z DESCRIPTION:A brief talk summarising the journey of an ML feature from a J upyter Notebook to production. At Freshworks\, given the diverse pool of c ustomers using our products\, each feature has dedicated models for each a ccount\, churning out millions of predictions every hour. This talk shall encompass the different tools and measures we've used to scale our ML prod ucts. Additionally\, I'll also be touching upon Apache Airflow\, a workflo w management platform and how we've used it to automate and parallelise va rious segments of our ML pipeline. \n\n### Outline\n1. **Introduction**  [3 minutes]\n a. About myself \n b. Challenges of a multi-tenant ML pipel ine\n c. Role of a Data Scientist vs an ML Engineer\n2. **Incentives to s cale your pipeline** [5 minutes]\n a. Reducing turnaround time (real-time vs batch-wise)\n b. Increasing availability & adhering to SLA \n c. Enabl ing diverse customers to use your ML features\n d. Automating workflows\n3 . **Brief Intro to Airflow** [5 minutes]\n a. Why not cron?\n b. DAGs\, T asks and Operators\n c. Executors: LocalExecutor\, CeleryExecutor\, Kubern etesExecutor\n d. Controls: Task Pools\, Queues and Scheduling rules\, Par allelism\, etc. \n e. Reasons to avoid Airflow\n4. **Scaling different par ts of an ML pipeline** [15 minutes]\n 1. **Data Ingestion and preprocessin g**\n a. Data pipelines at Freshworks\n b. Aggregating different types o f data from different sources (be it streams\, databases or S3)\n c. One datastore may not work all types of data \n d. Fetching at different int ervals\, without adding too much load \n e. Cleaning data before inserti on to optimise storage\n f. Optimising preprocessing layers to adapt to t he rate of incoming data\n 2. **Model Training\, Evaluation and Deployment **\n a. Offline ML platform & workflows at Freshworks\n b. Periodically training model to adapt to recent data\n c. Including customer-specific r ules and features\n d. Hyper-parameter tuning\n e. Leveraging spark clus ters to train faster\n f. Evaluating models and monitor metrics over time \n g. Maintaining model versioning to revert to older versions as a fallb ack\n 3. **Prediction\, Back-filling and Interpretability**\n a. Online M L platform & workflows at Freshworks\n b. Should be capable of scaling to handle more customers\n c. Avoid single point of failure with distribute d execution\n d. Establishing back-filling pipelines if historic predicti ons are of importance\n e. Capturing and handling errors without disrupti ng the entire workflow\n f. Setting up alerts to identify engineering and data science anomalies \n g. Provide interpretable insights to justify predictions to stakeholders\n 4. **Misc. engineering practices**\n a. Pla nning before execution : Be it a new module or picking a tool.\n b. Funct ional testing : Ensuring offline and online pipelines are on par\n c. App lication Security : Build data pipelines keeping regulations in mind\n d. Documentation : Add docstrings\, setup & deployment instructions and an e laborate README\n\n### Key Takaways : \n1. Building a multi-tenant ML pipe line to serve a diverse user base\n2. Tips\, hacks and practises for scali ng different parts of an ML pipeline\n3. Leveraging Airflow to accomplish the above\n4. Potential bottlenecks and issues one might face while solvin g the above LAST-MODIFIED:20230810T072606Z LOCATION:Online ORGANIZER;CN="The Fifth Elephant":MAILTO:no-reply@hasgeek.com URL:https://hasgeek.com/fifthelephant/mlops-conference-july-2021/schedule/ jupyter-to-jupiter-scaling-multi-tenant-ml-pipelines-AxCsq1vewpg1U8A4nzGyV T BEGIN:VALARM ACTION:display DESCRIPTION:Jupyter to Jupiter : scaling multi-tenant ML pipelines in 5 mi nutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Speaker connect session: with Pratik Bhavsar and Vishal Gupta DTSTART:20210723T083500Z DTEND:20210723T090500Z DTSTAMP:20260421T161610Z UID:session/7Z7uKHNJHTz7QmQ5QYmmUP@hasgeek.com SEQUENCE:0 CREATED:20210720T092535Z LAST-MODIFIED:20210722T050037Z LOCATION:Online ORGANIZER;CN="The Fifth Elephant":MAILTO:no-reply@hasgeek.com BEGIN:VALARM ACTION:display DESCRIPTION:Speaker connect session: with Pratik Bhavsar and Vishal Gupta in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Break DTSTART:20210723T090500Z DTEND:20210723T092000Z DTSTAMP:20260421T161610Z UID:session/Lerp6qN6gpCvvcK8TcxMPD@hasgeek.com SEQUENCE:0 CREATED:20210706T042830Z LAST-MODIFIED:20210720T104312Z LOCATION:Online ORGANIZER;CN="The Fifth Elephant":MAILTO:no-reply@hasgeek.com BEGIN:VALARM ACTION:display DESCRIPTION:Break in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:MLOps for startups DTSTART:20210723T092000Z DTEND:20210723T094500Z DTSTAMP:20260421T161610Z UID:session/4NAw6NkCXoJMqVuhZ5EM8u@hasgeek.com SEQUENCE:2 CATEGORIES:Slides for pre-recorded talk reviewed and approved,30 min talk, ML deployment workflows CREATED:20210715T122010Z DESCRIPTION:\n# Context: Early Startup\n\nWhile a plethora of MLOps work h as been done at large data and model serving scale\, this is an attempt to introduce you to different ops problems\, tools and how to pick among the m. In most cases\, I’ll share an example to anchor the idea set further. \n\nBased on my experiences at Verloop.io\, growing from 0 to 1. Verloop i s among India’s largest conversational automation platforms. We work wit h some of India’s most largest category defining companies to serve thei r users.\nTalk Objective\n\nThis is a primer on how to think about your ex perimentation/dev and deployment process. The bulk of this talk is organis ed in 2 columns: problem and a recommended process or tool around that. I borrow from my experience working on ML engineering challenges e.g. keepin g latencies low enough for chat to be usable\, things we learnt along the way e.g. need for data versioning and so on.\n\n*Slides for the Talk* : ht tps://bit.ly/startupmlops2021\n\n# Talk Outline\n\n - MLOps Primer 101: Intro to layout of this talk [2-3 minutes]\n - Assumptions/Prerequisite s: Team size\, Skills\, Organisation\, Shipping [4-5 minutes\n - Level 1: DevOps: Builds\, Tests and CI/CD [5-7 minutes]\n - Intent: You don’t repeat a mistake which you’ve made previously\n - Code Review: Github with the ReviewNB App\n - Setup testing for business logic and app at the very least. Add them to CI/CD\n - [Recommend] Test your ML models with a utomated tests like CheckList\n\n- Level 2 MLOps: Experiments and Dev Cycl e [8-10 minutes]\n - Intent: Automated Training\n - Our 3 key elements: da ta\, code/architecture\, model weights\n - Data needs to be mise en place - ideally pipelines and ETL processes are figured out\n - Reproducible pip elines -- focus on integrating with your existing infra\, and it’s okay to simply pull using a Jupyter notebook\n It’s also ok to aim for more c omplexity\, by setting up DAG tools like Airflow and ML FLow\n - Experimen t Tracking: Sacred\, Neptune.ai\, W&B\n - Both training code and resulting models are version controlled\n - DVC.org for Data Version Control\n - Ma nual release- Don’t bother automating it so early\n - Manage your own re leases\, add SDE to your team if needed\n - [Optional] Managed compute\n - Excellent time to adopt Docker and if your company is on K8\, port your s ervice to K8 as well\n- Level 3 MLOps: Model Deployment [Total: 5-7 minute s]\n - Intent: Automated Model Deployment\n - Data pipeline gathers data w ithout engineering time - including annotations at predictable cadence\n - Release engineering: Adopt what your engineering team is doing: Canary or Blue-Green deployment\n - Setup infra to A/B test your models or blends o f them in shadow mode\n - Concept\, Vocabulary\, Label Drift: Don’t both er. Your data pipeline will update your models for you\n\n - Questions fro m Audience [3-5 minutes]\n\nThis is a process/maturity example and recomme ndation of when you should solve for a specific problem within MLOps. It ’s not a hard and fast guide and you can always solve some problems soon er or later.\n\nI use the Microsoft MLOps Model as a reference.\n\nSome ex perience with the following would be useful for this talk:\n\n- Working in a team for 2-10 data folks\, across modeling\, engineering\, monitoring\, deployment and so on\n- Deploying models with specific requirements\, e.g . low latency\, high throughput\, large data volumes at inference (more th an 1TB)\n\n## Speaker Info:\n\nNirant has worked across startups and MNCs in Machine Learning and Data Science roles. These include:\n\n- Verloop - Natural Language Process - Conversational Automation for Enterprises\n- So roco - Computer Vision: Image Segmentation - building Search for Enterpris e Documents\n- Samsung Research at the Advanced Technologies Lab - Senor F usion & Event Classification\n- Belong.co (NLP/Predictive Analytics)\n\nAt his present role in Verloop.io\, he focused on Conversational AI\n\nHe ha s written a book on Practical NLP for Developers (Published by Packt). Thi s book is a Quickstart Guide for Developers interested in building NLP bas ed solutions\, without the patience for pedantic learning on Linguistics a nd Deep Learning.\n\n## Recognition & Contributions\n\n- Won the Kaggle NL P Kernel Prize from Kaggle and Explosion.AI (makers of spacy.io)\n Lead Maintainer for awesome-nlp with ~11K+ stars - recommended by Andrew Ng’ s Deep Learning course CS229 at Stanford\n- GitHub's official Machine Lear ning collection includes awesome-nlp as world's best NLP resource\n- FastA I International Fellowship: 2018 & 2019\n\n## Talks\n\n- PyCon India 2019 : http://bit.ly/pycon2019talk (Google Slides Talk)\n- inMobi Tech Talks: A Nightmare on the LM Street\; Slides\n - Wingify DevFest: NLP for Indian Languages\; Slides\, Video\n - PyData Bengaluru Inaugural Talk: Video\, Re sources\n\n## Speaker Links\n\n Personal Website: https://nirantk.com\n Twitter: https://twitter.com/NirantK/\n Github: https://github.com/Nir antK\n LinkedIn: https://linkedin.com/in/nirant\n Book: https://www.am azon.in/dp/B07L3PLQS1/\n LAST-MODIFIED:20230810T072606Z LOCATION:Online ORGANIZER;CN="The Fifth Elephant":MAILTO:no-reply@hasgeek.com URL:https://hasgeek.com/fifthelephant/mlops-conference-july-2021/schedule/ ml-ops-for-startups-4NAw6NkCXoJMqVuhZ5EM8u BEGIN:VALARM ACTION:display DESCRIPTION:MLOps for startups in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Birds of Feather (BOF) session - Running MLOps in large and small organizations DTSTART:20210723T094500Z DTEND:20210723T101500Z DTSTAMP:20260421T161610Z UID:session/3o223zhSaqSLT2ytm6xHN3@hasgeek.com SEQUENCE:0 CREATED:20210720T092716Z LAST-MODIFIED:20210722T055448Z LOCATION:Online ORGANIZER;CN="The Fifth Elephant":MAILTO:no-reply@hasgeek.com BEGIN:VALARM ACTION:display DESCRIPTION:Birds of Feather (BOF) session - Running MLOps in large and sm all organizations in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Flash talks DTSTART:20210723T101500Z DTEND:20210723T103000Z DTSTAMP:20260421T161610Z UID:session/MTjidwzrNApSu3Ya1zSWti@hasgeek.com SEQUENCE:0 CREATED:20210720T092813Z LAST-MODIFIED:20210720T092815Z LOCATION:Online ORGANIZER;CN="The Fifth Elephant":MAILTO:no-reply@hasgeek.com BEGIN:VALARM ACTION:display DESCRIPTION:Flash talks in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Tuning hyperparameters with DVC experiments DTSTART:20210723T103000Z DTEND:20210723T112000Z DTSTAMP:20260421T161610Z UID:session/4XWMrbp1LsRi3iLghyUZmu@hasgeek.com SEQUENCE:1 CATEGORIES:30 min talk,TBD - to be decided after the walkthrough.,Organize walkthrough between speaker and editor CREATED:20210706T042507Z DESCRIPTION:When you start exploring multiple model architectures with dif ferent hyperparameter values\, you need a way to quickly iterate. There ar e a lot of ways to handle this\, but all of them require time and you migh t not be able to go back to a particular point to resume or restart traini ng.\n\nIn this talk\, you will learn how you can use the open-source tool\ , DVC\, to compare training metrics using two methods for tuning hyperpara meters: grid search and random search. You'll learn how you can save and t rack the changes in your data\, code\, and metrics without adding a lot of commits to your Git history. This approach will scale with your data and projects and make sure that your team can reproduce results easily. LAST-MODIFIED:20230108T103046Z LOCATION:Online ORGANIZER;CN="The Fifth Elephant":MAILTO:no-reply@hasgeek.com URL:https://hasgeek.com/fifthelephant/mlops-conference-july-2021/schedule/ tuning-hyperparameters-with-dvc-experiments-4XWMrbp1LsRi3iLghyUZmu BEGIN:VALARM ACTION:display DESCRIPTION:Tuning hyperparameters with DVC experiments in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Model Health Assurance at scale at LinkedIn DTSTART:20210723T112000Z DTEND:20210723T121000Z DTSTAMP:20260421T161610Z UID:session/CJYss5gmnzF75gFFBJGkXh@hasgeek.com SEQUENCE:1 CREATED:20210719T111819Z DESCRIPTION:At LinkedIn\, AI models drive numerous use-cases - across rank ing (eg: Feed items\, jobs relevance)\, and classification (wg: content qu ality\, account protection) use-cases - with hundreds of experiments perfo rmed every week. At this scale\, it is also important to standardize and p latformize how the said monitoring is also done so that we don't reinvest on the same things\, we can effect change across all use-cases easily etc - this lets our engineers move faster with the confidence that unexpected behavior can be acted upon.\nWe have built a platform named Pro-ML\, which provides standardized components across the AI lifecycle that engineers u se to build out their AI pipelines. Pro-ML comes with Health Assurance bui lt-in - these are a set of components that provide in-built monitoring and alerting on AI models at different stages of the model’s lifecycle - in cluding real-time latency and distribution graphs\, data-drift\, time-seri es anomaly detection on model scores and feature values\, dark-canary base d model validation and a lot more. In this talk we will be taking you thro ugh what we are doing as part of Health Assurance and how we are building it. A glimpse at https://engineering.linkedin.com/blog/2021/model-health-a ssurance-at-linkedin LAST-MODIFIED:20230108T103046Z LOCATION:Online ORGANIZER;CN="The Fifth Elephant":MAILTO:no-reply@hasgeek.com URL:https://hasgeek.com/fifthelephant/mlops-conference-july-2021/schedule/ model-health-assurance-at-scale-at-linkedin-CJYss5gmnzF75gFFBJGkXh BEGIN:VALARM ACTION:display DESCRIPTION:Model Health Assurance at scale at LinkedIn in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Maintaining Machine Learning model accuracy through monitoring DTSTART:20210723T121000Z DTEND:20210723T123500Z DTSTAMP:20260421T161610Z UID:session/fqzq8u4iuKTgfNwjFhMii@hasgeek.com SEQUENCE:1 CATEGORIES:Speaker to prepare slides for pre-recording,30 min talk,Model g overnance CREATED:20210706T042547Z DESCRIPTION:Machine Learning models begin to lose accuracy as soon as they are put into production. At DoorDash\, we implemented a robust monitoring system to diagnose this issue and maintain the accuracy of our forecasts. \n\nDoorDash is a last-mile logistics platform. We use Machine Learning to improve the quality of the experience of our customers. For example\, rel iable estimates for how long it takes for a restaurant to prepare a food o rder ensures that when a Dasher (our term for delivery drivers) arrives at a restaurant\, the food is already ready\, this leads to (a) less queuein g at the restaurant by Dashers\, leading to a positive experience for rest aurants and Dashers both\, (b) maximizes the earning potential for Dashers per hour\, (c) minimizes the time for delivery to the consumer.\n\nHoweve r\, once an ML model is trained\, validated\, and deployed to production\, it immediately begins degrading\, a process called _model drift_. This de gradation negatively impacts the accuracy of our time estimates and other ML model outputs. Because ML models are derived from data patterns\, their inputs and outputs need to be closely monitored in order to diagnose and prevent model drift. Systematically measuring performance against real-wor ld data lets us gauge the extent of model drift.\n\nIn the past\, we’ve seen instances where our models became out-of-date and began making incorr ect predictions. These problems impacted the business and customer experie nce negatively and forced the engineering team to spend a lot of effort in vestigating and fixing them. Finding this kind of model drift took a long time because we did not have a way to monitor for it.\n\nThis experience i nspired us to build a solution on top of our ML platform. We set out to so lve this model drift problem more generally and avoid issues like it in th e future for all of the ML use cases on our platform. Ultimately\, our goa l was to create a solution that would protect all the different ML models DoorDash had in production. \n\nIn this talk\, we will walk through our st ory of how we surveyed our data scientists\, applied systems thinking to t his problem\, came up with an approach\, and implemented a platform-level solution to preventing model drift.\n\nWe hope this story will be useful f or other Machine Learning Platform teams who want to prevent issues such a s data drift and model drift.\n\nFull Article : https://doordash.engineeri ng/2021/05/20/monitor-machine-learning-model-drift/ \n\nNOTE: We will also prepare slides for review later.\n LAST-MODIFIED:20230108T103046Z LOCATION:Online ORGANIZER;CN="The Fifth Elephant":MAILTO:no-reply@hasgeek.com URL:https://hasgeek.com/fifthelephant/mlops-conference-july-2021/schedule/ maintaining-machine-learning-model-accuracy-through-monitoring-fqzq8u4iuKT gfNwjFhMii BEGIN:VALARM ACTION:display DESCRIPTION:Maintaining Machine Learning model accuracy through monitoring in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Summary for the day's sessions\; key takeaways DTSTART:20210723T123500Z DTEND:20210723T124500Z DTSTAMP:20260421T161610Z UID:session/8fqP27yFQYz97Sd583gFr@hasgeek.com SEQUENCE:0 CREATED:20210706T043424Z LAST-MODIFIED:20210721T060055Z LOCATION:Online ORGANIZER;CN="The Fifth Elephant":MAILTO:no-reply@hasgeek.com BEGIN:VALARM ACTION:display DESCRIPTION:Summary for the day's sessions\; key takeaways in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Theme: Machine Learning Operations DTSTART:20210724T063000Z DTEND:20210724T064000Z DTSTAMP:20260421T161610Z UID:session/XrHzqh7xzJr3vwnPXmhy3b@hasgeek.com SEQUENCE:0 CREATED:20210720T102123Z LAST-MODIFIED:20210722T013250Z LOCATION:Online ORGANIZER;CN="The Fifth Elephant":MAILTO:no-reply@hasgeek.com BEGIN:VALARM ACTION:display DESCRIPTION:Theme: Machine Learning Operations in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Automatic rollbacks for MLOps deployments in Kubernetes DTSTART:20210724T064000Z DTEND:20210724T070000Z DTSTAMP:20260421T161610Z UID:session/KJ9jV3zkAK3Q4ydGJLzRpX@hasgeek.com SEQUENCE:1 CATEGORIES:Speaker to submit deck with full outline of the talk,30 min tal k,ML deployment workflows CREATED:20210715T121901Z DESCRIPTION:While there are different tooling to automate deployments of M L models most of them require manually written rules for verifying deploym ents in production.\n\nWould love to show community how to enable automati c rollbacks in their model deployment pipelines without needing any log re gexes\, monitoring rules to verify the health of the deployments. This is enabled via HybridK8s Droid\, to learn more - https://docs.hybridk8s.tech\ n\nI will also talk more about how to embrace SRE mindset while managing p roduction models and ensuring maximum uptime without needing too many SRE/ DevOps tooling. \n\nKey Takeaway :\n- How to reduce production downtime/M TTR using progressive deployments?\n- How to reduce lead time of producti onising ML applications?\n- How to reduce efforts required to implement/m aintain deployment pipelines? \n\nAudience : \nAnyone concerned about the stability of production models - Data/DataOps Engineers\, Engineering Mana gement LAST-MODIFIED:20230108T103046Z LOCATION:Online ORGANIZER;CN="The Fifth Elephant":MAILTO:no-reply@hasgeek.com URL:https://hasgeek.com/fifthelephant/mlops-conference-july-2021/schedule/ automatic-rollbacks-for-mlops-deployments-in-kubernetes-KJ9jV3zkAK3Q4ydGJL zRpX BEGIN:VALARM ACTION:display DESCRIPTION:Automatic rollbacks for MLOps deployments in Kubernetes in 5 m inutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:How to open source your ML service DTSTART:20210724T070000Z DTEND:20210724T073000Z DTSTAMP:20260421T161610Z UID:session/2ukZhi5NachXG6n7hpnwpm@hasgeek.com SEQUENCE:2 CATEGORIES:TBD - to be decided,TBD - to be decided after the walkthrough., Organize walkthrough between speaker and editor CREATED:20210707T071515Z DESCRIPTION:Pic2Card is an Opensource ML service that helps to create Adap tiveCards from an Image.  We have recently contributed this service to [ AdaptiveCards](https://adaptivecards.io/)\, an Opensource card authoring f ramework from Microsoft.\n\nWould love to share the experience of making o ur Pic2Card Object Detection model production ready\, and what are the tra deoffs and constraints that we covered in this process. This helps to see how much software engineering practices require to maintain a production l evel ML service\, and ensure the quality of the inferences in a cost-effec tive manner.\n\n## Agenda:\n- High-level ideas of the End-to-End release process\n- How we used GitHub Actions to simplify the CICD process for our ML service\n- Running inference in a cost-effective manner using Docker and Azure Functions\n- How we optimized trained model and Application to p ack them in a single Docker Image or multiple.\n- Pluggable ML Service des ign for faster and independent iterations.\n\nAttaching the community call video\, where we introduced Pic2Card to the AdaptiveCards' community.\n\n Pic2Card service is available under AdaptiveCards designer\, you can try t his out by clicking `New Card` button (https://adaptivecards.io/designer/) and then select `Create Card from Image`.\n\nCommunity Blog: https://adap tivecards.io/blog/2020/Community-Call-November/\nPic2Card source code: htt ps://github.com/microsoft/AdaptiveCards/tree/main/source/pic2card\n\n## Le vel\nBegineers to Indermediate\n\n## Takeaway\n- How we met the standard o pensource demands.\n- How we setup the end-to-end pipeline and make the pr ocess streamlined.\n- Is there any real difference between DevOps and MLOp s ?\, Will get some clarity on this.\n- Solid Software Engineering Practic es wins always.\n\nThank you\,\nHaridas N (https://haridas.in) LAST-MODIFIED:20230810T072606Z LOCATION:Online ORGANIZER;CN="The Fifth Elephant":MAILTO:no-reply@hasgeek.com URL:https://hasgeek.com/fifthelephant/mlops-conference-july-2021/schedule/ process-of-open-source-your-ml-service-2ukZhi5NachXG6n7hpnwpm BEGIN:VALARM ACTION:display DESCRIPTION:How to open source your ML service in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Break DTSTART:20210724T073000Z DTEND:20210724T074000Z DTSTAMP:20260421T161610Z UID:session/KEHEg8Zj2uJZHJgZpTrP1e@hasgeek.com SEQUENCE:0 CREATED:20210706T043016Z LAST-MODIFIED:20210720T102040Z LOCATION:Online ORGANIZER;CN="The Fifth Elephant":MAILTO:no-reply@hasgeek.com BEGIN:VALARM ACTION:display DESCRIPTION:Break in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Theme: Privacy\, security and ethics in Machine Learning DTSTART:20210724T074000Z DTEND:20210724T075000Z DTSTAMP:20260421T161610Z UID:session/H4SQEZHgAdQgtj2ZuCB5Va@hasgeek.com SEQUENCE:0 CREATED:20210706T042935Z LAST-MODIFIED:20210722T013307Z LOCATION:Online ORGANIZER;CN="The Fifth Elephant":MAILTO:no-reply@hasgeek.com BEGIN:VALARM ACTION:display DESCRIPTION:Theme: Privacy\, security and ethics in Machine Learning in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Privacy attacks in Machine Learning systems - discover\, detect an d defend DTSTART:20210724T075000Z DTEND:20210724T081000Z DTSTAMP:20260421T161610Z UID:session/KQtj9CkkkbWmE4YntEbLM9@hasgeek.com SEQUENCE:2 CATEGORIES:TBD - to be decided,Model governance,Organize walkthrough betwe en speaker and editor CREATED:20210706T043109Z DESCRIPTION:My name is Upendra Singh. I work at Twilio as an Architect. As a part of this talk proposal I would like to shed some light on the new k ind of attacks machine learning systems are facing nowadays - Privacy Atta cks. During the talk we will explain and demonstrate how to discover\, det ect and defend Privacy related vulnerabilities in our machine learning mod els. Will also explain why it is so critical to have solid Model Governanc e to manage the risks associated with these kinds of vulnerabilities. One of the main objectives of model governance is to manage risks associated w ith machine learning models. How safe and secure machine learning models a re? And we are not talking about model security from an exposed api point of view but from a privacy point of view. Let’s try to understand what w e exactly mean by that. \nFueled by large amounts of available data and ha rdware advances\, machine learning has experienced tremendous growth in ac ademic research and real world applications. At the same time\, the impact on the security\, privacy\, and fairness of machine learning is receiving increasing attention. In terms of privacy\, our personal data are being h arvested by almost every online service and are used to train models that power machine learning applications. However\, it is not well known if and how these models reveal information about the data used for their trainin g. If a model is trained using sensitive data such as location\, health re cords\, or identity information\, then an attack that allows an adversary to extract this information from the model is highly undesirable. At the s ame time\, if private data has been used without its owners’ consent\, t he same type of attack could be used to determine the unauthorized use of data and thus work in favor of the user’s privacy.\nApart from the incre asing interest on the attacks themselves\, there is a growing interest in\ nuncovering what causes privacy leaks and under which conditions a model i s susceptible to\ndifferent types of privacy-related attacks. There are mu ltiple reasons why models leak information. Some of them are structural an d have to do with the way models are constructed\, while others are due to factors such as poor generalization or memorization of sensitive data sam ples. Training for adversarial robustness can also be a factor that affect s the degree of information leakage.\n Data Protection regulations\, such as GDPR\, and AI governance frameworks require personal data to be protect ed when used in AI systems\, and that the users have control over their da ta and awareness about how it is being used. For projects involving machin e learning on personal data\, it is mandatory from Article 35 of GDPR to p erform a Data Protection Impact Assessment (DPIA). Thus\, proper mechanism s need to be in place to quantitatively evaluate and verify the privacy of individuals in every step of the data processing pipeline in AI systems.\ nIn this talk we will focus on:\n1. What are different types of attacks on machine learning systems?\n attacks against integrity\, e.g.\, evas ion and poisoning backdoor attacks that cause misclassification of specifi c samples\,\n attacks against a system’s availability\, such as po isoning attacks that try to maximize the misclassification error\n a ttacks against privacy and confidentiality\, i.e.\, attacks that try to in fer information about user data and models(in this talk we will focus\, de monstrate and discuss about these types of attacks)\n\n2. How to d o threat modeling in any ML project from a privacy point of view for Machi ne Learning Models? Here we will define and explain terminology which we w ill use for the rest of the discussion. Under threat modeling we have to a nalyze and conclude whether our Machine Learning Models are safe against a ttacks mentioned below. \n3. What are different types of attacks on machin e learning models impacting privacy? Breif explanation to each followed by deep dive into "Reconstruction Attack"\nThe attacks are categorized in fo llowing groups:\n Membership Inference attack: This type attack tries to determine whether an input sample was part of the training set \n R econstruction attack: This type of attack tries to recreate one or more tr aining samples and/or their respective training labels.\n Property Inf erence attack: This type of attack tries to extract dataset properties whi ch were not explicitly encoded as features or were not correlated to the learning task.\n Model extraction attack: This is a type of black box attack where the attacker tries to extract information and potentially ful ly reconstruct a model.\n \n Each of the above attack type is a serious to pic of discussion and disection. And is a candidate of dedicated talk in i tself. In my talk we will focus specifically on "Reconstruction Attack".\n 4. What are the causes(in the design of the architecture of machine learni ng models) which lead to Reconstruction Attack on machine learning models? \n5. How is Reconstruction Attack implemented? How is Reconstruction Attac k implemented under different kinds of learning(centralized\, distributed) settings? Hands on demo of the same and the techniques used. It is critic al to understand how these attacks are implemented in order to avoid them. Just as a network security expert has to think like a hacker and understa nd the craft of hacking to better design network security we need to have a similar mindset while designing machine learning models to avoid privacy vulnerabilities.\n6. How to detect whether your existing machine learning models are susceptible to Reconstruction Attack and quantifying the same for the DPIA(Data Protection and impact assessment)? Will provide a hands on demo for the same and the technique to quantify the same.\n7. How to de fend against Recosntruction Attacks by applying state of the art technique s? For example:\n Differential Privacy Techniques\n Regularization T echniques\n Prediction vector tampering \nI believe in explaining by do ing. Whole talk will be sprinkled with hands on demonstrations wherever po ssible to explain the concept better.\n\nSlideLink: https://docs.google.co m/presentation/d/e/2PACX-1vRZO1I0KZhpIeMp-T1iK1as4kcsV0qDN_FozD2RS_skN6Hoe 5FpsLv3vnXF0fhsQsxH2c9PYMzeF7Jh/pub?start=false&loop=false&delayms=3000\n LAST-MODIFIED:20230108T103046Z LOCATION:Online ORGANIZER;CN="The Fifth Elephant":MAILTO:no-reply@hasgeek.com URL:https://hasgeek.com/fifthelephant/mlops-conference-july-2021/schedule/ privacy-attacks-in-machine-learning-systems-discover-detect-and-defend-KQt j9CkkkbWmE4YntEbLM9 BEGIN:VALARM ACTION:display DESCRIPTION:Privacy attacks in Machine Learning systems - discover\, detec t and defend in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:How do we build unbiased ML workflows and achieve fairness in Mach ine Learning? DTSTART:20210724T081000Z DTEND:20210724T084500Z DTSTAMP:20260421T161610Z UID:session/W3YpqPGt9U4JXWVarUjwVL@hasgeek.com SEQUENCE:2 CATEGORIES:Speaker to submit deck with full outline of the talk,30 min tal k,TBD - to be decided after the walkthrough. CREATED:20210715T121752Z DESCRIPTION:Biases often arise in automated workflows based on MachineLear ning models due to erroneous assumptions made in the learning process. Exa mples of such biases involve societal biases such as gender bias\, racial bias\, age bias and so on. \n\nIn this 15 minute talk\, we hope to cover prominent sources of such biases that lead to ML models producing unwanted outcomes. We will also look at ways to detect and measure such biases in our production workflows. \n\nOutline: \n\nIntroduction: Why do we care a bout biases? (1 min)\n\nSources of biases (3 minutes)\n→ Specification B ias\n→ Sampling Bias\n→ Measurement Bias\n→ Annotator/ Label Biases\ n→ Inherited Biases from other ML models \n\nMetrics to measure biases(5 minutes)\n(Will cover classification use-case only - since it might be ha rd to do more here)\n→ TPR across Groups\n→ FPR across Groups\n→ Acc uracy across groups\n→ Demographic Parity\n\nHow to avoid biased ML work flows(3 mins)\n\n→ Techniques for Debiasing Data going into model\n→ T echniques involving Post Processing model outcome\n\nIf time permits: (3 m ins)\nBiases in word embeddings: Case study: Examples from word2vec embedd ings \n\nClosing Remarks: \nWhat is fair and what is not is contextual. \n Importance of Human inputs/judgement in designing debiasing techniques in ML workflows\n\nSlides in Progress\nhttps://docs.google.com/presentation/d /1KrZBZLvaEKOHp76Idp33I_9UEK_NisLEukJ8bQ0JACo/edit?usp=sharing LAST-MODIFIED:20230108T103046Z LOCATION:Online ORGANIZER;CN="The Fifth Elephant":MAILTO:no-reply@hasgeek.com URL:https://hasgeek.com/fifthelephant/mlops-conference-july-2021/schedule/ fairness-in-ml-how-do-we-build-unbiased-ml-workflows-W3YpqPGt9U4JXWVarUjwV L BEGIN:VALARM ACTION:display DESCRIPTION:How do we build unbiased ML workflows and achieve fairness in Machine Learning? in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Break DTSTART:20210724T084500Z DTEND:20210724T085500Z DTSTAMP:20260421T161610Z UID:session/FXUDbC1hpSiUBV3eucv8gM@hasgeek.com SEQUENCE:0 CREATED:20210706T043141Z LAST-MODIFIED:20210722T045820Z LOCATION:Online ORGANIZER;CN="The Fifth Elephant":MAILTO:no-reply@hasgeek.com BEGIN:VALARM ACTION:display DESCRIPTION:Break in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Fighting fraudsters in email communication at Twilio using Machine Learning DTSTART:20210724T085500Z DTEND:20210724T092000Z DTSTAMP:20260421T161610Z UID:session/4Ke2DhgEfbC13a96w8zMdw@hasgeek.com SEQUENCE:3 CATEGORIES:Speaker to submit deck with full outline of the talk,TBD - to b e decided,TBD - to be decided after the walkthrough. CREATED:20210715T121824Z DESCRIPTION:My name is Sachin Nagargoje. I work at Twilio as a Staff Data Scientist. As a part of this talk proposal I would like to shed some light on the kind of attacks we are facing at Twilio nowadays and how we are ta ckling it via different innovative ways and Machine Learning techniques. I want to showcase what are the challenges we face\, and how we do and what we do to catch such unwanted communication. \n\nAt Twilio\, we serve 2T e mail addresses\, sending 90 B emails per Month with 80\,000+ paying custom ers\, which communicate with 56% of the world’s email addresses every ye ar. We also serve 3+ B phone numbers in 100+ countries with 172K+ paying customers\, and together sends Trillions of SMSs. \n\nAs per Litmus Benchm ark Report\, 2019\, orgs get ROI of $42 for each $1 spend on email. As per Tessian reports\, 2019\, 75% of orgs have faced some sort of phish attach and 96% of attacks arrive via emails. The FBI’s Internet Crime Report s hows that in 2020\, BEC scammers made over $1.8 billion. \n\nSo as you can see\, one of the major challenges we are facing today is the misuse of th e Twilio platform for sending phish/fraud/spam.\n\n## There are various at tacks we face on daily basis:\n- Phishing\n- ATO (Account Take Over) - Fra udster hacks the account by stealing credentials or attacking the account. \n- Fake Accounts - Fraudsters create an account with false info and stol en credit card.\n- Trial Accounts - Fraudster create a dummy account and u tilise the trial Money for sending phish (small scale)\n- Toll Fraud - Fra udster make a call to a premium phone number from small shady carriers\, s haring the revenue later with them.\n- Spam (Good/Bad)\n- Vishing \n\n## Below are the challenges with fraudulent communication:\n- Twilio reputati on is at stake since messages go from Twilio account\n- Hard to detect fra udsters in flight although we can sample a few messages post flight\, afte r stripping PIIs.\n- Generally good accounts are used by fraudsters for su ch frauds \, so they could have some good traffic as well. So Ban or No-Ba n?\n- Labelled Training data for ML modelling\n\n## There are various hint s we use as a features for ML algorithm:\n- Rate of emails per day differe nt from average\n- Engagement rate with emails at receiver’s end\n - Sp am\n - Soft Bounce\n - Mail box full\, recipient server was down\, mes sage was too large\, etc.\n - Hard Bouce\n - Email id does not exists\ , email id is invalid\, etc.\n - Open/Click\n- Email content (after stri pping PII)\n\n## Phish Detection Strategy:\n- Labelled Data Collection\n - Hand curated / Programmatic way\n- Data Preprocessing\n - Stemming\, S topwords\, etc\n- AI Modelling\n - Deep Learning (Bi-Directional LSTM)\n - BERT Language Models\n- Model Deployment at Downstream data\n\n## Below are the actions we take for the suspicious accounts:\n- Email/SMS/Call ca p\n- Manual Review by Fraud Ops team\n- Hard Ban\n\n## Key Takeaways:\n- H ow Twilio help to communicate at Scale\n- Various attacks we face at Twili o.\n- Hints/Symptoms we observe to identify attacks.\n- Solutions we use t o avoid attacks.\n- Illustrate Deep Learning Model for identifying Phish a ttacks.\n- Actions we take on fraudsters.\n- Learnings from our Journey so far.\n- Future steps to stop fraudsters by using AI.\n- Demo of Sift tool - one of the tools we use to detect fraudster’s entry at Twilio (Option al). \n\nPresentation: https://docs.google.com/presentation/d/1HYS1e9jx36k rPxOhYkrny4XLM0jF2PmUh-_KSPogQRQ/edit?usp=sharing LAST-MODIFIED:20230810T072606Z LOCATION:Online ORGANIZER;CN="The Fifth Elephant":MAILTO:no-reply@hasgeek.com URL:https://hasgeek.com/fifthelephant/mlops-conference-july-2021/schedule/ fighting-fraudsters-in-email-communication-at-twilio-using-machine-learnin g-4Ke2DhgEfbC13a96w8zMdw BEGIN:VALARM ACTION:display DESCRIPTION:Fighting fraudsters in email communication at Twilio using Mac hine Learning in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Birds of Feather session: Doing privacy\, security and ethics in M achine Learning DTSTART:20210724T092000Z DTEND:20210724T095000Z DTSTAMP:20260421T161610Z UID:session/XhTe8pXYjvXYrgHhvTgek6@hasgeek.com SEQUENCE:0 CREATED:20210720T102319Z LAST-MODIFIED:20210722T085758Z LOCATION:Online ORGANIZER;CN="The Fifth Elephant":MAILTO:no-reply@hasgeek.com BEGIN:VALARM ACTION:display DESCRIPTION:Birds of Feather session: Doing privacy\, security and ethics in Machine Learning in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Theme: Economies of Machine Learning DTSTART:20210724T095000Z DTEND:20210724T100000Z DTSTAMP:20260421T161610Z UID:session/3iAkECx1JSdBUTaqinzsiT@hasgeek.com SEQUENCE:0 CREATED:20210706T043229Z LAST-MODIFIED:20210722T045829Z LOCATION:Online ORGANIZER;CN="The Fifth Elephant":MAILTO:no-reply@hasgeek.com BEGIN:VALARM ACTION:display DESCRIPTION:Theme: Economies of Machine Learning in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Sponsored talk: Scribble Enrich - second generation Feature Engine ering platform DTSTART:20210724T100000Z DTEND:20210724T101500Z DTSTAMP:20260421T161610Z UID:session/DCN7kDcgwuBuJnKRuYpfnf@hasgeek.com SEQUENCE:2 CATEGORIES:30 min talk,Managing models in production,Organize walkthrough between speaker and editor CREATED:20210706T043254Z DESCRIPTION:Mid-Market Feature Stores - Whats the Big Deal?\n----\n\nMajor ity of feature store products and discussions have a specific\npattern: La rg(ish) company\, clean interfaces\, and large volumes of\ndata. They are very impressive but not good fit for mid-market\nenterprises. Scribble has built and operated feature stores for the\npast few years for mid-market enterprises. That experience has led to\na different architecture and thin king around the feature stores. This\ntalk expands on the same.\n\n1. Mid- market context - Are all feature stores the same? \n (a) RoI as a first class consideration\n (b) Data quality and volume\n (c) People and ski ll availability\n (d) Clarity on usecases\n (e) Diversity of needs (da ta\, integration\, applications)\n (f) Time to deliver pressures\n (g) Data/system complexity limits \n \n2. Designing for Mid-Market - Enrich story\n (a) Fast to deploy & use\n (b) Metadata services\n (c) Opti mize for people (skill\, time etc.)\n (d) Flexible interfaces to allow e volution of needs\n (e) Integrate into workflows/processes\n (f) KPIs = usecases/time\, cost/question\n (g) Extend upstream and downstream\n (h) Privacy as a built in capability\n\n3. Interoperation - How will syst ems emerge?\n (a) Not a binary decision\, no fundamental conflict\n (b ) Used in combination\, on different paths\, different subspaces\n LAST-MODIFIED:20230810T072606Z LOCATION:Online ORGANIZER;CN="The Fifth Elephant":MAILTO:no-reply@hasgeek.com URL:https://hasgeek.com/fifthelephant/mlops-conference-july-2021/schedule/ mid-market-feature-stores-whats-the-big-deal-DCN7kDcgwuBuJnKRuYpfnf BEGIN:VALARM ACTION:display DESCRIPTION:Sponsored talk: Scribble Enrich - second generation Feature En gineering platform in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Rethinking linguistic diversity and inclusion in the context of te chnology DTSTART:20210724T101500Z DTEND:20210724T104500Z DTSTAMP:20260421T161610Z UID:session/WzLomsYKyPuwCxX8iHq8Qd@hasgeek.com SEQUENCE:1 CATEGORIES:30 min talk,Model governance,Organize walkthrough between speak er and editor CREATED:20210706T042950Z DESCRIPTION:Natural Language processing is undergoing a paradigm shift rig ht now\; the models today are ever more powerful\, capable and accurate. T hus\, it looks like we are close to achieving the holy grail of AI. Howeve r\, these assertions are true only for a handful of the world’s language s which have enough language data to train powerful and large models. What about the rest of the languages?\nIn this talk\, we will ask and attempt to answer the following questions around linguistic inclusion and diversit y in the context of technology: Which languages are included in our curren t technologies\, and which are left out? What is in store for the speakers of languages which do not reap the benefit of modern technology? And most importantly\, how can we build technology that is more inclusive and a co mmunity that values the need for linguistic diversity and inclusion?\n LAST-MODIFIED:20230108T103046Z LOCATION:Online ORGANIZER;CN="The Fifth Elephant":MAILTO:no-reply@hasgeek.com URL:https://hasgeek.com/fifthelephant/mlops-conference-july-2021/schedule/ rethinking-linguistic-diversity-and-inclusion-in-the-context-of-technology -WzLomsYKyPuwCxX8iHq8Qd BEGIN:VALARM ACTION:display DESCRIPTION:Rethinking linguistic diversity and inclusion in the context o f technology in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Story-telling as a method for building production-ready Machine Le arning systems DTSTART:20210724T104500Z DTEND:20210724T113000Z DTSTAMP:20260421T161610Z UID:session/MkNFa4ZGDC7jA6uE9MUKy5@hasgeek.com SEQUENCE:2 CATEGORIES:30 min talk,Managing models in production,Organize walkthrough between speaker and editor CREATED:20210706T043333Z DESCRIPTION:This presentation is about the essential but overlooked role s torytelling plays in machine learning productionization. Although often as sumed to be at best a tangential concern\, machine learning systems only b ecome successful through successfully building trust in the model\, the da ta\, and the system in which the model and data operate. That trust-buildi ng happens through storytelling. Most often\, successful storytelling happ ens only by accident\, which is why so many machine learning systems fail to make good on the hopes and promises that motivate their construction in the first place. By learning to tell better stories - which requires us t o build better stories - we can build more stable\, maintainable\, and suc cessful machine learning systems.\n\n# Trust-building\n\nA big part of pro ductionizing machine learning is building trust that the models actually d o what they claim to do. That trust-building is different depending on who se trust you're trying to win.\n\nOn one level\, the data scientists and e ngineers building\, training\, and deploying the model need to have confid ence that it actually does what it's supposed to do. This trust is built t hrough tools that are both widely available and commonly taught in trainin g courses. These include things like goodness-of-fit measures\, precision and recall\, checks against overfitting\, cross-validation\, and basic mod el parameterization such as methods for dealing with imbalanced classes. A ll of these tools build trust in the model itself.\n\nOn another level\, d ata scientists and often product managers and other stakeholders need to h ave confidence that the data-generating process that feeds the model does not have any major structural features that will thwart the model and bias its results. The toolset for this level of trust-building is not very rob ust\, usually subsumed under the heading of "exploratory data analysis". M any companies have learned\, to their detriment\, that systemic biases are easy to overlook\, even by skilled practitioners. The productionization o f machine learning requires a more robust set of tools for helping model d esigners and other stakeholders to thoroughly explore the data for hidden biases. We need tools to assess the trustworthiness of the data\, regardle ss of how much trust we have in the model.\n\nAt still another level\, par ticularly non-technical stakeholders need to have confidence in how variou s models fit into one another\, and into other technical systems\, and int o other business processes\, regardless of the extent to which those proce sses are facilitated by technology. There are no well-defined tools for th is kind of trust-building\, though there are relevant skills. At the very least\, we need regular attempts to assess the trustworthiness of a machin e-learning model's place in the larger system that is supposed to produce the results that justify the implementation of machine learning in the fir st place.\n\n# Story-telling\n\nAs we move up through the levels of trust- building\, - from trust in the model\, to trust in the data\, to trust in the system - we increasingly transition from primarily writing code to pri marily telling stories. Productionized machine learning - the automation o f all or parts of decisions that would otherwise be left up to human judge ment - inherently involves the active\, iterative negotiation of a story a bout what is happening\, why it is happening\, and what we can do about it . The role of storytelling in machine-learning productionization is so oft en overlooked that it's rarely considered part of the relevant skill set a t all\, with many practitioners limiting it to notions of report-writing a nd data visualization\, or even considering all concerns about storytellin g as simply a marketing ploy that is hardly related to the machine learnin g problem at all.\n\nA greater focus on the storytelling aspect of product ionizing machine learning increases buy-in from both stakeholders and cust omers\, but it also makes for better models and better code. As we build t rust in the system we discover assumptions about the data and its intended uses that informs our exploration of the data-generating process. As we b uild trust in our understanding of the data-generating process\, we make b etter-informed choices about the models we use (and whether we really need to use a machine learning model at all) and can often mitigate data probl ems before they become problems.\n\n# Agenda\n\nI will only briefly discus s the issue of building trust in models\, as that topic is generally alrea dy very familiar to practitioners. One issue I will focus on is the fact t hat the selection of model-evaluation metrics often depends only a little on the nature of the model itself and more on the nature of the problem th e model is supposed to solve. I’ll illustrate this point using a case st udy from a charter school network\, where my team designed a model to pred ict student need for an academic intervention (either negative - special a ccommodations or being held back in school - or positive - extra challenge s or being skipped ahead a grade). Our evaluation of the model’s perform ance was wholly tied up in our stakeholder’s judgements about the cost o f different kinds of wrong decisions - namely\, how many students were the y willing to give special accommodations to\, even though those students d idn’t really need them\, in order to get the accommodations to one stude nt who really did need them? The success on our model depended upon minimi zing false positives and false negatives\, but only when a vastly differen t priority was assigned to one than the other.\n\nI will first approach th e topic of building trust in data through the perspective of numerous scan dals and exposes that roiled the tech industry over the last several years . While some of these scandals involved misuse of technical systems by mal icious actors\, much more commonly\, data scientists and engineers seemed to have been content to trust their model without thoroughly exploring whe ther they had fed that model trustworthy data. I will discuss a couple too ls that have been developed to evaluate data trustworthiness. In each case \, the tool is really facilitating a conversation between the data and the person using the data\, allowing the data to offer “surprising” comme ntary on itself\, even if the practitioner doesn’t ask for it. This back -and-forth between data and scientist is the core pattern for successfully building trust in data. I will illustrate this principle with a case stud y from a digital advertising firm\, where I built and open-sourced a tool for rapid exploration of geospatial data. The tool allowed me to discover multiple biases in the company’s mobile-device location data that allowe d us to improve several downstream systems as well as develop entirely new products.\n\nI will approach the topic of building trust in systems with case studies from my current company\, Aampe\, where we enable companies t o constantly adapt their push notifications to their users’ preferences through continuous\, massively-parallel experimentation. Most of the real breakthroughs we have had in the design of our machine learning systems ha ve come through our attempts to frame our capabilities in a way that allow s non-technical customers to understand how Aampe fits into their business and their other technical systems. Repeatedly\, as we’ve struggled to t ell the story of how a piece of Aampe’s analytic pipeline works\, we hav e discovered that a crucial piece of that pipeline was either missing or p oorly designed. I will focus on two examples: our decision to make our sys tem learn about one particular thing - app content - in an entirely differ ent way than we learn about everything else\; and our ongoing efforts to d esign KPI tracking and multi-KPI optimization in our learning systems. Thi s last effort has been greatly informed by a recent effort that has been t he most obviously story-telling experience of my career: the creation of a children’s book\, *The User Story*\, that explains our entirely analyti c pipeline through the perspective of a little orange dot that represents an app user.\n\nI will end with some guidelines for building better storie s for machine learning systems\, with an emphasis on ethnography\, a metho d from anthropology that prioritizes iterative\, negotiated construction o f stories. The ability of a story to positively shape a machine learning s ystem depends less on how the story is told (though that matters\, too)\, and more on how the story was built. When stories are systematically itera ted upon - in a process very similar to code review - that process itself provides most of the information needed to build better systems.\n\n LAST-MODIFIED:20230810T072606Z LOCATION:Online ORGANIZER;CN="The Fifth Elephant":MAILTO:no-reply@hasgeek.com URL:https://hasgeek.com/fifthelephant/mlops-conference-july-2021/schedule/ story-telling-as-a-method-for-building-production-ready-machine-learning-s ystems-MkNFa4ZGDC7jA6uE9MUKy5 BEGIN:VALARM ACTION:display DESCRIPTION:Story-telling as a method for building production-ready Machin e Learning systems in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Summary for the day's sessions\; key takeaways DTSTART:20210724T113000Z DTEND:20210724T114000Z DTSTAMP:20260421T161610Z UID:session/8sCCPQ1E85MruVTUyFz9bL@hasgeek.com SEQUENCE:0 CREATED:20210706T043432Z LAST-MODIFIED:20210723T104320Z LOCATION:Online ORGANIZER;CN="The Fifth Elephant":MAILTO:no-reply@hasgeek.com BEGIN:VALARM ACTION:display DESCRIPTION:Summary for the day's sessions\; key takeaways in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Theme: Feature Store in ML DTSTART:20210727T083000Z DTEND:20210727T084000Z DTSTAMP:20260421T161610Z UID:session/Ek997qaNpJdFWhWr8GYdbC@hasgeek.com SEQUENCE:0 CREATED:20210706T043534Z LAST-MODIFIED:20210722T013453Z LOCATION:Online ORGANIZER;CN="The Fifth Elephant":MAILTO:no-reply@hasgeek.com BEGIN:VALARM ACTION:display DESCRIPTION:Theme: Feature Store in ML in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Past and future of feature stores DTSTART:20210727T084000Z DTEND:20210727T091000Z DTSTAMP:20260421T161610Z UID:session/GvexRCG9yzeaS1Sfxt9XbE@hasgeek.com SEQUENCE:1 CATEGORIES:Editor to share feedback on the outline,TBD - to be decided,TBD - to be decided after the walkthrough. CREATED:20210706T043543Z DESCRIPTION:Audience Level: Intermediate\nNature: Conceptual\n\nScribble h as built and operate feature stores for companies for the\npast few years. This is a perspective talk on why feature stores came\nabout\, what is be ing built today\, and what we foresee over the next\nfew years.\n\n1. Feat ure store introduction and history\n\n2. Understanding existing feature st ores\n (a) Architecture: Integrated/standalone\n (b) Scale: Peta/Tera\ n (c) Core abstraction: SQL-like/program\n (d) Application scope: ML/N on-ML\n (e) Programming Iterface: Tight/Open\n (f) Data classes: Strea ming/Timeseries\, document\, transactions\n\n3. Classes of decisions\n ( a) What\, Shallow why\, Deep why\, Why not\n (b) How these are addressed today & gaps\n \n4. Feature Stores 1.0: Passive\, robust\, scalable\n (a) Focused on ML usecases\n (b) Focus on scale & abstractions\n (c) Passive but robust\n\n5. Feature Stores 2.0: Intelligent\, trusted\, end-t o-end\n (a) Context-aware - Integrates with upstream and downstream\n About data\, nature of processing\, risks involved\n Changes ope rations\, resources\, observation levels\n (b) Knowledge management - He lp ip creation\n Better and efficient processes \n (c) Risk manage ment - Trust and safety as first class goal\n Reduce risks from inse cure\, poor/changing code & data\n Change handling\, impact assessme nt\n (d) Proactive - Actively observes and recommends\n Suggests f eatures\, impact assessment\n (e) Scope - Expanded classes of decisions and users \n All classes of advanced data needs (shallow why...)\n (f) Distributed - Handle constraints (time\, volume etc)\n Data can not/should not flow to centralized\n Distributed discovery\n\n6. Som e niche contexts where new classes of\n stores might emerge:\n (a) Con strained devices (handhelds) \n (b) Classes of data (geospatial)\n ( c) Computational complexity (1000s of models)\n\n7. Key Takeaways\n (a) Feature stores are now a standard component\n (b) Understanding the jour ney will help future-proof your implementation\n (c) Feature stores 2.0 will be different from 1.0\n LAST-MODIFIED:20230108T103046Z LOCATION:Online ORGANIZER;CN="The Fifth Elephant":MAILTO:no-reply@hasgeek.com URL:https://hasgeek.com/fifthelephant/mlops-conference-july-2021/schedule/ past-and-future-of-feature-stores-GvexRCG9yzeaS1Sfxt9XbE BEGIN:VALARM ACTION:display DESCRIPTION:Past and future of feature stores in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Myntra home page personalization: optimizing online Feature Store at scale DTSTART:20210727T091000Z DTEND:20210727T094500Z DTSTAMP:20260421T161610Z UID:session/H8FPtWjRRooqQkmheWetco@hasgeek.com SEQUENCE:1 CATEGORIES:Speaker to submit deck with full outline of the talk,30 min tal k,ML deployment workflows CREATED:20210716T072136Z DESCRIPTION:This talk is mainly about productionizing ML Model and Optimiz ing Online Feature Store at the scale of India’s Biggest Fashion E-Comme rce. \nMyntra Home Page consists of widgets(targeted cards). Currently\, their ranking is the same for all the users based on the business metrics( CTR\, Revenue). To truly personalize the home page\, we’ve deployed the ranking ML Model which talks to feature stores to get user & widget featur es in real-time. \nArchitecture design explains all the components in a de tailed fashion. Components discussed: App service\, ML Platform\, Feature Store Lookup\, Aggregation\, Encoder\, Model Predict\, Ranking. There are four broad categories of features used: user embedding\, user activity cou nts\, widget embedding\, widget business metrics. One of the major tech ch allenges is to fetch all the features in real-time with very low latency. In many cases feature lookup is more expensive than the actual Model predi ction. To reduce the feature lookup time\, we’ve done five types of Opti mization\, which is giving us ~ 10X improvement. I’ll be explaining why we choose Aerospike as the feature store\, will share a detailed compariso n with another alternative DB. \nWill be talking about the end-to-end syst em design comprising different layers like Sources\, Ingestion\, Processin g\, Storage\, MLP\, Serving\, Client. In the end\, we’ll demonstrate the benchmark and load test of the personalization service for a very huge sc ale. \n\nAgenda:\n- Objective\n- Tech Challenges\n- Features\n- System Des ign\n- Feature Store Lookup\n- Architecture Design\n- Why Aerospike\n- Opt imizations\n- Benchmarking\n\nKey takeaways: \n- Ideal Feature Store Choic e for real-time feature lookup from Millions of user’s feature data.\n- Different Optimizations to reduce the Feature lookup latency in aerospike. \n- Why Aerospike? \n- How to productionize ML Models at the scale.\n- How ML Platform talks to different components & layers in real-time for each API call.\n- Learning about the end-to-end system design\, starting from S ource\, Ingestion\, Processing\, Storage\, MLP\, Serving.\n- Benchmarking & Load test for the entire ML service.\n\nAudience:\nAnyone looking for Pr oductionization of ML Models\, Feature Pipelines\, Feature Store working a t scale. \nIdeal for the entire ML community.\n\nAudience Level: Intermedi ate\n\nslides: https://drive.google.com/file/d/1TKpiGwKA3ETf-IHE8yjozRtyEs g9XsUI/view?usp=sharing LAST-MODIFIED:20230108T103046Z LOCATION:Online ORGANIZER;CN="The Fifth Elephant":MAILTO:no-reply@hasgeek.com URL:https://hasgeek.com/fifthelephant/mlops-conference-july-2021/schedule/ myntra-home-page-personalization-optimizing-online-feature-store-at-scale- H8FPtWjRRooqQkmheWetco BEGIN:VALARM ACTION:display DESCRIPTION:Myntra home page personalization: optimizing online Feature St ore at scale in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Managed Feature Store: Improving data reusability & Providing a me ans for low latency real-time prediction at Udaan DTSTART:20210727T094500Z DTEND:20210727T102500Z DTSTAMP:20260421T161610Z UID:session/HsZnfC4VUNdWUyJXXwfp5m@hasgeek.com SEQUENCE:2 CATEGORIES:Speaker to submit deck with full outline of the talk,TBD - to b e decided,TBD - to be decided after the walkthrough. CREATED:20210719T111915Z DESCRIPTION:A brief talk on Managed Feature Store built on top of Open Sou rce Feast. We will start with a brief walkthrough of the open source Feast feature store including the architecture and core capabilities. We would call out some of the challenges/limitations of the open source Feast featu re store. We would then describe some of the enhancements which enables us to have a more robust\, secure and scalable deployment by using a) manage d resources on Cloud platforms for eg\, Kafka vs Event Hub (Azure)\, open source Spark vs Databricks\; b) Integration of RBAC & Table Level Access C ontrol to maintain controlled usage c) Scalable batch ingestion by using Spark instead of Pandas & addition of new capabilities to increase data re usability.\n\n__Speakers__: \n [Dr Mohit Kumar](https://www .linkedin.com/in/mohitkum/) (Head - Data Science\, Product Analytics and D ata Platform)\n [Sai Sharan Tangeda](https://www.linkedin.c om/in/sai-sharan-tangeda/) (Data Scientist)\n__Time__: 30 mins\n\n\n### Ag enda\n1. __Introduction__\n 1. Introduction \n 2. Motivation for m aintaining a Managed Feature Store\n2. __Feast (Open Source): Constructs\, Core Capabilities & Limitation__\n 1. Constructs & Architecture of Fea st\n 2. Point In Time Join Capabilities with Batch Retrieval\n 3. Ba tch Ingestion into Historical Store & Scale Limitations\n 4. Streaming Capabilities with Apache Kafka & Redis\n 5. Reliability issues with sel f deployed resources like Kafka\, Redis\, PostgreSQL\n3. __Managed Feature Store as a fork of Feast__\n 1. Overview of Core Architecture\n 2. Integration of Azure Eventhubs as a replacement for Apache Kafka\n 3. I ntroducing Databricks as Spark Backend\n 4. Ensuring Scalability for la rge data sizes via Spark\n 5. RBAC & Table Level Access Control for con trolled usage\n 6. End-to-End flow for real-time model serving\n4. __Cl osing Arguments__\n 1. Increase in Productivity with ready-to-use Featu res\n\n\nLink to slides: https://drive.google.com/file/d/1ocJNDbEUxXVJqyBV D35k-Vvr1y8hjN5k/view?usp=sharing\n LAST-MODIFIED:20230810T072606Z LOCATION:Online ORGANIZER;CN="The Fifth Elephant":MAILTO:no-reply@hasgeek.com URL:https://hasgeek.com/fifthelephant/mlops-conference-july-2021/schedule/ managed-feature-store-improving-data-reusability-providing-a-means-for-low -latency-real-time-prediction-at-udaan-HsZnfC4VUNdWUyJXXwfp5m BEGIN:VALARM ACTION:display DESCRIPTION:Managed Feature Store: Improving data reusability & Providing a means for low latency real-time prediction at Udaan in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Birds of Feather (BOF) session - Feature Stores - adaptations\, us e cases and ROI DTSTART:20210727T102500Z DTEND:20210727T110500Z DTSTAMP:20260421T161610Z UID:session/3v2Kh2r9ysBTCpX1727QTj@hasgeek.com SEQUENCE:1 CREATED:20210720T102809Z DESCRIPTION:The following questions will be discussed: \n\n1. Where do fea ture stores fit in in data science process and stack?\n2. Do organizations need one?\n3. When should an organization build one?\n4. What are the req uirements? (correctness\, integration\, latency etc.)\n5. How does one go about building one?\n6. What day to day operations of a feature store look like?\n7. What problems does a feature store solve? and where does this f it in the lifecycle of a ML model?\n8. Is Feature Store a replacement for the data warehouse (such as delta lake etc)?\n9. How does a Feature Store help an e-commerce organization?\n10. What are some advantages of using Fe ature Store in online serving? How does it make the life of an engineer an d data scientist better?\n11. How does the biggest fashion e-commerce in I ndia does the home page personalization? What's role is played by the Feat ure Store\n12. How Online feature Store is different from the Offline feat ure store?\n13. Ideal choice for the Online Feature Store? What are the ke y parameters based on which this decision is made? \n14. How does the late ncy for feature lookup impact the overall response time? LAST-MODIFIED:20230108T103046Z LOCATION:Online ORGANIZER;CN="The Fifth Elephant":MAILTO:no-reply@hasgeek.com BEGIN:VALARM ACTION:display DESCRIPTION:Birds of Feather (BOF) session - Feature Stores - adaptations\ , use cases and ROI in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Flash talks by participants DTSTART:20210727T110500Z DTEND:20210727T112500Z DTSTAMP:20260421T161610Z UID:session/2tW59L1oyQyTibAMQJqFVz@hasgeek.com SEQUENCE:0 CREATED:20210720T102831Z LAST-MODIFIED:20210723T104351Z LOCATION:Online ORGANIZER;CN="The Fifth Elephant":MAILTO:no-reply@hasgeek.com BEGIN:VALARM ACTION:display DESCRIPTION:Flash talks by participants in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Summary for the day's sessions\; key takeaways. DTSTART:20210727T112500Z DTEND:20210727T113500Z DTSTAMP:20260421T161610Z UID:session/2ZXVgqc2VbdSSV3qB8P4uo@hasgeek.com SEQUENCE:0 CREATED:20210706T043703Z LAST-MODIFIED:20210723T104353Z LOCATION:Online ORGANIZER;CN="The Fifth Elephant":MAILTO:no-reply@hasgeek.com BEGIN:VALARM ACTION:display DESCRIPTION:Summary for the day's sessions\; key takeaways. in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Oct-Nov edition of MLOps DTSTART:20210727T113500Z DTEND:20210727T114000Z DTSTAMP:20260421T161610Z UID:session/BwZ2K5cTrsCM5mAvc7SDGT@hasgeek.com SEQUENCE:0 CREATED:20210721T044601Z LAST-MODIFIED:20210723T104356Z LOCATION:Online ORGANIZER;CN="The Fifth Elephant":MAILTO:no-reply@hasgeek.com BEGIN:VALARM ACTION:display DESCRIPTION:Oct-Nov edition of MLOps in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT END:VCALENDAR