Anthill Inside 2019

A conference on AI and Deep Learning

Tickets Propose a session

Tools for AI & ML for machine learning at Scale.

Submitted by Saurabh Misra (@saurabh-appd) on Tuesday, 30 April 2019


Preview video

Section: Full talk Technical level: Intermediate Session type: Lecture

Abstract

As applications become more business critical and application teams are receiving monitoring data for these mission critical business applications as a continuous stream it becomes difficult to manually monitor them and create dashboards/reporting around these applications.

It is becoming increasingly clear that the only way to fix this is to have the right set of tools in place that can help teams to monitor their applications in an automated manner using various machine learning techniques.

We at Appdynamics have built a product that helps address the above requirement using AI and Machine learning. Our algorithms continuously monitor business critical applications, find anomalies and root causes for these anomalies, and give users insights that would have otherwise taken days and weeks to find.

Outline

This talk revolves around the various open source tools being used while implementing the above solution.

Our ML/AI platform learns the normal behavior of an application’s data and find anomalies instantly. We are also able to leverage our understanding of the application’s architecture and the correlation between different metrics. Once the anomalies are detected we automatically correlate the anomalies and events for the fastest Root Cause Analysis.

Collecting , Ingesting , storing, and processing billions of events per second in real time for monitoring the application is not a simple process. For doing this in a seamless manner we would need to identify the right set of tools as well as the Infrastructure/Cost requirements for running these tools.

Some of the Open Source Tools being used by us to achieve the above functionality are
- Apache HBase - Kafka/kafka Streams - Neo4j - Redis - Confluent Avro Schema Registry - Kubernetes

In this session we would talk around why we chose these set of tools considering some of the challenges around Machine Learning at a scale.

Speaker bio

Saurabh is Principal Software Engineer at AppDynamics, where he work on building solutions around real time streaming datapipelines and various machine learning algorithms for automated anomaly detection and Root Cause Analysis of problems.
His interests lies on how to combine data science with software engineering and solve real time business critical problems.

Slides

https://drive.google.com/open?id=1-A_u31znCzxxkBbrLIMMgVxd0JhpKi6m_z8KdoLy0fk

Preview video

https://drive.google.com/open?id=1vdoFRDjXZtt2oIpvumBGCNY044bshXMM

Comments

  • Abhishek Balaji (@booleanbalaji) Reviewer 5 months ago

    Hi Saurabh,

    Thanks for submitting your proposal. The talk in it’s current format would not be suitable as a talk at The Fifth Elephant or Anthill Inside since it lacks focus and a clear takeaway for the audience. Here’s additional feedback:

    1) The proposal only talks about the various open source tools that AppDynamics has used to get ML in production. This might not be interesting for the audience to hear as much as the problems you were facing and how these open source tools helped.

    2) Most of the audience at The Fifth Elephant would be familiar with these tools and are probably already using them in production. What would be interesting to hear is if there were any new tools developed in the course of building this workflow or was there a novel approach in using these tools.

    3) The proposal also does not consider alternate solutions available, including proprietary solutions and platforms.

    Your proposal has to address the following points:

    • The problem statement and what led you to exploring these solutions
    • Approaches taken to solve the problem
    • Why this approach/tool was chosen and how was it benchmarked with others? Add metrics and learnings here.
    • Deep dive into the details
    • Key takeaway for the audience.

    As next steps, incorporate the feedback above into your slides and add them to your proposal by 21 May, so we can consider your proposal for evaluation. If we dont receive the slides by 21 May, we have to move your proposal for consideration under a future event.

Login with Twitter or Google to leave a comment