Submitted Jun 17, 2019

Real-Time DataQuality on Flink

Session type: Full talk of 40 mins

My use case is to provide monitoring, and improving the overall search data quality, also to find the unusual patterns of user’s search behavior, and notifying the intent on-site back to the respective business stakeholders. To achieve the same, I explored various big data processing engines, which can process the huge data with complex business logic in real time. Eventually, I used Flink Stream processing. This talk will showcase how I used Flink to accomplish my goal.

Outline

What is Real Time Aggregation ?
System Requirement
Flink vs Spark
Flink Cluster setup
Flink on Yarn
Architecture
100% data completeness
Batch vs Realtime
Open Items

Speaker bio

I am a Staff Software Engineer in Walmart and Apache Oozie Committer. I am currently trying to solve some of the search problems. I am in Big Data space since last 10 years.

Slides

https://www.slideshare.net/jaydeepmail/real-time-data-quality-on-flink

Comments

Up next

Interpretable NLP Models

Deep learning models are always known to be a black box and lacks interpretability compared to traditional machine learning models. So,There is alway a hesitation in adopting deep learning models in user facing applications (especially medical applications). Recent progress in NLP with the advent of Attention based models , LIME and other techniques have helped to solve this. I would like to walkthough each of the techniques and share my experience in deploying explainable models in production.

May 31, 2019

The Fifth Elephant round the year submissions for 2019