The Fifth Elephant 2017

On data engineering and application of ML in diverse domains


Unless you measure it; you can’t improve it - Data pipelines for your business KPIs and KRAs

Submitted by Ketan Khairnar (@ketankhairnar) on Thursday, 8 June 2017

Preview video

Section: Workshops Technical level: Intermediate



Any business can gain unfair advantage through actionable insights using data pipelines and some common sense. We’re already experiencing this through our interactions online (amazon , and through mobile apps (uber, ola and many more)

Why Data Infrastructure is important?

Important advantages of data pipelines is

  • surge in productivity - through nice CQRS interface
  • informed decision making - through trends, aggregations, leaderboards
  • Think about it as platform for A/B testing your business goals

In this workshop; you’ll be building data pipeline stack for your airbnb clone. You’ll be wearing multiple hats including.

  • Engineering or SRE team
  • Customer support team
  • Product managers
  • CXO

and come up with quick and nimble solutions for questions at hand.  You’ll also make sure to grow, change and evolve your data pipeline along with the business needs   Add to this two important interaction paradigms which can make

  • conversational patterns using chat bots  - NLP and custom bot server
  • real time dashboards to track KPIs and KRAs – for different stakeholders

Note: Actual businesses are 10x complex but this is good starting point to experiment and explore these ideas.


Course Content

Key actionable insights are worth the effort of building data highway network within your company.

This workshop would introduce data pipelines as concept and help them to build one for pseudo business aka Airbnb clone. It will help them to get conversant with technology as well as thinking about data engineering. i.e. Data you generate and data you consume.

There are many vendors in this segment but impedance mismatch in your engineering practices and data and their feature set is big issue. You building your data pipelines is worth the effort as long as you stick to basics. We’ll talk about this key architectural decisions as well.

Key Takeaways:

  • Telemetry and Audit events within the application
  • Time series database and schema patterns associated with it
  • Few important patterns - Event sourcing, Polyglot persistence, CQRS
  • Transactions and Events as log
  • Chat bots using NLP as interaction pattern for customer support as well as for SRE/Engineering for remote troubleshooting
  • KPI & KRA dashboards using grafana


  • Basic programming skills in java, javascript etc
  • Basic understanding of AWS services EC2, S3, Kinesis, Lambda

We can accommodate 40 participants to make sure we finish on time ( 3 hours )

We’ll be sharing service account keys for AWS with custom built app stack for each participant. It would need 3-4 hours to complete whole set of exercises. I’ll share costs associated with it shortly.

Speaker bio

Ketan has been working on key data pipeline projects for last few years. Building transation log for ad-tech transactions along with audit and throughput event stream helped my earlier employer ( startup ) to solve very disparate goals through same engineering infrastructure (reduce latency and optimize business ).

In his current job Ketan, Rupesh, Sumeet and rest of Ketan’s team has built complete fault monitoring solution for few thousand servers using few off the shelf open soure components. This includes home grown alert management component, bot server, custom reporting jobs. We consider all of these as applications on continuously evolving data pipeline.
This not only helps them solve production issues but helps product management to decide which features are most bang for the buck.Also helps SRE team to remotely troubleshoot data using conversational pattern with smart bot integration in slack.



Preview video


  • Jose Henton 8 months ago

    How can KPI be measured? And which factors need to be assessed?
    Indeed, KPI (Key Performance Indicators) is quite measurable. If you take measurements at certain intervals of time, then you can evaluate more and gradual changes in indicators . This is very useful information for analytical reports.

  • elijah william (@elijahwilliam) 7 months ago

    The crude material of current advanced economy is information. Notwithstanding, much the same as physical crude merchandise, it is useless without unique instrument that can recover and after that procedure it, in order to change over it into helpful products. Consider AI an instrument that causes you in abstracting, dissecting, and interpreting information with the goal that you can utilize it very quickly and can result to better choices.

  • TheTexting (@smsapi) 3 months ago

    Businesses have spread their wings online with consumer trends shifting to the internet and digital communication. This begs the need of smart communication channels that work in real-time and an SMS Gateway is the only channel that can execute real-time communication. visit website:

Login with Twitter or Google to leave a comment