Rootconf 2014

On devops and cloud infrastructure

Scaling ElasticSearch for Log and Event monitoring

Submitted by Arpit Mohan (@mohanarpit) on Tuesday, 21 January 2014

Section: Full talk Technical level: Beginner Status: Rejected


Tracking application logs and events using ElasticSearch thereby giving a fast and simple way to query them and generate useful metrics.


In any application, logs and events form the basis of measuring, debugging and monitoring the application. Tracking events also allows businesses to measure the impact of features, new deployments and closely keep an eye on numbers that matter and take decisions based on these metrics.

There are many great tools for working with either logs or metrics (StatsD, Loggly etc). Unfortunately, most tools out there focus on only one of them. ElasticSearch is a generic (and very scalable) tool which allows developers to not only store and parse logs but also store rich event data. This event data is then used to create usable metrics.

In this talk, I aim to highlight how to leverage and scale an ElasticSearch cluster for the purposes of log/event tracking.

Speaker bio

Arpit Mohan - CTO, Clinknow
A technology enthusiast with a penchant for leveraging technology to solve hard problems, he started his foray into the startup world with Gharpay. Currently, he leads the technology efforts at Clinknow, a BigData initiative with a focus on financial analysis.



  •   Sreekandh Balakrishnan (@gnuyoga) 6 years ago

    Hi Arpit,
    Very intersting talk. Will you be demonstrating the limitation of the mentioned tools like Loggly/Papertrail ? I was always curious to create a smart Operational Dashboard for my servers / services that i run, Do you have something to showcase around the same ?

    •   Arpit Mohan (@mohanarpit) Proposer 6 years ago

      We are currently using the Kibana dashboard to not only view the logs but also the events that we capture when the application runs. We do have plans to customize a dashboard and use it. I’ll try to demo our dashboard during the talk. Hopefully, we’ll have something stable by then.

  •   Harpreet singh Wadhwa (@wharpreet) 6 years ago

    Hi Arpit,
    Topic is interesting and very vast. Will it be possible to add some more info on the depth that you might be covering in the talk.

  •   anand verma (@rahulvit09) 6 years ago

    Are you using logstash for log collection and parsing?

Login with Twitter or Google to leave a comment