Rootconf 2019

On infrastructure security, DevOps and distributed systems.

Log Analytics with ELK stack (Architecture for aggressive cost optimization and infinite data scale)

Submitted by Denis Dsouza (@denisdsouza) on Jan 13, 2019

Technical level: Beginner Section: Full talk of 40 mins duration Status: Confirmed & Scheduled


Should you build your own log analytics platform or buy one of the many many services out there? Well, we evaluated, compared and decided to build a self managed ELK stack because none of them fit our requirements.
In this session, we will walk through various design choices we made to have a high performing log analytics cluster, aggressively optimized for cost and support for infinite data scale without exponentially increasing cost.
If you are planning on setting up or re-looking at your log analytics, this could be interesting to you.


Business Requirements/Use Cases
- Log analysis platform (Application, Web-Server, Database logs)
- Data Ingestion rate: ~300GB/day
- Frequently accessed data: last 8 days
- Infrequently accessed data: 82 days (90 - 8 days)
- Uptime: 99.90
- Hot Retention period: 90 days
- Cold Retention period: 90 days (with potential to increase)
- Cost effective solution

Areas of optimization
- Application
- Infrastructure

Cost Optimization
- Replica counts and its impacts
- How to run ELK on Spot instances correctly.
- EBS Costs can be high, how to set up Hot / Cold data storage
- Auto Scaling
- On-demand ELK Cluster

Infinite Data Retention
- How to setup S3 as a hot backup
- Recover on Demand

- Cost/GB data ingested
- Trade-offs made
- DR mechanisms

- Building a log analytics is not rocket science. But it can be painfully iterative if you are not aware of the options. Be aware of the trade-offs you are OK making and you can roll out a solution specifically optimized for that.


Have a need for setting up a log-analytics system at scale or has already done the same.

Speaker bio

I am a DevOps Engineer at Moonfrog Labs.
I have over 6 years of experience and have worked with a variety of technologies in both service-based and product-based organisations.
Now exploring technology in gaming at its best in Moonfrog Labs for the past 1.5 year.

How do I spend my free time ?
Learning new technologies and playing PC games



  • Zainab Bawa (@zainabbawa) a year ago

    Share draft slides and preview video by 18 February to complete evaluation of your proposal.

  • Pooja Shah (@p00j4) a year ago

    Hi @gurugv,
    Do you plan to add a live demo which can help attendees undertsand how and impact more effective way?

    • Guruprasad GV (@gurugv) a year ago

      Can do, but won’t be able to do the justice in 20 mins. Essentially want to cover many aspects of cost optimizations, and not just setting up a log analytics stack.

      Would be very interested to hear if you feel there are specific aspects that make sense to do a live demo keeping the above in mind.


  • Hari C M (@haricm) a year ago


    I am not able to open the slides. Can you please set permission correctly.

  • Joseph Pridham 8 months ago

    The server is the place known as the database of the device that is generated in the good and designed matter. The previous proposal has the log analysis with the stack of for testing the infrastructure security by the launch of a builder that gets.

Login to leave a comment