Rootconf 2019

On infrastructure security, DevOps and distributed systems.

Participate

Log Analytics with ELK stack (Architecture for aggressive cost optimization and infinite data scale)

Submitted by Denis Dsouza (@denisdsouza) on Saturday, 12 January 2019

videocam_off
Section: Crisp talk of 20 mins duration Technical level: Beginner

Vote on this proposal

Login to vote

Total votes:  +6

Abstract

Should you build your own log analytics platform or buy one of the many many services out there? Well, we evaluated, compared and decided to build a self managed ELK stack because none of them fit our requirements.
In this session, we will walk through various design choices we made to have a high performing log analytics cluster, aggressively optimized for cost and support for infinite data scale without exponentially increasing cost.
If you are planning on setting up or re-looking at your log analytics, this could be interesting to you.

Outline

Business Requirements/Use Cases
- Log analysis platform (Application, Web-Server, Database logs)
- Data Ingestion rate: ~300GB/day
- Frequently accessed data: last 8 days
- Infrequently accessed data: 82 days (90 - 8 days)
- Uptime: 99.90
- Hot Retention period: 90 days
- Cold Retention period: 90 days (with potential to increase)
- Cost effective solution

Areas of optimization
- Application
- Infrastructure

Cost Optimization
- Replica counts and its impacts
- How to run ELK on Spot instances correctly.
- EBS Costs can be high, how to set up Hot / Cold data storage
- Auto Scaling
- On-demand ELK Cluster

Infinite Data Retention
- How to setup S3 as a hot backup
- Recover on Demand

Numbers/Tradeoffs
- Cost/GB data ingested
- Trade-offs made
- DR mechanisms

Conclusion
- Building a log analytics is not rocket science. But it can be painfully iterative if you are not aware of the options. Be aware of the trade-offs you are OK making and you can roll out a solution specifically optimized for that.

Requirements

Have a need for setting up a log-analytics system at scale or has already done the same.

Speaker bio

I am a DevOps Engineer at Moonfrog Labs.
I have over 6 years of experience and have worked with a variety of technologies in both service-based and product-based organisations.
Now exploring technology in gaming at its best in Moonfrog Labs for the past 1.5 year.

Slides

https://docs.google.com/presentation/d/1-t8J7h0Bv8vtF-CyXdtLWsKgxznlb9v6fWjryF635gE/edit?usp=sharing

Comments

  • 1
    Zainab Bawa (@zainabbawa) Reviewer 3 months ago

    Share draft slides and preview video by 18 February to complete evaluation of your proposal.

    • 1
      Guruprasad GV (@gurugv) 3 months ago
  • 1
    Pooja Shah (@p00j4) 3 months ago

    Hi @gurugv,
    Do you plan to add a live demo which can help attendees undertsand how and impact more effective way?

    • 1
      Guruprasad GV (@gurugv) 3 months ago

      Can do, but won’t be able to do the justice in 20 mins. Essentially want to cover many aspects of cost optimizations, and not just setting up a log analytics stack.

      Would be very interested to hear if you feel there are specific aspects that make sense to do a live demo keeping the above in mind.

      Thanks.

  • 1
    Hari C M (@haricm) Reviewer 2 months ago

    Guruprasad,

    I am not able to open the slides. Can you please set permission correctly.

    • 1
      Guruprasad GV (@gurugv) 2 months ago

Login with Twitter or Google to leave a comment