Rootconf 2019

Rootconf 2019

On infrastructure security, DevOps and distributed systems.

About Rootconf 2019:

The seventh edition of Rootconf is a two-track conference with:

  1. Security talks and tutorials in audi 1 and 2 on 21 June.
  2. Talks on DevOps, distributed systems and SRE in audi 1 and audi 2 on 22 June.

Topics and schedule:

View full schedule here: https://hasgeek.com/rootconf/2019/schedule

Rootconf 2019 includes talks and Birds of Feather (BOF) sessions on:

  1. OSINT and its applications
  2. Key management, encryption and its costs
  3. Running a bug bounty programme in your organization
  4. PolarDB architecture as Cloud Native Architecture, developed by Alibaba Cloud
  5. Vitess
  6. SRE and running distributed teams
  7. Routing security
  8. Log analytics
  9. Enabling SRE via automated feedback loops
  10. TOR for DevOps

Who should attend Rootconf?

  1. DevOps programmers
  2. DevOps leads
  3. Systems engineers
  4. Infrastructure security professionals and experts
  5. DevSecOps teams
  6. Cloud service providers
  7. Companies with heavy cloud usage
  8. Providers of the pieces on which an organization’s IT infrastructure runs – monitoring, log management, alerting, etc
  9. Organizations dealing with large network systems where data must be protected
  10. VPs of engineering
  11. Engineering managers looking to optimize infrastructure and teams

For information about Rootconf and bulk ticket purchases, contact info@hasgeek.com or call 7676332020. Only community sponsorships available.

Rootconf 2019 sponsors:

Platinum Sponsor

CRED

Gold Sponsors

Atlassian Endurance Trusting Social

Silver Sponsors

Digital Ocean GO-JEK Paytm

Bronze Sponsors

MySQL sumo logic upcloud
platform sh nilenso CloudSEK

Exhibition Sponsor

FreeBSD Foundation

Community Sponsors

Ansible PlanetScale

Hosted by

Rootconf is a forum for discussions about DevOps, infrastructure management, IT operations, systems engineering, SRE and security (from infrastructure defence perspective). more

Denis Dsouza

@denisdsouza

Log Analytics with ELK stack (Architecture for aggressive cost optimization and infinite data scale)

Submitted Jan 13, 2019

Should you build your own log analytics platform or buy one of the many many services out there? Well, we evaluated, compared and decided to build a self managed ELK stack because none of them fit our requirements.
In this session, we will walk through various design choices we made to have a high performing log analytics cluster, aggressively optimized for cost and support for infinite data scale without exponentially increasing cost.
If you are planning on setting up or re-looking at your log analytics, this could be interesting to you.

Outline

Business Requirements/Use Cases
- Log analysis platform (Application, Web-Server, Database logs)
- Data Ingestion rate: ~300GB/day
- Frequently accessed data: last 8 days
- Infrequently accessed data: 82 days (90 - 8 days)
- Uptime: 99.90
- Hot Retention period: 90 days
- Cold Retention period: 90 days (with potential to increase)
- Cost effective solution

Areas of optimization
- Application
- Infrastructure

Cost Optimization
- Replica counts and its impacts
- How to run ELK on Spot instances correctly.
- EBS Costs can be high, how to set up Hot / Cold data storage
- Auto Scaling
- On-demand ELK Cluster

Infinite Data Retention
- How to setup S3 as a hot backup
- Recover on Demand

Numbers/Tradeoffs
- Cost/GB data ingested
- Trade-offs made
- DR mechanisms

Conclusion
- Building a log analytics is not rocket science. But it can be painfully iterative if you are not aware of the options. Be aware of the trade-offs you are OK making and you can roll out a solution specifically optimized for that.

Requirements

Have a need for setting up a log-analytics system at scale or has already done the same.

Speaker bio

I am a DevOps Engineer at Moonfrog Labs.
I have over 6 years of experience and have worked with a variety of technologies in both service-based and product-based organisations.
Now exploring technology in gaming at its best in Moonfrog Labs for the past 1.5 year.

How do I spend my free time ?
Learning new technologies and playing PC games

Slides

https://docs.google.com/presentation/d/1PiLsy-UwkB1IYhorTq6KWYTCmcuVg2OOK-8D92tlFQs/edit?usp=sharing

Comments

{{ gettext('Login to leave a comment') }}

{{ gettext('Post a comment…') }}
{{ gettext('New comment') }}
{{ formTitle }}

{{ errorMsg }}

{{ gettext('No comments posted yet') }}

Hosted by

Rootconf is a forum for discussions about DevOps, infrastructure management, IT operations, systems engineering, SRE and security (from infrastructure defence perspective). more