Miniconf on Cloud Server Management (Chennai)

On costs, scaling and securing cloud servers



Building and scaling a log analytics platform - a serverless approach

Submitted Oct 28, 2017

Serverless architectures has been around for past few years and there has been quite a few skepticism surrounding it. Few might argue that it’s just another buzzword for marketing. But serverless architectures offer more than a catchy buzzword. In this talk we will discuss, what is serverless, when to and when not to use them and how can we use Amazon Web Services to implement a real-time, production grade serverless logging pipeline. By the end of the talk, audience will get an introduction to serverless and also get to know how to design, deploy and scale infrastructures using the same.


Being a Product Engineer who uses serverless functions as a part of products I build, I closely experience how one can leverage serverless architectures to design a resource efficient and highly scalable infrastructure. The infrastructure provider we use which is AWS, provides a range of FaaS components from the popular lambda functions to other managed services like athena, kinesis firehose and quick sight. In this talk, as I give an introduction about serverless, We will walk through how we use them in production enabling resource optimization and low maintenance time. By the end of this talk we would’ve implemented an end to end logging pipeline and brought the generated sample logs to presentation tier for business insights. Not to mention, the system we setup can scale to handle 1000s of microservices and billions of log messages.


  1. Introduction
  2. How our microservices looks like?
  3. Architecting a logging framework
  4. How the framework should be?
  5. Metrics we needed
  6. The conventional approach
  7. Kafka, Cold storage, ELK
  8. Problems we faced
  9. Going Serverless
  10. What is serverless
  11. Advantages
  12. The FaaS logging architecture
  13. AWS Athena
  14. AWS lambda
  15. Kinesis firehose
  16. Kinesis analytics
  17. AWS quicksight
  18. AWS s3
  19. Demo
  20. Caveats
  21. Summary

Speaker bio

Naren is a Product Engineer with specific focus on building robust backend and scalable systems. He works on open source projects in his spare time. He loves speaking at tech conferences and currently helping MadStreetDen in scaling their Artifical Intelligence products. In his 4 years of industry experience he’s worn plenty of hats- like the one of a Trainer, Embedded Engineer and Backend/Product Engineer and sometimes even helmets- when he’s out cycling.
When he’s not stirring up code, you can find him whipping up a delicious gluten-free treat or travelling/cycling.



{{ gettext('Login to leave a comment') }}

{{ gettext('Post a comment…') }}
{{ gettext('New comment') }}
{{ formTitle }}

{{ errorMsg }}

{{ gettext('No comments posted yet') }}

Hosted by

We care about site reliability, cloud costs, security and data privacy