Building and scaling a log analytics platform - a serverless approach
Submitted by Narendran (@dudewhocode) on Friday, 27 October 2017
Serverless architectures has been around for past few years and there has been quite a few skepticism surrounding it. Few might argue that it’s just another buzzword for marketing. But serverless architectures offer more than a catchy buzzword. In this talk we will discuss, what is serverless, when to and when not to use them and how can we use Amazon Web Services to implement a real-time, production grade serverless logging pipeline. By the end of the talk, audience will get an introduction to serverless and also get to know how to design, deploy and scale infrastructures using the same.
Being a Product Engineer who uses serverless functions as a part of products I build, I closely experience how one can leverage serverless architectures to design a resource efficient and highly scalable infrastructure. The infrastructure provider we use which is AWS, provides a range of FaaS components from the popular lambda functions to other managed services like athena, kinesis firehose and quick sight. In this talk, as I give an introduction about serverless, We will walk through how we use them in production enabling resource optimization and low maintenance time. By the end of this talk we would’ve implemented an end to end logging pipeline and brought the generated sample logs to presentation tier for business insights. Not to mention, the system we setup can scale to handle 1000s of microservices and billions of log messages.
- How our microservices looks like?
- Architecting a logging framework
- How the framework should be?
- Metrics we needed
- The conventional approach
- Kafka, Cold storage, ELK
- Problems we faced
- Going Serverless
- What is serverless
- The FaaS logging architecture
- AWS Athena
- AWS lambda
- Kinesis firehose
- Kinesis analytics
- AWS quicksight
- AWS s3
Naren is a Product Engineer with specific focus on building robust backend and scalable systems. He works on open source projects in his spare time. He loves speaking at tech conferences and currently helping MadStreetDen in scaling their Artifical Intelligence products. In his 4 years of industry experience he’s worn plenty of hats- like the one of a Trainer, Embedded Engineer and Backend/Product Engineer and sometimes even helmets- when he’s out cycling.
When he’s not stirring up code, you can find him whipping up a delicious gluten-free treat or travelling/cycling.