Being a Product Engineer who uses serverless functions as a part of products I build, I closely experience how one can leverage serverless architectures to design a resource efficient and highly scalable infrastructure. The infrastructure provider we use which is AWS, provides a range of FaaS components from the popular lambda functions to other managed services like athena, kinesis firehose and quick sight. In this talk, as I give an introduction about serverless, We will walk through how we use them in production enabling resource optimization and low maintenance time. By the end of this talk we would’ve implemented an end to end logging pipeline and brought the generated sample logs to presentation tier for business insights. Not to mention, the system we setup can scale to handle 1000s of microservices and billions of log messages.

Outline:

Introduction
How our microservices looks like?
Architecting a logging framework
1. How the framework should be?
2. Metrics we needed
The conventional approach
1. Kafka, Cold storage, ELK
2. Problems we faced
Going Serverless
1. What is serverless
2. Advantages
The FaaS logging architecture
1. AWS Athena
2. AWS lambda
3. Kinesis firehose
4. Kinesis analytics
5. AWS quicksight
6. AWS s3
Demo
Caveats
Summary

Miniconf on Cloud Server Management (Chennai)