The Fifth Elephant 2018

Microservices are the building blocks that power a post-cloud digital landscape. They help us in building services which are scalable and eases the deployment and development process.

As scale rises, simple, direct http calls from service to service become bottlenecks, and cascede failures through the system. A distributed, horizontally scalable bus like Kafka becomes essential.

At GO-JEK, we handle 2X of India’s total food tech daily order volume. The restaurants that make the food depend on GO-RESTO, our merchant facing app.

As we scaled, GO-RESTO started having slow queries for displaying the complete order report for a merchant in a day. We could not delete the underlying table as the table were used by other queries. We needed to move this data and the query away from the main service to uphold our SLA.

In order to move this data away from the main service, we started to publish the data to Kafka. Using the Esb Generic Log Consumer that was written for streaming data from Kafka to Postgres, we were able to get the data required to the backend database of the new microservice that we were writing. Then we wrote the backend service capable of handling the queries. In the end, we were able to reduce the latencies as well as take out the load on the main service.

In this talk I will

1. Discuss the approach taken to move away the data required for order report to the new micro services.
2. How the tools that we developed to publish to Kafka and stream the Kafka messages in understandable format to postgres helped us in developing the micro service.
3. The deployment practices that we followed so that the switch to the new micro service was painless from customer point of view

Outline

1.Brief about speaker and Gojek
2. Goresto: What it does
3. Problem: The slow query that resulted in degraded performance in goresto
4. Kafka: What is kafka and how we use kafka in our organisation.
5. Esb Log Client: The client writtem in ruby to send the data in the correct schema to Kafka
6. Esb Generic Log Consumer: The Kafka consumer we wrote to stream data from kafka to postgres.
7. Microservices: What Why and When
8. Mocha: The new microservice in Golang
9. The optimisations: The optimisations in mocha
10. Deployment practices: The practices that ensured that the switch to mocha was painless
11. Comparison: The latencies before and after
12. Conclusion: What we learned
13. Q&A

Speaker bio

Anugrah is a product engineer with GO-JEK who primarily works at food backend. He has worked with data engineering team at GO-JEK previously and helped in building the data pipeline.

Slides

https://www.slideshare.net/anugrahnayar/building-microservcies-using-kafka-93993867