Rootconf Hyderabad edition

Metrics, logging and tracing are what is commonly described as the three pillars of observability, however, limiting ourselves to these signals makes us miss out on many other aspects of operating a system. A signal that is not well developed enough is continuous profiling. Have you ever been in the situation where you really needed a memory profile of a process just before it was OOMKilled? Or wanted to see the difference of memory allocations across releases? These are common scenarios especially when running on Kubernetes as we try to be ever more efficient with resource constraints. This talk covers a brand new open source project to continuously profile applications, which borrows many concepts from existing popular systems such as Prometheus. We also cover a couple of bugs that we could only debug using Conprof.

Join this talk to learn about the next evolution of performance engineering as part of observability.

Outline

We first talk about how the existing observability solutions fall short when debugging performance. There are random CPU/Memory spikes in production and there is no way of know which module/function is responsible for that. Further, it is not visible in local benchmarks and the spikes are short enough that it’s not possible to grab profiles after being alerted.

Then we introduce conprof and how it works and the tradeoff’s we made. We also explain the overheads of continuously profiling applications and how we offset some of them. Finally we end with a case-study of a production outage that conprof helped debug and fix.

Speaker bio

Goutham is a developer from India who started his journey as an infra intern at large company where he worked on deploying Prometheus. After that initial encounter, he started contributing to Prometheus and interned with CoreOS, working on Prometheus’ new storage engine. He is now a maintainer for TSDB, the engine behind Prometheus 2.0. He now works at Grafana Labs on open-source observability tools. When not hacking away, he is either on his bike, or is binge watching GCN!

Links

• https://github.com/Go-SIP/conprof