SRE Conf 2023

Abstract

Observability is the key tenet of running a multi-tenant Kubernetes environment. In this talk, I’ll cover the what, why, and how of setting up multi-tenant Observability using open-source tools. This multi-tenant system supported 800+ B2B customers across multiple public clouds and private data centers.

I recently worked on setting up this multi-tenant observability stack for one of the largest health-tech companies in India. We built this using open-source tooling on Kubernetes. The system was designed to handle 10000s of metrics from each tenant and support long-term storage and querying of these metrics.

When thinking about multi-tenancy for Observability, one has to think about the following:

• What are the options and tradeoffs when setting up Kubernetes observability in multi-tenant environments?
• If you decide to use Prometheus, what should you pick? One Prometheus per tenant or one Prometheus for all tenants?
• How do you achieve per-tenant data separation? How do you configure RBAC for access control across multiple tenants?
• How do you monitor not just the “server”, but also the “service” along with the supporting application services (databases, queues, cache, etc.)
• What are some optimizations in ingesting, storing, and querying these metrics such that the cost for the observability infrastructure doesn’t blow up?

Takeaways for the audience

• Learn how to evaluate open-source tooling for large-scale -observability needs.
• Understand the challenges related to security, cost, data separation, etc, with respect to multi-tenant observability.
• Learn how a homogenous infrastructure setup allows you to automate, optimize, and maintain an observability stack at scale.