Chaos Engineering for stateful applications on kubernetes
A hands-on workshop on testing failover, recovery, and data durability in PostgreSQL on Kubernetes
Chaos Engineering for stateful applications on kubernetes
A hands-on workshop for platform, SRE, DevOps, and database engineers who run stateful workloads on Kubernetes and need to know how they behave when things fail.
Running databases on Kubernetes is no longer an experiment. Operators like CloudNativePG promise automated failover, self-healing, and reconciliation, and teams increasingly rely on those promises in production. But a database pod is not like a stateless one. It holds a role, a replication position, and a volume that must be reattached before it can serve. When it fails, the risks are split-brain, stale reads from a lagging replica, lost writes after an ungraceful promotion, and recovery that silently stalls.
This workshop focuses on testing those promises deliberately, before an incident.
What is this workshop about?
This workshop will help engineers understand what actually happens to a replicated database on Kubernetes when parts of it fail.
Participants will learn how failures move through the data plane (the PostgreSQL instances serving traffic) and the control plane (the operator, API server, and etcd that keep the cluster healthy), and why those two planes fail independently. The workshop covers how to state a hypothesis before injecting a fault, how to bound blast radius, and how to measure failover time, data loss, and recovery rather than assume them.
What will participants work on?
Participants will work with a three-instance CloudNativePG cluster running on a local Kind cluster, with synchronous replication, Prometheus and Grafana for monitoring, Chaos Mesh for fault injection, and Jepsen for correctness checks.
Through this setup, participants will examine how the cluster behaves under failure, including:
- Failover: when the primary disappears, how quickly does a replica take over, and how long are writes unavailable?
- Replica loss: when replicas drop out, do reads keep working, and do replicas catch up cleanly when they return?
- Durability trade-offs: when synchronous replication can’t be satisfied, does the cluster stop accepting writes or quietly accept more risk?
- Control plane loss: when the operator or Kubernetes API is unavailable, what happens if something else fails at the same time?
- Correctness: after the chaos, was any acknowledged write actually lost?
Participants will be introduced to the lab setup, the Chaos Mesh fault model, and a sequence of experiments that show which resilience guarantees operator-managed databases provide.
Target audience
- Engineers running stateful workloads like databases, message brokers, object stores on Kubernetes who want to verify resilience claims rather than trust them.
- SREs and platform engineers who own failover, HA, and disaster recovery for operator-managed data services and are accountable for RTO and RPO numbers.
- Anyone who has read “the operator handles failover automatically” and wants to find out what that actually means under a network partition, a disk returning IO errors, or a node that vanishes mid-transaction.
- Familiarity with Kubernetes primitives (Pods, StatefulSets, PVCs, Services) and basic PostgreSQL concepts is assumed. Prior chaos engineering experience is not needed.
Learning outcomes
By the end of the workshop, participants will be able to:
- Define a steady-state hypothesis for a stateful system that covers durability and consistency, not just availability, and state it before injecting anything.
- Install and operate Chaos Mesh, and express faults as version-controlled CRDs with selectors and modes that bound blast radius precisely.
- Inject pod, network, IO, stress, and time faults against a replicated PostgreSQL cluster and read the resulting failover behaviour from operator logs, Kubernetes events, and metrics.
- Measure failover time, replication lag recovery, and the write-loss window rather than estimating them.
- Detect correctness violations: lost updates, stale reads using Jepsen.
- Compose multi-fault scenarios with Chaos Mesh Workflows, define abort conditions, and promote a validated experiment into CI as a resilience regression gate.
Workshop contents: hands-on experiments
We’ll run these experiments, where each one starts with a stated hypothesis and ends with measured results.
1. Operator pod partition
We isolate the CNPG operator from the cluster. PostgreSQL should keep serving normally, since the data plane doesn’t depend on the operator. The catch: reconciliation and failover are unavailable for the duration, so losing an instance during that window could leave the cluster unrecoverable. Once the partition heals, the operator should reconcile.
We measure: availability throughout, time to reconcile, and any drift.
2. Primary partition
We isolate the primary from the replicas, the operator, and the control plane. The operator should promote a healthy replica, and the -rw service should redirect traffic to it.
We measure: failover duration, the write-unavailability window, and whether the old primary diverged before rejoining.
3. Replica partition, MinSyncReplica satisfiable
We isolate a single replica. The cluster should keep operating normally, with the -ro service routing reads only to healthy replicas. When the replica rejoins, it should catch up from the primary.
We measure: -ro endpoint membership, catch-up lag, and WAL retained on the primary for the absent replica.
4. Replica partition, MinSyncReplica not satisfiable
We isolate enough replicas to break the sync requirement. Here CNPG favours continuity over strict synchrony: it lowers the effective sync count so reads and writes can continue.
We measure: whether writes stall or proceed, and what durability guarantee was traded away.
5. Control plane outage, staged
We take down the API server and confirm the cluster keeps serving normally. Then, still inside the outage, we terminate the primary. The instance can’t reconcile until the API server returns, while -ro keeps serving reads.
We measure: availability during the outage, reconcile time after restore, and WAL growth proportional to restarts.
Tooling & prerequisites
Local tooling
- Docker Desktop
- Visual Studio Code
- Linux or macOS preffered
Provisioned in the environment
- kind, kubectl, helm, jq, curl
- kubectl cnpg plugin (via krew)
- Chaos Mesh
- Prometheus Operator and Grafana Operator, via the CNPG Playground monitoring stack
- Jepsen test harness image
Reference material
- Chaos Mesh documentation — https://chaos-mesh.org/docs/
- CloudNativePG chaos testing (reference implementation, Litmus-based) — https://github.com/cloudnative-pg/chaos-testing
- CNPG Playground — https://github.com/cloudnative-pg/cnpg-playground
- CloudNativePG documentation — https://cloudnative-pg.io/documentation/
- Jepsen — https://jepsen.io/
- PostgreSQL high availability — https://www.postgresql.org/docs/current/high-availability.html
About the instructors
Srinivas Anant is senior Member of Technical Staff at Nutanix.
Rutwij Nerkar and Vishal Sharma are Members of Technical Staff 2 at Nutanix.
How to attend this workshop
This workshop is open to Rootconf members: https://hasgeek.com/rootconf#memberships and ticket holders of Rootconf 2026 Annual Conference: https://hasgeek.com/rootconf/2026/#tickets
The workshop is open to 30 in-person participants, with hybrid access available for remote attendees. Seats for in-person participants will be available on a first-come, first-served basis.
Contact information
For inquiries about the workshop, contact +91-7676332020 or write to info@hasgeek.com.