This Chaos engineering workshop comes straight out of our talk at the July Platform Engineering meetup, Keep Calm and Fail Over, where we shared how we battle-tested CloudNativePG with Chaos Mesh before trusting it with 500+ applications across 30+ clusters. Thirty minutes was enough to show the results, but not how we performed the experiments. The most common question afterwards was how to run these experiments on your own clusters. This workshop answers that.

Teams now run databases on Kubernetes the same way they run everything else, and trust the operator to handle failover and self-healing. But a database pod isn’t interchangeable like a stateless one. It holds a role, a replication position, and data that has to survive the failure. The trouble only shows up when a second failure lands while nothing is around to fix it.

We will be demonstrating different scenarios with a three-instance CloudNativePG cluster and Chaos Mesh. Participants partition the primary, isolate replicas until synchronous replication can no longer be satisfied, cut off the operator, and take down the API server mid-failure. Each experiment starts with a prediction and ends with a measurement.

Breaking a database yourself, on a cluster you can safely destroy, teaches you more than any runbook. You see the real signals in failover, replication lag, WAL growth, and reconciliation, and those are the signals you’ll be watching for when it happens in production.

If you’re a platform engineer, SRE, DevOps engineer, or DBA running stateful workloads on Kubernetes, this workshop is for you. You just need to care about what your database actually does when things go wrong, not what the README says it does.

If you missed the talk? Catch it here: https://hasgeek.com/rootconf/platform-engineering-meet-up-july-25/sub/keep-calm-and-fail-over-engineering-chaos-on-cloud-RovJKYikPrEm8dQd2VVQjZ

If that sounds like your kind of afternoon, come join us!

Date and time: Saturday, 28 November | 2 PM - 6 PM
Register here: https://hasgeek.com/rootconf/chaos-engineering-workshop/

Chaos Engineering for stateful applications on kubernetes

{{ gettext('Draft') }}