BEGIN:VCALENDAR VERSION:2.0 PRODID:-//HasGeek//NONSGML Funnel//EN DESCRIPTION:A hands-on workshop on testing failover\, recovery\, and data durability in PostgreSQL on Kubernetes X-WR-CALDESC:A hands-on workshop on testing failover\, recovery\, and data durability in PostgreSQL on Kubernetes NAME:Chaos Engineering for stateful applications on kubernetes X-WR-CALNAME:Chaos Engineering for stateful applications on kubernetes REFRESH-INTERVAL;VALUE=DURATION:PT12H SUMMARY:Chaos Engineering for stateful applications on kubernetes TIMEZONE-ID:Asia/Kolkata X-PUBLISHED-TTL:PT12H X-WR-TIMEZONE:Asia/Kolkata BEGIN:VEVENT SUMMARY:Chaos Engineering for stateful applications on kubernetes DTSTART:20261128T083000Z DTEND:20261129T123000Z DTSTAMP:20261019T231346Z UID:session/63mKfimvCmcegQciG7iA75@hasgeek.com SEQUENCE:17 CREATED:20261002T025630Z DESCRIPTION:# Chaos Engineering for stateful applications on kubernetes\nA hands-on workshop for platform\, SRE\, DevOps\, and database engineers wh o run stateful workloads on Kubernetes and need to know how they behave wh en things fail.\n\nRunning databases on Kubernetes is no longer an experim ent. Operators like CloudNativePG promise automated failover\, self-healin g\, and reconciliation\, and teams increasingly rely on those promises in production. But a database pod is not like a stateless one. It holds a rol e\, a replication position\, and a volume that must be reattached before i t can serve. When it fails\, the risks are split-brain\, stale reads from a lagging replica\, lost writes after an ungraceful promotion\, and recove ry that silently stalls.\n\nThis workshop focuses on testing those promise s deliberately\, before an incident.\n\n## What is this workshop about?\nT his workshop will help engineers understand what actually happens to a rep licated database on Kubernetes when parts of it fail.\n\nParticipants will learn how failures move through the data plane (the PostgreSQL instances serving traffic) and the control plane (the operator\, API server\, and et cd that keep the cluster healthy)\, and why those two planes fail independ ently. The workshop covers how to state a hypothesis before injecting a fa ult\, how to bound blast radius\, and how to measure failover time\, data loss\, and recovery rather than assume them.\n\n## What will participants work on?\nParticipants will work with a three-instance CloudNativePG clust er running on a local Kind cluster\, with synchronous replication\, Promet heus and Grafana for monitoring\, Chaos Mesh for fault injection\, and Jep sen for correctness checks.\n\nThrough this setup\, participants will exam ine how the cluster behaves under failure\, including:\n- **Failover:** wh en the primary disappears\, how quickly does a replica take over\, and how long are writes unavailable?\n- **Replica loss:** when replicas drop out\ , do reads keep working\, and do replicas catch up cleanly when they retur n?\n- **Durability trade-offs:** when synchronous replication can't be sat isfied\, does the cluster stop accepting writes or quietly accept more ris k?\n- **Control plane loss:** when the operator or Kubernetes API is unava ilable\, what happens if something else fails at the same time?\n- **Corre ctness:** after the chaos\, was any acknowledged write actually lost?\n\nP articipants will be introduced to the lab setup\, the Chaos Mesh fault mod el\, and a sequence of experiments that show which resilience guarantees o perator-managed databases provide.\n\n## Target audience\n* Engineers runn ing stateful workloads like databases\, message brokers\, object stores on Kubernetes who want to verify resilience claims rather than trust them. \n* SREs and platform engineers who own failover\, HA\, and disaster recov ery for operator-managed data services and are accountable for RTO and RPO numbers. \n* Anyone who has read "the operator handles failover automati cally" and wants to find out what that actually means under a network part ition\, a disk returning IO errors\, or a node that vanishes mid-transacti on. \n* Familiarity with Kubernetes primitives (Pods\, StatefulSets\, PVC s\, Services) and basic PostgreSQL concepts is assumed. Prior chaos engine ering experience is not needed.\n\n## Learning outcomes\nBy the end of the workshop\, participants will be able to:\n* Define a steady-state hypothe sis for a stateful system that covers durability and consistency\, not jus t availability\, and state it before injecting anything. \n* Install and operate Chaos Mesh\, and express faults as version-controlled CRDs with se lectors and modes that bound blast radius precisely. \n* Inject pod\, net work\, IO\, stress\, and time faults against a replicated PostgreSQL clust er and read the resulting failover behaviour from operator logs\, Kubernet es events\, and metrics. \n* Measure failover time\, replication lag reco very\, and the write-loss window rather than estimating them. \n* Detect correctness violations: lost updates\, stale reads using Jepsen. \n* Comp ose multi-fault scenarios with Chaos Mesh Workflows\, define abort conditi ons\, and promote a validated experiment into CI as a resilience regressio n gate.\n\n## Workshop contents: hands-on experiments\nWe'll run these exp eriments\, where each one starts with a stated hypothesis and ends with me asured results.\n\n#### 1. Operator pod partition\nWe isolate the CNPG ope rator from the cluster. PostgreSQL should keep serving normally\, since th e data plane doesn't depend on the operator. The catch: reconciliation and failover are unavailable for the duration\, so losing an instance during that window could leave the cluster unrecoverable. Once the partition heal s\, the operator should reconcile.\n\n**We measure:** availability through out\, time to reconcile\, and any drift.\n\n#### 2. Primary partition\nWe isolate the primary from the replicas\, the operator\, and the control pla ne. The operator should promote a healthy replica\, and the `-rw` service should redirect traffic to it.\n\n**We measure:** failover duration\, the write-unavailability window\, and whether the old primary diverged before rejoining.\n\n#### 3. Replica partition\, MinSyncReplica satisfiable\nWe i solate a single replica. The cluster should keep operating normally\, with the `-ro` service routing reads only to healthy replicas. When the replic a rejoins\, it should catch up from the primary.\n\n**We measure:** `-ro` endpoint membership\, catch-up lag\, and WAL retained on the primary for t he absent replica.\n\n#### 4. Replica partition\, MinSyncReplica not satis fiable\nWe isolate enough replicas to break the sync requirement. Here CNP G favours continuity over strict synchrony: it lowers the effective sync c ount so reads and writes can continue.\n\n**We measure:** whether writes s tall or proceed\, and what durability guarantee was traded away.\n\n#### 5 . Control plane outage\, staged\nWe take down the API server and confirm t he cluster keeps serving normally. Then\, still inside the outage\, we ter minate the primary. The instance can't reconcile until the API server retu rns\, while `-ro` keeps serving reads.\n\n**We measure:** availability dur ing the outage\, reconcile time after restore\, and WAL growth proportiona l to restarts.\n\n## Tooling & prerequisites\n### Local tooling\n* Docker Desktop \n* Visual Studio Code\n* Linux or macOS preffered\n\n### Provisi oned in the environment\n* kind\, kubectl\, helm\, jq\, curl \n* kubectl cnpg plugin (via krew) \n* Chaos Mesh\n* Prometheus Operator and Grafana Operator\, via the CNPG Playground monitoring stack \n* Jepsen test harne ss image\n\n## Reference material\n* Chaos Mesh documentation — [https:/ /chaos-mesh.org/docs/](https://chaos-mesh.org/docs/) \n* CloudNativePG ch aos testing (reference implementation\, Litmus-based) — [https://github. com/cloudnative-pg/chaos-testing](https://github.com/cloudnative-pg/chaos- testing) \n* CNPG Playground — [https://github.com/cloudnative-pg/cnpg- playground](https://github.com/cloudnative-pg/cnpg-playground) \n* CloudN ativePG documentation — [https://cloudnative-pg.io/documentation/](https ://cloudnative-pg.io/documentation/) \n* Jepsen — [https://jepsen.io/]( https://jepsen.io/) \n* PostgreSQL high availability — [https://www.pos tgresql.org/docs/current/high-availability.html](https://www.postgresql.or g/docs/current/high-availability.html)\n\n## About the instructors\nSriniv as Anant is senior Member of Technical Staff at Nutanix. \n\nRutwij Nerka r and Vishal Sharma are Members of Technical Staff 2 at Nutanix. \n\n## Ho w to attend this workshop\nThis workshop is open to Rootconf members: http s://hasgeek.com/rootconf#memberships and ticket holders of Rootconf 2026 A nnual Conference: https://hasgeek.com/rootconf/2026/#tickets\n\nThe worksh op is open to 30 in-person participants\, with hybrid access available for remote attendees. Seats for in-person participants will be available on a first-come\, first-served basis.\n\n## Contact information\nFor inquiries about the workshop\, contact +91-7676332020 or write to info@hasgeek.com. LAST-MODIFIED:20261009T041108Z LOCATION:Bangalore and hybrid - https://hasgeek.com/rootconf/chaos-enginee ring-workshop/ ORGANIZER;CN=Rootconf:MAILTO:no-reply@hasgeek.com URL:https://hasgeek.com/rootconf/chaos-engineering-workshop/ BEGIN:VALARM ACTION:display DESCRIPTION:Chaos Engineering for stateful applications on kubernetes in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT END:VCALENDAR