Rootconf Mini 2024

Geeking out on systems and security since 2012

Tickets

Loading…

Raj Suvariya

@rajsuvariya

Maintenance operator for TiDB running on Kubernetes

Submitted Oct 30, 2024

Description:

TiDB is distributed SQL and horizontally scalable datastore developed by PingCAP. TiDB makes it easy to deploy and run database clusters on kubernetes by providing a official TiDB operator. However, this operator generally assumes that the database is running on network attached disk, which makes it easier for operator to not worry about compute failures - as compute can be rescheduled to any other compute server and disk failutes - as majority of NAS comes with redundancy at disk level.

However, working at Flipkart we realised that most of our workloads were running on local attached disks, which makes it difficult to perform any planned downtime / destructive maintenances at hardware leyar, considering two major problems, (a) compute cannot be rescheduled as it’s locally attached with disk (b) disk doesn’t have redundancy. To overcome this problem we developed another in-house operator which does end to end handling of proactive move out before the scheduled maintenance.

Talk will be covering lot of content present on this blog post written by me. https://blog.flipkart.tech/automating-tidb-planned-maintenance-with-k8s-operator-5893f3ffb657

Key takeaways

  • Running databases on kubernetes and it’s challenges wrt to different kind of hardwares.
  • Development of kubernetes operator which works side by side with another operator, both operating on same resource.

Target Audience

  • System engineers working on developing databases on kubernetes

Comments

{{ gettext('Login to leave a comment') }}

{{ gettext('Post a comment…') }}
{{ gettext('New comment') }}
{{ formTitle }}

{{ errorMsg }}

{{ gettext('No comments posted yet') }}

Hybrid Access Ticket

Hosted by

We care about site reliability, cloud costs, security and data privacy