Bengaluru Systems Meetup #4

Hot Partitions & Load Variance in Distributed Systems

Duration: 45 minutes

Covering stories about what leads to load variance in various distributed systems, the behavior it shows when it happens, and finally how to avoid and handle production incidents.

Simple Cases

• CPU Hot Partition Stories
• Memory Hot Partition Stories
• Disk Hot Partition Stories
• Network Hot Partition Stories

Complicated Cases

• Load Variance in Persistent Connection Systems

  • WebSocket Connections
  • gRPC Connections
  • HTTP Persistent Connections
  • MQTT Persistent Connections
  • Control Plane / Data Plane (Istio / Kong, etc.)
  • Multi-Proxy Layers
  • Load Variance in Network Load Balancers

• Load Variance in Application Workloads

  • Kafka Consumer Groups
  • In-App Pub/Sub
  • In-App Cron Job Libraries
    • Celery
    • Sidekiq
    • Gocron
  • Load Variance Due to Noisy Neighbors

• Hot Writer in Writer/Reader Replication Systems

• Hot Leader in Consensus-Based Distributed Systems

• Daily Cron

• Load Distribution Efficiency

Takeaways

• Techniques & Solutions to Avoid and Handle Load Variance & Hot Partitions in Systems

References

• TODO: I will add the slides 4-5 days in advance.