Rootconf Mini 2024

Geeking out on systems and security since 2012

Tickets

Loading…

Ajay Ravichandran

Ajay Ravichandran

@ajay1105

Sobering Noisy Background Jobs at Scale

Submitted Oct 10, 2024

Background jobs, more commonly referred to as asynchronous tasks or jobs, are a technique in software development for managing tasks that can be executed independently of the primary user interaction or request-response cycle. Background jobs are utilized to enhance system responsiveness, manage time-consuming tasks, and offload resource-intensive operations from the main application thread or process.

In Freshdesk, the bulk of in-app background processing — jobs enqueued and processed by the Rails application — happens using Sidekiq. Sidekiq is a super-fast Redis-based queueing system that uses threads to handle many jobs at the same time and in the same process. For a product as big as Freshdesk, we have numerous background queues for various types of functionalities.

However, such jobs, when executed with high concurrency, can exhaust system resources, such as CPU, memory, or network bandwidth. These operations can also strain downstream systems, including RDS, Redis, Elasticsearch, and others. This can impact the performance of other genuine jobs that share the same resources, resulting in performance degradation or system failures, thereby becoming a “noisy neighbor.” An effective job throttling mechanism can help in easing the load on the resources.

Throttling gives you granular control in such critical situations, especially in a production environment for jobs where it’s not important to finish as quickly as possible, and at the same time you don’t want to pile up your queues with lots of jobs that are heavy - which can potentially impact your resources, leading to an outage.

This talk will focus on the primary motivations, architecture, benefits, and challenges companies should prepare for - if they decide to use a high concurreny Redis-based job framework for their application.

Key Takeaways

  • Get insights into the many of the unexpected challenges a redis based queuing system poses at scale.
  • Explore how a highly configurable throttling system helps ease the load on system resources and mitigates potential production outages.
  • Empower the audience to make informed decisions when designing queuing systems for their applications.

Target Audience

  • SRE, Systems, and DevOps Engineers.
  • Software Developers and Architects.
  • System Design Enthusiasts.
  • Technically Curious Individuals.
    `

Comments

{{ gettext('Login to leave a comment') }}

{{ gettext('Post a comment…') }}
{{ gettext('New comment') }}
{{ formTitle }}

{{ errorMsg }}

{{ gettext('No comments posted yet') }}

Hybrid Access Ticket

Hosted by

We care about site reliability, cloud costs, security and data privacy

Supported by

Platinum Sponsor

Nutanix is a global leader in cloud software, offering organizations a single platform for running apps and data across clouds.

Platinum Sponsor

PhonePe was founded in December 2015 and has emerged as India’s largest payments app, enabling digital inclusion for consumers and merchants alike.

Silver Sponsor

The next-gen analytics engine for heavy workloads.

Venue host - Rootconf workshops

Community Partner

FOSS United is a non-profit foundation that aims at promoting and strengthening the Free and Open Source Software (FOSS) ecosystem in India. more

Community Partner

A community of rust-lang contributors and end-users from Bangalore. We also have presence on the following telegram channels https://t.me/RustIndia https://t.me/keralars https://t.me/fpncr Twitter handle: https://twitter.com/rustlangin more