Nov 2024
4 Mon
5 Tue
6 Wed
7 Thu
8 Fri
9 Sat
10 Sun
Nov 2024
11 Mon
12 Tue
13 Wed
14 Thu
15 Fri
16 Sat
17 Sun
Nov 2024
18 Mon
19 Tue
20 Wed
21 Thu
22 Fri 09:00 AM – 03:20 PM IST
23 Sat 09:30 AM – 06:15 PM IST
24 Sun
Background jobs, more commonly referred to as asynchronous tasks or jobs, are a technique in software development for managing tasks that can be executed independently of the primary user interaction or request-response cycle. Background jobs are utilized to enhance system responsiveness, manage time-consuming tasks, and offload resource-intensive operations from the main application thread or process.
In Freshdesk, the bulk of in-app background processing — jobs enqueued and processed by the Rails application — happens using Sidekiq. Sidekiq is a super-fast Redis-based queueing system that uses threads to handle many jobs at the same time and in the same process. For a product as big as Freshdesk, we have numerous background queues for various types of functionalities.
However, such jobs, when executed with high concurrency, can exhaust system resources, such as CPU, memory, or network bandwidth. These operations can also strain downstream systems, including RDS, Redis, Elasticsearch, and others. This can impact the performance of other genuine jobs that share the same resources, resulting in performance degradation or system failures, thereby becoming a “noisy neighbor.” An effective job throttling mechanism can help in easing the load on the resources.
Throttling gives you granular control in such critical situations, especially in a production environment for jobs where it’s not important to finish as quickly as possible, and at the same time you don’t want to pile up your queues with lots of jobs that are heavy - which can potentially impact your resources, leading to an outage.
This talk will focus on the primary motivations, architecture, benefits, and challenges companies should prepare for - if they decide to use a high concurreny Redis-based job framework for their application.
Hosted by
Supported by
Platinum Sponsor
Platinum Sponsor
Venue host - Rootconf workshops
Community Partner
Community Partner
{{ gettext('Login to leave a comment') }}
{{ gettext('Post a comment…') }}{{ errorMsg }}
{{ gettext('No comments posted yet') }}