Call for submissions: Platform Engineering meet-ups
Rootconf
Rootconf

Call for submissions: Platform Engineering meet-ups

Share your journey of building platforms that power engineering teams

All submissions

Saravana Priyan

Before trusting at-least-once delivery: SIGKILLing BullMQ workers on Amazon EC2

Submitted Nov 6, 2026

Session type: Demo (5-10 mins) Submission made for: Bangalore meet-ups

BullMQ’s docs promise at-least-once delivery. I wanted to watch that promise survive an actual worker crash, so I built a small failure-injection harness on an EC2 t3.medium: 30 jobs per variant, 5 trials each, SIGKILL the worker with 5 jobs in flight. With no guard, all 5 restarted jobs wrote their side effect twice. A Redis dedupe check placed immediately before the effect stopped every duplicate in those trials - but moving the same check ahead of a gap flipped the outcome: the crash landed in the gap, 0 duplicates, yet 5 jobs completed with their effect never written.

The variant that committed the dedupe check and the effect in one atomic Redis Lua step had neither duplicates nor missing effects in the observed trials, and that only covers side effects inside the same Redis; side effects in other systems need their own transactional or idempotency design. I walk through the measured runs and Grafana timelines and show where the at-least-once boundary actually sits.

Takeaways: at-least-once means every crashed job can run again, and “completed” does not mean “done”. The dedupe record and the effect must commit atomically, or a crash will find the gap between them.

Audience: engineers running BullMQ or other Redis-backed background jobs in Node.js services, and anyone designing idempotent workers on queues.

Bio: I’m Saravana Priyan, a Product Testing Intern at Nokia in Chennai, and a final-year AI and Data Science student. I learn backend engineering by running small failure and recovery experiments on my own EC2 lab and writing them up on AWS Builder Center, including this experiment. This work is a personal project and does not represent my employer.

All submissions

Comments

{{ gettext('Login to leave a comment') }}

{{ gettext('Post a comment…') }}
{{ gettext('New comment') }}
{{ formTitle }}

{{ errorMsg }}

{{ gettext('No comments posted yet') }}

Register

Hosted by

Rootconf
Rootconf
We care about site reliability, cloud costs, security and data privacy