JSFoo 2019

On component architecture, front-end engineering and Developer Experience (DX)

Tickets

Building painless scheduling systems in Node

Submitted by Deepak Pathania (@deepakpathania789) on Wednesday, 21 August 2019

Section: Crisp talk (20 mins) Technical level: Intermediate

View proposal in schedule

Abstract

Scheduling stuff sounds like a pretty straight forward thing to do, doesn’t it? Simply set up a cron with a schedule (after looking at the cron syntax online, of course), tell it what to do and forget about it.

Well, if you have ever worked with scheduling systems, you know that isn’t the end of the story. Monitoring, failure isloation, retries, choking preventions etc. make sure you think about that 5 AM job in your sleep.

Idempotency can help - slightly. This talk tries to encapsulate some of my learnings building a distributed scheduler in Node, some of the pain points I faced and how embracing idempotency helped relieve some of those pains.

Outline

  1. Introduction
  2. Scheduling tasks
  3. Weapon of choice - Node
  4. Code sample for a basic schedule.
  5. Pains
    • Failure isolation
    • Retry mechanisms for individual entries
    • Monitoring
    • Testing
  6. Idempotency
    • Introduction
    • Importance in scheduling
  7. Embracing failures and implicit retries
  8. Slack webhooks - monitoring 101
  9. Stubbing your way around testing
  10. Conclusion

Speaker bio

Senior Software Engineer @INDwealth. Full stack web developer with a keen eye for building elegant interfaces and a proven record of writing scalable backend code. Loves everything javascript. Speaks at conferences, meetups, hackathons and generally all the time. Likes open source, startups, pets, anime and people - in no particular order.

Links

Slides

https://speakerdeck.com/deepakpathania/building-painless-scheduling-systems-in-node

Comments

  • Zainab Bawa (@zainabbawa) Reviewer 15 days ago

    Thanks for the submission, Deepak. I have one question about the proposal: what are the other approaches to solving cron jobs problem? Did you compare these approaches to the approach you finally chose? How does this approach stand out when compared with others?

  • Deepak Pathania (@deepakpathania789) Proposer 14 days ago

    Hey Zainab, great question. There are actually a lot of managed services out there which manage some of the problems out of the box, like Cloud Scheduler by Google, lambda functions for on demand handling by Amazon etc. It was a classic build vs buy problem and when we evaluated these options at the beginning of the project, we realized that since ours was an experimental project, we would not like to spend a lot of time and resources in setting these up.

    We wanted to build something quickly with native crons, see if the project makes sense and then move to a managed solution later when we hit a certain scale. We also realised that the opportunity cost of spending on managed solutions from the very beginning was too much for us. Also, native crons helped us build solutions that could interact with our database heavily, instead of sending too much data over the wire for let’s say, a serverless handler, this allowed us to iterate quickly.

    Keeping these points in mind, we felt this was the right way to go for the project at hand.

    • Zainab Bawa (@zainabbawa) Reviewer 4 days ago

      These insights should be articulated in the presentation.

Login with Twitter or Google to leave a comment