JSFoo 2019

On component architecture, front-end engineering and Developer Experience (DX)

Tickets

Building painless scheduling systems in Node

Submitted by Deepak Pathania (@deepakpathania789) on Wednesday, 21 August 2019

Section: Crisp talk (20 mins) Technical level: Intermediate

View proposal in schedule

Abstract

Scheduling stuff sounds like a pretty straight forward thing to do, doesn’t it? Simply set up a cron with a schedule (after looking at the cron syntax online, of course), tell it what to do and forget about it.

Well, if you have ever worked with scheduling systems, you know that isn’t the end of the story. Monitoring, failure isloation, retries, choking preventions etc. make sure you think about that 5 AM job in your sleep.

Idempotency can help - slightly. This talk tries to encapsulate some of my learnings building a distributed scheduler in Node, some of the pain points I faced and how embracing idempotency helped relieve some of those pains.

Outline

  1. Introduction
  2. Scheduling tasks
  3. Weapon of choice - Node
  4. Code sample for a basic schedule.
  5. Pains
    • Failure isolation
    • Retry mechanisms for individual entries
    • Monitoring
    • Testing
  6. Idempotency
    • Introduction
    • Importance in scheduling
  7. Embracing failures and implicit retries
  8. Slack webhooks - monitoring 101
  9. Stubbing your way around testing
  10. Conclusion

Speaker bio

Senior Software Engineer @INDwealth. Full stack web developer with a keen eye for building elegant interfaces and a proven record of writing scalable backend code. Loves everything javascript. Speaks at conferences, meetups, hackathons and generally all the time. Likes open source, startups, pets, anime and people - in no particular order.

Links

Slides

https://speakerdeck.com/deepakpathania/building-painless-scheduling-systems-in-node

Comments

  • Zainab Bawa (@zainabbawa) Reviewer 2 months ago

    Thanks for the submission, Deepak. I have one question about the proposal: what are the other approaches to solving cron jobs problem? Did you compare these approaches to the approach you finally chose? How does this approach stand out when compared with others?

  • Deepak Pathania (@deepakpathania789) Proposer 2 months ago

    Hey Zainab, great question. There are actually a lot of managed services out there which manage some of the problems out of the box, like Cloud Scheduler by Google, lambda functions for on demand handling by Amazon etc. It was a classic build vs buy problem and when we evaluated these options at the beginning of the project, we realized that since ours was an experimental project, we would not like to spend a lot of time and resources in setting these up.

    We wanted to build something quickly with native crons, see if the project makes sense and then move to a managed solution later when we hit a certain scale. We also realised that the opportunity cost of spending on managed solutions from the very beginning was too much for us. Also, native crons helped us build solutions that could interact with our database heavily, instead of sending too much data over the wire for let’s say, a serverless handler, this allowed us to iterate quickly.

    Keeping these points in mind, we felt this was the right way to go for the project at hand.

    • Zainab Bawa (@zainabbawa) Reviewer a month ago

      These insights should be articulated in the presentation.

  • Arindam Paul (@arpaul) a month ago

    Hey Deepak,

    Great content, couple of things which might help enhance your talk even more, * Explanation of the cron schedule expression in 1 or 2 slides as a refresher would help. (Ref: https://blog.abelotech.com/posts/nodejs-scheduler-cron-timeout-interval/) * In Node.js callbacks are never guranteed in terms of time on which they execute (that’s why they are callbacks :)), so even if the job is scheduled in a callback queue, if the event loop is busy processing some CPU bound task, it will block all callbacks and the time of run won’t be guranteed. This in general is good callout while talking about scheduling in Node.js * It might be a good idea to introduce webworkers and push down the cron jobs to the web workers so that the main thread can be always available to handle interrupts, or other admin signals/messages. * Lastly, once you create a cron job you can cancel/delete/deactivate it and resume it later as well, if you can demo a simple UI based cron scheduler which uses npm cron to manage cron jobs and how a you can read a simple JSON config (may be the output from the UI) to dynamically load all cronjobs, that would really help people see the practical usages of task scheduling.

  • Deepak Pathania (@deepakpathania789) Proposer a month ago

    Hey Arindam,

    Thanks for the comments, just wanted to address some of them here in case other reviewers happen to see this.

    1. I explain the cron schedule expression verbally, would see if time permits me to add a slide for it based on the rehearsal today.
    2. That’s a great point, I actually planned to touch on this in the problem statement slide, where I mention that eventual consistency is acceptable for our use case. I’ll see if I can revisit this in the flow section.
    3. I intentionally avoided talking about web workers since I wanted to talk about the v1 of the product we built and treat this like a case study. I’ll see if I need to add a section for further improvements at the end, which could cover this.
    4. In our case, the user comes and creates tasks via an interface and specifies a schedule along with it. They are not really setting up cron jobs, it is the responsibility of the system to batch those tasks and handle them via system crons. But I like the idea of talking about cancelling and deactivating because we did build that. I’ll try and add this.

    Thanks again for taking out the time to review this.

  • Arindam Paul (@arpaul) a month ago

    Sounds Great! All the best and share the link of your talk with me @geek_paul

  • Zainab Bawa (@zainabbawa) Reviewer a month ago

    Thanks for an excellent rehearsal, Deepak. Here is the consolidated feedback:

    1. Explain eventual consistency carefully. In the current flow, the mention is abrupt.
    2. Many people don’t recognize the complexity of scheduling. For example, why do we need a tool like Buffer? Mention early in your talk why scheduling is hard and what is the problem you are trying to solve.
    3. Many people don’t understand star notation. Give one or two examples.
    4. A lot of the monitors available out-of-the-box solve limited problems. But your talk shows that out-of-the-box solutions can be efficient but they are not silver bullet. Mention this.
    5. When you spoke about time zone in cron, this is one of the biggest gotchas that you have to talk a lot about. Go into detail rather than putting this in a mention.
    6. The luxury of eventual consistency is not always there. When this luxury is not there, what do you do?

    On the visual aspects, the following changes have to be incorporated:

    1. Some slides have lots of text. Reduce text on the slides. Break some slides into two slides.
    2. Increase font slides for slides with code samples.

Login with Twitter or Google to leave a comment