IN/Clojure 2020

This talk explores a job scheduler for Clojure (now extracted into a library), which, in addition to being capable of persisting state, is resilient to the variability of a distributed system.

In this talk, we will look at the APIs that make up a core part of this library. As we discuss the internals, we’ll reflect on the ease of interop with Java’s Executor APIs. We’ll then discuss the architecture of the scheduler, the rationale for persistence, the database schema, the modeling of jobs and tasks, and the fringe benefits of persisting contextual information for each job.

To lace it all together, we will define the inputs and dissect what it translates to: initiating the schedulers, creation of jobs, discovering pending jobs on the database, and executing jobs. Towards the end, we will explore the library’s extensibility and the scope of its utilization.

Outline

1. Context:

• What is a general-purpose job scheduler?
• What currently exists, and how does this library compare?

2. Architecture:

• Walkthrough
• Motivations
• Internal APIs

3. Library APIs:

• Defining jobs
• Defining tasks, rules, and metadata for jobs
• Scheduling jobs

4. Rationales:

• Database schema
• Why is it required for job schedulers to be persistent, and what purpose does persisting jobs’ and tasks’ state serve?
• What purpose does metadata serve?

5. Lacing it all together.
6. Roadmap of the library.

Requirements

Minimal experience of working with Clojure and Postgres is a nice-to-have prerequisite, but not a compelling requirement.

Speaker bio

I’m a software engineer at nilenso, where I primarily work on building a platform for incentivising Gojek drivers. Prior to nilenso, I spent a large chunk of my programming career working with Kotlin and Java (among other things) to solve various problems across healthcare technology.