FP Web Conf

Conventional background processing libraries have failed to keep up with the demands of distributed systems in 2023. This talk will explore the key features that modern web developers require from an asynchronous system: Reliability, Scalability, Pluggability, and Extensibility.

Furthermore, we’ll talk about the journey of building Goose, a versatile background processing library for Clojure designed with next-level features like pluggable message brokers. Goose serves as specimen for open-source libraries, as it exemplifies community-collaboration, thorough documentation, and built-in reliability.

This talk is divided into three sections:

1. What is lacking in the background processing ecosystem?
2. Internals of Goose
3. My perspective on how a modern open-source library should be built

Reference links:

• Github: https://github.com/nilenso/goose
• Wiki: https://github.com/nilenso/goose/wiki
• Cljdocs: https://cljdoc.org/d/com.nilenso/goose/
• Slack: https://clojurians.slack.com/channels/goose

What is lacking in the background processing ecosystem?

1. Reliability is not a built-in feature.

• Free versions of Sidekiq and Celery do not guarantee against loss of Jobs.
• If developers need reliability, they need to pay for the enterprise version of Sidekiq.
• This deters developers from deploying most open-source libraries for critical asynchronous tasks.

2. The choice of message-broker is restricted to a library’s implementation.

• Users have to choose from either Redis, RabbitMQ, AWS SQS or PostgreSQL.
• Different message brokers are better suited for different use cases, depending on factors like scale, performance, and reliability.
• If users want to plugin Kafka, MongoDB, XTDB or Datomic, that is not possible with any background-processing library apart from Goose.

3. For the reasons above, organisations need to create internal tooling for asynchronous processing.

• My intention for building Goose was to make background processing one less thing to worry about for the Clojure community.

Internals of Goose

Goose was designed with reliability, scalability, pluggability, and extensibility in mind. I will delve into seven next-level features that set it apart:

1. Reliability

• Reliability is a fundamental feature, not a paid extra.
• I will walk through reliability concerns like network errors, failed enqueues, worker crashes, and broker crashes.
• These concerns are addressed through Goose’s unique queuing design, heartbeats and orphan-checkers, publisher/consumer acknowledgments, and HA architecture.

2. Scalability

• Goose allows users to scale their applications by leveraging Java threads and concurrent broker connections.
• Before execution, a lock is acquired on all jobs to avoid duplication.

3. Message-broker agnostic design

• Goose has a Broker Protocol to abstract enqueuing, scheduling, and dequeuing jobs.
• Redis and RabbitMQ message-brokers are supported out of the box.
• If developers require a different broker, they can plug it into Goose by virtue of this protocol.

4. Pluggable metrics and logging

• Different organisations prefer different metrics protocols, such as StatsD, Prometheus, or dropwizard/metrics.
• Goose provides a StatsD implementation out of the box and allows users to plugin a custom metric backend by virtue of a Metrics protocol.
• Like metrics, organizations also prefer different logging formats, such as SLF4J, Log4J, or Apache Commons Logging.
• Goose allows developers to plugin their logging system by virtue of clojure/tools.logging.

5. Extensibility

• Middlewares allow users to extend Goose’s functionality.
• For instance, users can build DAGs and data pipelines based on arguments and results before and after job execution.

6. Job enqueue interface

• I have gone to great lengths to ensure that enqueuing a function has the same UX as executing one.
• I will be talking about the 3 iterations and community-feedback it took to arrive at the current API of Goose, which ensures asynchronous invocation does not have any drawbacks compared to synchronous invocation of functions.
• Goose’s enqueue API takes variadic arguments to allow for flexibility in the number of arguments that a function can accept.

7. Runtime config over init-time config

• Complex systems require config that can be adjusted at runtime, rather than just at initialization.
• Goose allows users to inject config during runtime, enabling them to distribute load by enqueuing to different brokers.

My perspective on how a modern open-source library should be built

1. The primary focus of any library should be on correctness.

• Unit, integration, and performance tests help with this.

2. A library should have extensive documentation.

• A README, Wiki, Docstrings, and Specs help developers navigate seamlessly through your library’s features.
• Default settings with an option for customization go a long way in helping power users gain a quick start.

3. Community collaboration should be encouraged to gain developers’ trust in trying a new library.

• Over the last year, Goose has had three major and two patch releases.
• After and before every release, users of Goose were asked for feedback, feature requests, and blockers.

4. Language-agnostic design.

• It should be possible to serialize function invocations (like data) and send them over the wire.
• This idea is still in its infancy and would require support from multiple programming languages.

Summary

Through this talk, I wish to educate the audience on gaps in conventional libraries, give insights into possible solutions, talk about my attempt at building a user-friendly library, and explore what the future holds for this field.