Audience take-aways - 5 minutes (to set expectations)

More problem-solving bandwidth for the developer

Caveats: May not appeal if you like to think in types

Less fighting with frameworks

Powerful but small set of abstractions - they apply in the same manner across use-cases

Learn once, apply everywhere

• Core language is designed to make the developer productive across two major host platforms - the JVM and JS
• Code/Data duality allows for an interesting ecosystem - configuration, build logic, or business logic, all can be in Clojure
• Shared logic across be/fe systems is a superpower

Clojure strengths - 5 minutes (to prepare for what is coming)

Data oriented

Focus on modeling the domain well through a few, well-defined and mature data-types and data-structures.

Functional, low cognitive overload

Few core language types that form a strong basis for modeling all kinds of business domain, but are agnostic of them. The core language contains all functions that you need to deal with these types, leaving the application developers with mostly the task of writing core business logic.
“It is better to have 100 functions operate on one data structure than to have 10 functions operate on 10 data structures.” – Alan Perlis

Single language (data + code) is a superpower

Front-end, back-end, build scripts, configuration, all expressed in a single language

Problems solved - stories

[Rewrite Story] Reliable, large-scale crawling platform

The problem

• Business critical data fetching system that required to fetch a billion+ data-points every single day
• Existing system took days to run (generally 3, but sometimes more).
  • Complex, hard to monitor
  • It would sometimes fail to finish, but we’d only know after 3 days (or, when someone would keep checking logs regularly)
  • Since it ran over multiple days, there would be multiple instances making it harder to keep track of status.
  • Operationally a very cumbersome system. Sometimes the system would stall after multiple days and data had to be salvaged because we couldn’t afford to lose any data.
• Data was hardly fresh by the time it was loaded into the system for further processing
• Fragile, and not well architected.
  • Requests to fix bugs were met with fear, let alone adding new features.
  • 105K+ lines of Java code (Of course, not Java’s fault)

Replacement system: Clojure

• First stage - the alternative implementation with better features - about 5K LoC.
  • Though it’s not about LoC but more about expressibility of a solution in Clojure
• New features
  • Had support for policies to handle different APIs (FB, YT, Twitter and so on)
    • A new feature - appropriate retry mechanisms - that became possible
  • Prioritization mechanism for different data-points - those that needed more frequent refreshes to those that could do with longer cycles
    • Allowed for some data to refresh within minutes too. Unlike the old system where everything took 5 days+ (long postprocessing cycle after the fetch)
  • Ability to add new crawlers that could focus on the API while the platform took care of scheduling, retrying, token policies.
  • About 2K additional lines, including for a few new crawlers.
• The seam-less transformation between Clojure-types (the map) with JSON made the entire flow quite a breeze
• Key libraries used: core.async, durable-queue

Production support - super-charged

The context - Manufacturing

• An assembly-line manufacturing system created with Springboot
  • Unlike a regular web application, this was a stateful application that tracked material movement, stage of production and quality across multiple assembly lines in a factory
  • Being stateful, no easy way to scale horizontally. System had to be up at all times. Downtimes would be planned.
  • The plants are located at multiple geographical locations
  • The application was to be deployed for the first time, and there was no precedent.
  • Obviously, incremental deliveries was not an option.
    • Made it a challenge to be sure everything was in place.
    • But unknown unknowns not possible to handle.

Clojure + nREPL - A safety net

• We had to prepare for handling the unknown unknowns at runtime
• Embedded nREPL server in Springboot applications
• Utilities to plug into the Springboot system created
• It was a huge help - we handled multiple unanticipated situations through dynamically executed Clojure code in a running system!
• The plain-vanilla approach may have required multiple redeployments with planned, coordinated downtimes
• We covered all unanticipated needs via dynamically executing Clojure, while we noted the changes that needed to be done in the primary Kotlin codebase
• The system held up for 3 weeks, after which we redeployed with updated code for the newly discovered scenarios.