Building data platforms from business stores using Clojure

Feb 2020

10 Mon

11 Tue

12 Wed

13 Thu

14 Fri 09:30 AM – 09:00 PM IST

15 Sat 08:45 AM – 06:00 PM IST

16 Sun

Make a submission

Accepting submissions till 20 Jan 2020, 11:59 PM

Novotel Pune Nagar Road, Crimson Hall, Pune

Tickets

IN/Clojure 2020 is the 4th edition of India’s annual Clojure and ClojureScript conference. Follow twitter.com/in_clojure for updates.

Backstory

We began in Pune in 2016 and after two editions in namma Bengaluru, we’re delighted to come full circle back to apla Pune (and all those delicious bakarwadis and modaks).

As India’s premiere Clojure and ClojureScript conference, we strive to facilitate the free exchange of ideas between new and experienced programmers alike.

Over the years, IN/Clojure has served as a platform to kickstart a series of Clojure workshops and meet-ups across the country. And it has helped attendees make a strong case for Clojure, and Clojurescript adoption in their companies.

We believe that this edition of the conference, like previous editions will foster the growing Clojure community in Asia.

Who should attend?

Whether you are knee-deep writing macros that write macros, or struggling to escape the beginner’s plateau, or wondering if/why/how to adopt Clojure in your organization, or are in it purely for the joy of learning, IN/Clojure is the event bringing together a great mix of Clojure/Script masters, practitioners, and newcomers from across India and beyond.

Why Attend?

We welcome you to join nearly 149 other people and...

Speak: CFP is open till 20 Jan 2020.
Learn: We run two all-day hands-on Clojure workshops. Check 'em out here.
Converse: Derive some sweet cerebral expansion from the talks, the tweetstorms, and the “hallway tracks”.
Inspire: Trade notes, ideas, tools, and techniques with new and experienced Clojure programmers alike (photostream).
Be Inspired : Start that project, that meetup, that hack night, that business, that beautiful work of art.
Sponsor: Help foster the growing Clojure and FP community in India/Asia.
Network: Meet seasoned practitioners and awesome sponsors (listed below). Hire or get hired. Acquire or get acquired ;-)
Grab: Some cool swag, and some fun swag, and some zany swag.
Party: Eat, drink, and make merry with some of the nicest people around.

Plus, this year we enjoy the company of the perennially effervescent Bozhidar Batsov; prolific Clojurist, Emacs fanatic, maintainer of CIDER, and Lisp hacker extraordinaire.

(Oh, and tickets are going, going, going... grab yours while stocks last.)

#Sponsors:

Click here to view the Sponsorship Deck.

IN/Clojure 2020 Sponsors:

#Platinum Sponsor

#Gold Sponsor

#Bronze Sponsor

#Community Sponsor

#Speaker travel support

Hosted by

IN/Clojure

IN/Clojure is India’s annual Clojure and ClojureScript conference, including talks about features, libraries, tools, usage, and more. IN/Clojure 2020, our 4th edition, is happening on 14th and 15th Feb 2020 (Fri/Sat) in Pune, Maharashtra. more

All submissions

Previous Next

This submission has been added to the schedule

Building data platforms from business stores using Clojure

Submitted Nov 14, 2019

🌑 Submission Type: Full (40 minutes)

You have just bootstrapped and is catering thousands(or maybe millions) of happy users. Like most good applications, tech stack starts with a battle-tested RDBMS/NoSQL crud operations. Then, you start looking into user behavior, user’s interaction with the system to provide a customized experience or to deliver the next set of cool features. The ideal way to achieve such analysis is by sending custom events that can derive required metrics. Another hassle free approach is to capture change data of existing databases as an event stream. In this session, I will discuss the latter approach, it’s benefits and kind of use cases we solved with the power of Clojure. Clojure is a core part this architecture which handles spawning/destroying on-demand EMR clusters independent of cloud providers, DAG execution of EMR jobs, etc. Clojure REPL helped to speed up our development by reducing the time required to write and validate adhoc EMR queries.

Outline

Introduction
Use cases
- Building OLAP on top of RDBMS
- Hyper scalable function triggering platform(alternative to RDBMS triggers but scalable)
- Cheaper Point in time backups
- Data replication across multi-variety of databases or across multi versions
Architecture
- Independent building blocks which include:
  - Log Reader
  - Distributed Queue System (For eg Apache Kafka)
  - Cloud Storage
  - Query services
  - Visualization tools
- Benefits
- Components Synchronisation
Power of Clojure
- Writing EMR jobs
- Spawning and destroying Hadoop cluster on-demand
- Support for AWS, GCP
- By-default auto-scaling supported
- DAG Execution
- Development with REPL
- Integration with Spark, Hive, Pig
Obstacles and Learnings
Low-cost Scalable platform
Conclusion
Demo showing for any operations in DB
- Kafka being populated
- S3 dumps
- Lambda function being triggered

NOTE: Slides WIP

Requirements

Basic knowledge of Clojure and data analytics

Speaker bio

Hi, I am co-founder of Dataorc, a Data oriented startup based out of Pune. I started my professional journey with Clojure at my previous company Helpshift and have been coding in same for last 6 years. Even at Dataorc, almost every project have some part developed in Clojure. I have used Clojure for architecting automation frameworks, building super scalable backends, distributed crawlers, munching TBs of data with it.