The Fifth Elephant 2015

A conference on data, machine learning, and distributed and parallel computing

Bhasker Kode


Instrumenting your kafka & storm pipeline

Submitted May 11, 2015

tips to design your stream processing setup.
what all can go wrong, how to instrument it.


introduction to a production setup that handles billions of events per week through our home-grown apache kafka client, process the stream with storm - and then aggregate into postgres. will also share pro’s and con’s of building our own kafka client as opposed to re-using one.

Speaker bio

Bosky (@bhaskerkode) leads a product engg team at Helpshift & works on erlang, clojure and golang.

building distributed systems since ‘06 across edtech, adtech & mobile in erlang, clojure & go.
=> &

Built a kafka producer/micro-service used in production at helpshift, layer, and several other companies.
( Recommended by Apache kafka )

Using storm in production for sentiment analysis, topic extraction, naive bayes classification, etc.
Eager to learn more about best practices in storm deployment/management. Incidentally another part that feeds into this system is in golang, uses the Shopify kafka producer.




{{ gettext('Login to leave a comment') }}

{{ gettext('Post a comment…') }}
{{ gettext('New comment') }}
{{ formTitle }}

{{ errorMsg }}

{{ gettext('No comments posted yet') }}