The Fifth Elephant 2014

A conference on big data and analytics

Rohit Yadav


Scaling with Queues

Submitted Apr 2, 2014

Share the experience of using queues based backend infra architecture for scalability, failover and data accuracy.


Talk on design and implementation of a distributed queue based scalable CQRS[1] architecture at Wingify for doing A/B testing analytics, data acquisition and distributed processing using RabbitMQ, OpenResty, Lua, Python, Redis, C++/Thrift and RocksDB.

The talk on architecture will be around distributed queue and how queueing as a scaling solution works and the rest of the talk will cover infra and scalability challenges we have solved using this architecture at Wingify where we use it for analytics, data processing, database updates and for supporting niche features within the VWO app that uses a homegrown high volume writes db called HarvestDB based on RocksDB.



Speaker bio

Speaker is a systems engineer at Wingify, a Delhi based bootstrapped startup that develops the A/B testing tool -- Visual Website Optimizer (VWO). He is an opensource enthusiast and committer with Apache CloudStack and VideoLAN VLMC. More on:


{{ gettext('Login to leave a comment') }}

{{ gettext('Post a comment…') }}
{{ gettext('New comment') }}
{{ formTitle }}

{{ errorMsg }}

{{ gettext('No comments posted yet') }}

Hosted by

All about data science and machine learning