The Fifth Elephant 2014

Share the experience of using queues based backend infra architecture for scalability, failover and data accuracy.

Outline

Talk on design and implementation of a distributed queue based scalable CQRS[1] architecture at Wingify for doing A/B testing analytics, data acquisition and distributed processing using RabbitMQ, OpenResty, Lua, Python, Redis, C++/Thrift and RocksDB.

The talk on architecture will be around distributed queue and how queueing as a scaling solution works and the rest of the talk will cover infra and scalability challenges we have solved using this architecture at Wingify where we use it for analytics, data processing, database updates and for supporting niche features within the VWO app that uses a homegrown high volume writes db called HarvestDB based on RocksDB.

[1] http://martinfowler.com/bliki/CQRS.html

[2] http://engineering.wingify.com/scaling-with-queues/

Speaker bio

Speaker is a systems engineer at Wingify, a Delhi based bootstrapped startup that develops the A/B testing tool -- Visual Website Optimizer (VWO). He is an opensource enthusiast and committer with Apache CloudStack and VideoLAN VLMC. More on: bhaisaab.org