The Fifth Elephant 2014

A conference on big data and analytics

Scaling with Queues

Submitted by Rohit Yadav (@bhaisaab) on Tuesday, 1 April 2014

videocam_off

Technical level

Intermediate

Section

Full talk

Status

Submitted

Vote on this proposal

Login to vote

Total votes:  +11

Objective

Share the experience of using queues based backend infra architecture for scalability, failover and data accuracy.

Description

Talk on design and implementation of a distributed queue based scalable CQRS[1] architecture at Wingify for doing A/B testing analytics, data acquisition and distributed processing using RabbitMQ, OpenResty, Lua, Python, Redis, C++/Thrift and RocksDB.

The talk on architecture will be around distributed queue and how queueing as a scaling solution works and the rest of the talk will cover infra and scalability challenges we have solved using this architecture at Wingify where we use it for analytics, data processing, database updates and for supporting niche features within the VWO app that uses a homegrown high volume writes db called HarvestDB based on RocksDB.

[1] http://martinfowler.com/bliki/CQRS.html

[2] http://engineering.wingify.com/scaling-with-queues/

Speaker bio

Speaker is a systems engineer at Wingify, a Delhi based bootstrapped startup that develops the A/B testing tool -- Visual Website Optimizer (VWO). He is an opensource enthusiast and committer with Apache CloudStack and VideoLAN VLMC. More on: bhaisaab.org

Comments

  • 1
    Govind Kanshi (@govindsk) 4 years ago

    Thanks Rohit. This will be definitely useful to see how ingestion + analytics is done.

  • 1
    Rohit Yadav (@bhaisaab) Proposer 4 years ago

    Sure, looking forward to presenting the talk.

Login with Twitter or Google to leave a comment