The Fifth Elephant 2013

An Event on Big Data and Cloud Computing

Piyush Verma

Piyush Verma

@meson10

Build a Queue Based Concurrent Task Processor (using Python)

Submitted Apr 13, 2013

Learn how to develop a Persistent Queue based Task Processor using simple tools like MongoDB and Python.

Outline

At siminars.com we call it “Party”, responsible for handling all the non real-time stuff varying from Video encoding, Indexing Search Documents, Updating cache, tracking participant progress, sending out emails, payments, releasing Node.js sockets etc.

Targets to be achieved:
Evaluate the need a task processor, use cases where it would fit || Build a Queue based Task Processor using Python & MongoDB || Scale up number of processes/workers || Implement Cleanup. Keep the Queue fast and small || Scale up number of nodes

Requirements

Familiarity with MongoDB || Good Understanding of Python || Prior Experience/ Knowledge of multiprocessing || Chips to Munch every time you get restless or bored

Speaker bio

Loves Coding. Currently siminars.com
Otherwise, Loves Cycling, Loves Coffee & Loves Cartoons.

Slides

http://siminars.com/134936656289522642318098/summary.sv

Comments

{{ gettext('Login to leave a comment') }}

{{ gettext('Post a comment…') }}
{{ gettext('New comment') }}
{{ formTitle }}

{{ errorMsg }}

{{ gettext('No comments posted yet') }}

Hosted by

Jump starting better data engineering and AI futures