arrow_back Building a high performance distributed crawler
Build a Queue Based Concurrent Task Processor (using Python)
Submitted by Piyush Verma (@meson10) on Saturday, 13 April 2013
Learn how to develop a Persistent Queue based Task Processor using simple tools like MongoDB and Python.
At siminars.com we call it "Party", responsible for handling all the non real-time stuff varying from Video encoding, Indexing Search Documents, Updating cache, tracking participant progress, sending out emails, payments, releasing Node.js sockets etc.
Targets to be achieved: Evaluate the need a task processor, use cases where it would fit || Build a Queue based Task Processor using Python & MongoDB || Scale up number of processes/workers || Implement Cleanup. Keep the Queue fast and small || Scale up number of nodes
Familiarity with MongoDB || Good Understanding of Python || Prior Experience/ Knowledge of multiprocessing || Chips to Munch every time you get restless or bored
Loves Coding. Currently siminars.com Otherwise, Loves Cycling, Loves Coffee & Loves Cartoons.