The Fifth Elephant 2012

Finding the elephant in the data.

Big Data in Real Time: Processing the Social Web within a tolerable elapsed time

Submitted by Harish (@harish) on Thursday, 14 June 2012

videocam_off

Technical level

Intermediate

Section

Big Data Infrastructure & Processing

Session type

Lecture

Status

Confirmed

Vote on this proposal

Login to vote

Total votes:  +5

Objective

Understanding the challenges posed while dealing with 'Big Data', in terms of time, processing and storage with special emphasis on social media intelligence systems which require the crawling and processing terabytes of data within a 'reasonable' time frame.

Description

The extreme virality of Social data makes near real-time alerting and response systems essential to the success of Social Media monitoring and engagement platforms. From live storage systems, modified caching mechanisms and combinations of SQL and NoSql systems, we discuss new requirements, challenges and limitations that have forced us to combine, modify and at times rebuild several of these systems.

Requirements

None

Speaker bio

Harish is the CTO at Webfluenz Pte Ltd and Director at 4am Design and Technology Labs.

Harish drives the creation of proprietary Webfluenz Intelligence and Data Gathering systems by combining established computer science concepts with cutting edge and continuously evolving technologies.

In the past Harish has worked on various intelligent search systems in different environments.

Comments

Login with Twitter or Google to leave a comment