Making sense out of semi-structured log data

Jul 2012

23 Mon

24 Tue

25 Wed

26 Thu

27 Fri 09:30 AM – 05:30 PM IST

28 Sat 09:30 AM – 05:00 PM IST

29 Sun

Nimhans Convention Centre, Bangalore

Making sense out of semi-structured log data

Submitted Jun 29, 2012

Section: Big Data Infrastructure & Processing Technical level: Intermediate Session type: Lecture

Discuss how to use big-data systems for analyzing Log data in real time.

Outline

Any complex OLTP system comprises of many distributed modules running on separate nodes. Failure of a single module or a bug in any one can cause a request to fail completely or respond with incorrect data. Debug and error level logs for each module helps developers to troubleshoot such issues. But for a large scale distributed system collecting the log files from all nodes and analyzing them is a cumbersome task because of the unstructured/semi-structured nature of the log entries.

To address the above challenge, the platforms team at Capillary Technologies has developed a Log Processing Framework which helps us understand and analyze all the logs at a single place. The framework has been built using Flume, MongoDB, Hive, Hadoop and custom components which provides us the capability to analyze large amounts of log data using Map Reduce Jobs. The framework enables us to search the exact causes of failure and performance metrics from large amounts of raw text using SQL like interface in near realtime.

This talk will focus on the design decisions that were made and the challenges that we encountered while building this framework.

Requirements

Basic understanding of MongoDB, Hive/Hadoop.

Speaker bio

Pravanjan Choudhury is an Architect with Platforms & Applications group at Capillary Technologies. He has more than 9 years of experience in building mission-critical large scale software & cloud based products. Previously, he was an Architect in Minekey, a silicon valley start-up and in the past has done consulting for several research projects of National Semiconductor.

Piyush Goel is Team Lead with the Platforms group at Capillary Technologies. Before joining Capillary he worked at Yahoo! Bangalore as Senior Software Engineer with the Emerging Markets team. He holds a BTech & MTech in Computer Science from Indian Institute of Technology, Kharagpur.

The Fifth Elephant 2012