The Fifth Elephant 2014

A conference on big data and analytics

Developing Real-Time Data Pipelines with Apache Kafka

Submitted by Manisha Sethi (@manishasethi) on Thursday, 27 March 2014

videocam_off

Technical level

Advanced

Section

Full talk

Status

Submitted

Vote on this proposal

Login to vote

Total votes:  +4

Objective

The audience would be benefitted in terms of understanding "A High-throughput distributed Messaging system"- KAFKA, which is developed used at Linkedin.

Audience will be understanding :

What is Apache Kafka , What Problem Apache Kafka Solves, Brief overview about its components, Its High-throughput and Durable data persistence System , Sample Use cases, Comparison with existing solutions, API overview, Kafka powered Solutions, Q&A.

KAFKA can be in conjunction with realtime computaion systems like Storm can help us to scale at millions of records processing per second.

In nutshell, Audience will be able to understand the scenarios where kafka can be plugged in the architecture where its competitors like JMS, flume ,scribe are limiting.

kafka features of Compression and log compaction can be useful for many participants worried about network bandwidth and disk space.

Description

The Session will have an overiew , concepts , Architecture Details of KAFKA.
Where to fit it, the benefits and features.
API discussion and a Simple Demo or application.
And The Support for Kafka from other products for integration, deployement and monitoring.

Requirements

A standard VM with JAVA >1.6 and and editor like eclipse or any preferred one.

Speaker bio

I Manisha Sethi,have been working in BigData technologies like Hadoop , YARN and NoSQL DBs for many years. With three years of experience i have got the opportuninty to work on kafka in AWS as well to handle TB,s of DATA among various DC's. And I have also developed applications on kafka with Storma and Cassandra for real time Data Processing.
Currently Working with GODATADRIVEN- The Cloudera partners.

Comments

  • 1
    Vinayak Hegde (@vin) 4 years ago

    What will you be covering that is not present in the documentation ? We are looking for talks that showcase practical producion experience. Can you shed more light on these aspects ?

Login with Twitter or Google to leave a comment