The Fifth Elephant 2014

A conference on big data and analytics

Real Time Secure API delivering data @ scale

Submitted by Akash Mishra on Wednesday, 4 June 2014

videocam_off

Technical level

Beginner

Section

Crisp talk

Status

Submitted

Vote on this proposal

Login to vote

Total votes:  +4

Objective

At ThoughtWorks, we have used a Hybrid Approach for designing a Real Time secure API, which gives various adhoc querying capability on large amount of data.

This talk will focus on various pro's and con's of traditional database system [RDBMS] as well as big data system [Hadoop/Hive] to build an API. Also I will present how both of these system counterbalance each others problems and deliver large amount of data in Real Time.

Description

Traditional databases are known for their fast response and security whereas Big Data system are known for their scalability and ability to handle large amount of data.

What happens when we want to build a system that required following?

  • Real Time Response

  • Security

  • Handle GB's of Data

This presentation will walk you through various requirement, system architecture and key design decision for designing a Secure Real time API. This hybrid approach uses both batch processing on Hadoop and real time processing using a Java application and RDBMS.

Various tech used for API are:
1. Hadoop/Hive
2. Oozie
3. Sqoop
4. RDBMS

Speaker bio

I am a Developer @ ThougtWorks where I've been helping our clients on delivery projects using Big Data Technologies. I have deep interest in Distributed computing and Real time system.

Links

Comments

  • 1
    Jamshed Katta (@jamshedkatta) 4 years ago

    Wish you all the best.

Login with Twitter or Google to leave a comment