The Fifth Elephant 2016

India's most renowned data science conference


A large scale IOT platform architecture using open source apache projects like Nifi, Kafka, Storm, Spark and Hadoop.

Submitted by Satish Duggana (@satishd) on Saturday, 30 April 2016

Section: Full talk Technical level: Intermediate

Vote on this proposal

Login to vote

Total votes:  +10


Gartner predicts there will be 26 billion devices on the Internet of Things by 2020. Capturing and analyzing data from connected devices provides a wealth of opportunity. In this session we will look at how open source Apache projects like Apache NiFi, Kafka, Storm, Spark and Hadoop can work in concert to analyze and provide insights in a large scale distributed IoT architecture.


  • IoT Platform high level architecture
  • Edge processing (Apache NiFi)
  • Streaming and analytics (Kafka, Storm, Spark)
    • Ingestion
    • Transformation
    • Analytics
  • Notifications (Storm)
  • Alerts (Hadoop, Storm)
  • Storage (Hadoop)
  • Application development using SDK and UI

Speaker bio

Currently, helping design and development of IoT platform at Hortonworks and contributing various features in Apache Storm. Built large scale distributed systems, application servers, ESBs and backend infrastructure platforms for last 14 years in Hortonworks, BEA Systems/Oracle and Pramati Technologies.


  • 1
    t3rmin4t0r (@t3rmin4t0r) 2 years ago

    [~satishd]: is this about the technologies listed (there are too many) or about how they are relevant for IoT?

    NiFi’s reverse connectivity to the things from the internet is crucial in its success, not the thing into the data center.

    A crisp talk on NiFi, particularly how a data center can send out a message back to the connected device would be very interesting.

    • 1
      Satish Duggana (@satishd) Proposer 2 years ago (edited 2 years ago)

      [~t3rmin4t0r] It is about how these technologies are relevant in building IoT platform. Right, I will cover more on NiFi including how a datacenter can send messages back to the device as you suggested.

Login with Twitter or Google to leave a comment