Make a submission

Submissions are closed for this project

PyCon, the gathering for the community using and developing the open-source Python programming language. This is the second year of the PyCon Pune where the community will meet for two days of talks and working on upstream projects in two days of dev sprint. CFP ends on 15th September AoE.

Website: https://pune.pycon.org/2018/

Note to submitters: The talks will be selected by a team. We will not count the public votes.

Hosted by

PyCon Pune 2017 more

Mahendra Yadav

@userimack

Airflow: To Manage Data Pipelines

Submitted Aug 17, 2017

A significant part of the IT/Data Engineering team is spent on writing and scheduling jobs, monitoring and troubleshooting the issues. Enterprise data originates from various sources and there are various business rules and processes that govern how that data can be consumed.

Airflow is a platform to programmatically author, schedule and monitor workflows. (https://airflow.incubator.apache.org/)

The various tasks in the workflow(s) are configured as a Directed Acyclic Graph. This talk covers how Airflow is used to establish better workflows for data engineering.

P.S: This talk is inspired from Bargava Subramanian (@barsubra) proposal.

Outline

  1. Introduction to Airflow
  2. Existing challenges in data engineering - creating/monitoring/troubleshooting workflows
  3. Main advantages of Airflow
  4. Basic Concepts
  5. Airflow in practice - case study

Speaker bio

Mahendra Yadav is a Data Engineer at Azri Solutions, Hyderabad. In his day to day work he processes a lot of data from different sources.

Links

Slides

https://userimack.github.io/airflow_slides/airflow_slides/#/

Comments

{{ gettext('Login to leave a comment') }}

{{ gettext('Post a comment…') }}
{{ gettext('New comment') }}
{{ formTitle }}

{{ errorMsg }}

{{ gettext('No comments posted yet') }}

Bhavesh Anand

Melissa : A lovely virtual assistant for OS X, Windows and Linux systems.

Melissa is a virtual assistant for OS X, Windows and Linux systems. She currently uses Google Chrome’s speech-to-text engine, OS X’s say command, Linux’s espeak command or Ivona TTS along with some magical scripting which makes her alive, developed by Tanay Pant and a group of sorcerers. The Web UI for Melissa has been designed by Nakul Saxena more

18 Aug 2017