Anthill Inside 2017

On theory and concepts in Machine Learning, Deep Learning and Artificial Intelligence. Formerly Deep Learning Conf.

Synthetic Gradients – Decoupling Layers of a Neural Nets

Submitted by Anuj Gupta (@anujgupta82) on Friday, 9 June 2017

videocam_off

Technical level

Intermediate

Section

Full talk

Status

Confirmed & Scheduled

View proposal in schedule

Vote on this proposal

Login to vote

Total votes:  +6

Abstract

Once in a while comes an (crazy!) idea that can change the very fundamentals of an area. In this talk we will see one such idea that can change how neural networks are trained.

As of now Back propagation algorithm is at the heart of training any neural net. However, the algorithm suffers from certain drawbacks which forces layers of the neural net to be trained strictly in sequential manner. In this talk we see a very powerful technique to break free from this severe limitation.

Outline

Refresher on Back propagation [5 mins]

  • Basics

Problems with Back propagation [5 mins]

  • Forward locking
  • Backward locking
  • Update locking
  • Impact of locking

Why does it matter [1 mins]

Applications [3 mins]

Solution [12 mins]

  • Synthetic Gradients
  • Breaking backward & update locking

Results [5 mins]

  • Backprop vs Synthetic Gradients

Complete unlock [2 mins]

  • Breaking forward locking

Closing remarks [3 mins]

To facilitate better understanding, I will be giving a github repo as a take away so that the audience can go back, download the code and play with it.
Code assosiated with this talk : https://github.com/anujgupta82/Synthetic_Gradients

Requirements

Basic understanding of Back propagation algorithm

Speaker bio

Anuj Gupta is a senior ML researcher at Freshdesk; working in the area NLP, Machine Learning, Deep learning. Earlier he was heading ML efforts at Airwoot(Now acquired by Freshdesk). He dropped out of Phd in ML to work with startups. He graduated from IIIT H with specialization in theoretical comp science.

He has given tech talks at prestigious forums like PyData DC, Fifth Elephant, ICDCN, PODC, IIT Delhi, IIIT Hyderabad and special interest groups like DLBLR. More about him - https://www.linkedin.com/in/anuj-gupta-15585792/

Work from my past life

Links

Slides

https://docs.google.com/presentation/d/10qQeuHkQ9ZkzEXD7IQS9MCpnnFEH-oMBgvq8S5bHQdM/

Comments

Login with Twitter or Google to leave a comment