Anthill Inside 2017

On theory and concepts in Machine Learning, Deep Learning and Artificial Intelligence. Formerly Deep Learning Conf.

Anuj Gupta


Synthetic Gradients – Decoupling Layers of a Neural Nets

Submitted Jun 9, 2017

Once in a while comes an (crazy!) idea that can change the very fundamentals of an area. In this talk we will see one such idea that can change how neural networks are trained.

As of now Back propagation algorithm is at the heart of training any neural net. However, the algorithm suffers from certain drawbacks which forces layers of the neural net to be trained strictly in sequential manner. In this talk we see a very powerful technique to break free from this severe limitation.


Refresher on Back propagation [5 mins]

  • Basics

Problems with Back propagation [5 mins]

  • Forward locking
  • Backward locking
  • Update locking
  • Impact of locking

Why does it matter [1 mins]

Applications [3 mins]

Solution [12 mins]

  • Synthetic Gradients
  • Breaking backward & update locking

Results [5 mins]

  • Backprop vs Synthetic Gradients

Complete unlock [2 mins]

  • Breaking forward locking

Closing remarks [3 mins]

To facilitate better understanding, I will be giving a github repo as a take away so that the audience can go back, download the code and play with it.
Code assosiated with this talk :


Basic understanding of Back propagation algorithm

Speaker bio

Anuj Gupta is a senior ML researcher at Freshdesk; working in the area NLP, Machine Learning, Deep learning. Earlier he was heading ML efforts at Airwoot(Now acquired by Freshdesk). He dropped out of Phd in ML to work with startups. He graduated from IIIT H with specialization in theoretical comp science.

He has given tech talks at prestigious forums like PyData DC, Fifth Elephant, ICDCN, PODC, IIT Delhi, IIIT Hyderabad and special interest groups like DLBLR. More about him -

Work from my past life



{{ gettext('Login to leave a comment') }}

{{ gettext('Post a comment…') }}
{{ gettext('New comment') }}
{{ formTitle }}

{{ errorMsg }}

{{ gettext('No comments posted yet') }}

Hosted by

Anthill Inside is a forum for conversations about risk mitigation and governance in Artificial Intelligence and Deep Learning. AI developers, researchers, startup founders, ethicists, and AI enthusiasts are encouraged to: more