Submitted Jun 11, 2017

Neural Stack: Augmenting Recurrent Neural Networks with Memory

Section: Full talk Technical level: Intermediate

Recurrent neural networks (RNNs) offer a compelling tool for processing natural language input in a straightforward sequential manner. Though they suffer from various limitations which do not allow them to easily model even the simple transduction tasks. In this talk, we will discuss new memory-based recurrent networks that implement continuously differentiable analogues of traditional data structures such as Stacks, Queues, and DeQues.

Outline

NLP Applications

Language modelling
Machine Translation
Parsing

Introduction to RNN [5 mins]

RNN architecture

Limitations of a simple RNN model[8 mins]

RNN Equivalence to Finite State Machine[10 mins]

Stack[2 mins]

Stack API

Neural stack architechture[10 mins]

Neural stack architecture
Architecture
Operation

Results[5 mins]

Performance

Closing remarks [3 mins]

To faclitate better understanding of the talk, I will be giving a github repo as a take away so that the audiance can go back, download the code and play with it.

Speaker bio

Satyam Saxena is a ML researcher at Freshdesk. An IIT alumnus, his interest lie in NLP, Machine Learning, Deep Learning. Prior to this, he was a part of ML group Cisco. He was a visiting researcher at Vision Labs in IIIT Hyd where he used computer vision and deep learning to build applications to assisting visually impaired people. He presented some of this work at ICAT 2014, Turkey. https://www.linkedin.com/in/sam-iitj/

Slides

https://docs.google.com/presentation/d/17UOEgy7dq6rdJZ0c63dDo6Fj148d-iYk5ghvXDj3Sqk/edit?usp=sharing

Comments

Up next

Highway Networks and ResNet : A deeper approach towards Deep Learning .

Deep Learning though termed so but as the network becomes deeper the neural networks are more difficult to train and their preformance also start to degrade. Residual learning framework(ResNet and Highway Networks) is an Newer kind of architecture which ease the training of networks that are substantially deeper than those used previously, helps overcome the degradation problem and **lets the network decide how deep it needs to be** and also give significant boost to the performance of task in hand. The layers are reformulated as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. The Architecture won the 2015 ImageNet Challenge along with ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.

Jun 10, 2017

Anthill Inside 2017