RWKV: reinventing RNNs for the transformer era

About the Paper

In the last three years, RNNs have caught up with the unparalleled capabilities of Transformers. The promise of Receptive Weighted Key Value (RWKV) is that this novel architecture combines the desirable aspects of both RNNs and Transformers: the massively parallelizable transformer-esque training, and RNN’s consistent computational and memory complexity during inference.
RWKV (pronounced as “RwaKuv”) is an attention-free language model, theoretically capable of handling an “infinite” context length.

Key takeaways for audience

• Intuitive understanding of RWKV’s formulation, using math and code.
• Information about performance benchmarks and the scaling laws - and evaluate how you can use RMKV in your work.
• A demo of RWKV’s inference prowess.

About the presenter

Yashodeep Deshmukh is Deputy Manager at Ashok Leyland.

RSVP

This paper discussion will be held at Atlassian’s office in EGL, Bangalore. In-person attendance is free. The Fifth Elephant members can join remotely to watch the live stream.

About The Fifth Elephant monthly paper discussions

The monthly discussions are organized to understand popular papers in Generative AI, DL, and ML domains. Papers are curated to benefit the community. The paper discussion is organized on the first Friday of each month, from 5:30 PM - 7:00 PM.

How you can contribute

1. Suggest a paper at https://hasgeek.com/fifthelephant/call-for-papers/sub
2. Moderate/discuss a paper someone else is proposing.
3. Pick up a membership to support The Fifth Elephant’s activities.
4. Spread the word among colleagues and friends. Join The Fifth Elephant Telegram group or WhatsApp group.

Contact

For inquiries, leave a comment or call The Fifth Elephant at +91-7676332020.