RWKV: reinventing RNNs for the transformer era

RWKV: reinventing RNNs for the transformer era

The Fifth Elephant paper reading meet-up - April 2024

Tickets

Loading…

About the Paper

In the last three years, RNNs have caught up with the unparalleled capabilities of Transformers. The promise of Receptive Weighted Key Value (RWKV) is that this novel architecture combines the desirable aspects of both RNNs and Transformers: the massively parallelizable transformer-esque training, and RNN’s consistent computational and memory complexity during inference.
RWKV (pronounced as “RwaKuv”) is an attention-free language model, theoretically capable of handling an “infinite” context length.

Key takeaways for audience

  • Intuitive understanding of RWKV’s formulation, using math and code.
  • Information about performance benchmarks and the scaling laws - and evaluate how you can use RMKV in your work.
  • A demo of RWKV’s inference prowess.

About the presenter

Yashodeep Deshmukh is Deputy Manager at Ashok Leyland.

RSVP

This paper discussion will be held at Atlassian’s office in EGL, Bangalore. In-person attendance is free. The Fifth Elephant members can join remotely to watch the live stream.

About The Fifth Elephant monthly paper discussions

The monthly discussions are organized to understand popular papers in Generative AI, DL, and ML domains. Papers are curated to benefit the community. The paper discussion is organized on the first Friday of each month, from 5:30 PM - 7:00 PM.

How you can contribute

  1. Suggest a paper at https://hasgeek.com/fifthelephant/call-for-papers/sub
  2. Moderate/discuss a paper someone else is proposing.
  3. Pick up a membership to support The Fifth Elephant’s activities.
  4. Spread the word among colleagues and friends. Join The Fifth Elephant Telegram group or WhatsApp group.

Contact

For inquiries, leave a comment or call The Fifth Elephant at +91-7676332020.

Videos

See all
RWKV: reinventing RNNs for the transformer era

RWKV: reinventing RNNs for the transformer era

1 hour 5 April 2024

Venue

Atlassian

3rd Floor, Sunriver,

Wework Embassy Golf Links Business Park, Domlur

Bangalore - 560071

Karnataka, IN

Loading…

Hosted by

Jump starting better data engineering and AI futures

Supported by

Venue host

Atlassian unleashes the potential of every team. Our agile & DevOps, IT service management and work management software helps teams organize, discuss, and compl