The Fifth Elephant
The Fifth Elephant

The Fifth Elephant 2026 Winter Edition

Call for Data Problems and Proposals

All submissions

Gopi

@gopiks

Beyond Transformers: Exploring Gradient-Boosted Trees for Language Modeling

Submitted Oct 11, 2026

💭 I am submitting: Have a problem I want to discuss I have a submission for: A talk - 30-40 mins
  • Talk/session proposal - 30-40 mins

Talk duration: 30–40 minutes

Abstract

Transformers dominate modern language modeling, but their computational and memory requirements motivate exploring alternative architectures. This talk investigates whether gradient-boosted decision trees (GBMs), particularly LightGBM, can provide an alternative for next-token prediction.

I will present experiments with multiclass prediction, error-correcting output codes, and learned decoding, and discuss potential directions for improving model quality and efficiency. The goal is to identify the opportunities and limitations of GBMs for language modeling and explore how far they can go as an alternative to Transformers.

About the Speaker

Gopi Krishna Suvanam is a fintech entrepreneur and AI/ML practitioner, and Co-Founder and Director of G-Square Solutions, which develops AI and analytics solutions for financial institutions. He is also the founder of Scribbler - an on-device AI project. He holds an MBA from IIM Ahmedabad and a B.Tech in Computer Science from IIT Madras, and previously worked in global markets at Deutsche Bank in London and New York. He is also the author of Liberty & Prosperity: Liberal Economics for Achieving Universal Prosperity. His current explorations span machine learning, software systems, and alternative approaches to AI architecture.

Open Questions

  • How can GBMs scale to large-vocabulary language modeling?
  • Can better representations and layered architectures improve predictive performance?
  • What trade-offs exist between accuracy, memory use, and inference speed?

Tags

Machine Learning, NLP, Gradient-Boosted Trees, Model Efficiency

What I Hope to Gain

Technical feedback, ideas for improving the approach, and connections with researchers and practitioners exploring alternatives to Transformer-based architectures.

All submissions

Comments

{{ gettext('Login to leave a comment') }}

{{ gettext('Post a comment…') }}
{{ gettext('New comment') }}
{{ formTitle }}

{{ errorMsg }}

{{ gettext('No comments posted yet') }}

Hosted by

The Fifth Elephant
The Fifth Elephant
Jumpstart better data engineering and AI futures