- We aim to revolutionize the way classical LMs adapt to different languages and domains.
- Our goal is to create a group of domain-adaptive pretrained models, each specializing in a unique language, through the application of Mixture of Experts (MoE) in our Domain Adaptive Pre-training.
- This approach will enable us to leverage the strengths of individual models, enhancing their performance across various domains.
- We are pushing the boundaries of current techniques, seeking to create a more efficient and versatile modeling strategy for LLMs.
Project Baarat is an open-source initiative to leverage the power of LLMs on Indic-NLP tasks. We aim to build Continually pre-trained, Task Specific Language Models in a Mixture of Experts (MoE) setup. We plan on making a multilingual and cross-lingual LLM that is :
-
Pre-trained on a large text corpus containing various sources of knowledge including crawled wikipedia articles, textbooks, news, social media sites, magazines etc.
-
Fine-tuned on different downstream tasks. We first train a 7B LLaMa-2 model on a text corpus in the target language and save it as a base model. We have considered the following tasks as downstream tasks that will be incorporated in the fine-tuning process:
- Machine Translation
- Mathematical and Logical Reasoning
- Question Answering
- Instruct Fine-Tuning
- Tokenizers for Indian Languages: Robust tokenization tools tailored for the unique structures of regional Indian languages.
- Fine-tuned Language Models: Leveraging the power of Large Language Models (LLMs) fine-tuned for Indian languages to understand and generate text with high accuracy.
- Collection of models and data: Completely free and open-source hub of datasets , models, all leveraged into a single python module with documentation for easy usage.
- High Quality Datasets: Take a look at our suite of cleaned datasets ready for your own downstream training purposes.
https://github.com/asphytheghoul/Baarat
Here is the link to our presentation : https://docs.google.com/presentation/d/1in4MhQkY6N5SnO-PJ9OVhVIe9K6jOXLRrU45GJbNPF0/edit?usp=sharing
This is the link to our project video demo :
https://drive.google.com/file/d/19YY1dBt0t29NtIZGjQsZivuKwfy9IOkC/view?usp=sharing
Login to leave a comment
Akshobhya
@akshobhya_j Editor & Promoter
Hello Akash, can you reply with the demo video link and the project presentation link?
Akash Kamalesh
@asphytheghoul Submitter
Hello Akshobya, apologies for the late reply!
Here is the link to our presentation : https://docs.google.com/presentation/d/1in4MhQkY6N5SnO-PJ9OVhVIe9K6jOXLRrU45GJbNPF0/edit?usp=sharing
This is the link to our project video demo :
https://drive.google.com/file/d/19YY1dBt0t29NtIZGjQsZivuKwfy9IOkC/view?usp=sharing
Thank you and apologies once again for the delayed response.
Akshobhya
@akshobhya_j Editor & Promoter
Can you reply to this comment with the base model that you are using in this project?
Arvind Saraf
@arvinds
Quick question (apologies, I haven't used Switch transformers yet) - MoE usually struggles with context across individual experts. If the text has mix of say Hindi & Kannada, how will the routing be handled - since different parts of the output may get tokens from differetnt LLMs. How are they combined?
Akash Kamalesh
@asphytheghoul Submitter
Hello Arvind! This is a very interesting case and is quite probable while entering an input. There's two possible cases here that could happen, (we are still exploring about possible alternatives but this is what we have in mind). If a user types a mix in kannada and english (say), the query is converted to an embedding and outputs a probability distribution across the experts. This will involve rigorous training as it will understand the task that the user wants to perform and route it appropriately to the expert. If the input is a case of english , romanized kannada and kannada, the model will be able to handle this because the adapters are trained with such data and initial testing from our end shows plausible results in doing translation between the among 3 languages in any combination. The issue is when you combine two primary languages - Hindi and Kannada in one single prompt. We have developed a language identification model which will identify the language of each sentence and use this information to route the tokens to the appropriate adapter (for that language). So an input with sentences in a mix of languages should be handled appropriately albeit we can only comment on the same after we finish training and have our results! The problem that might arise is what if an input sentence itself has tokens with different languages in it . That is a case we are yet to decide on a method to handle and is also currently under research.
Akshobhya
@akshobhya_j Editor & Promoter
Congratulations, @asphytheghoul, @tanisthahota, Anirudh. This project has been shortlist as part of the Fifth Elephant Open Source AI Hackathon. You can apply to Microsoft Founders Hub for credits. Application details are explained at http://has.gy/o4RD
Do connect with your mentor @cerebraltangent to refine the idea and continue building the project.
All the best!
Akshobhya
@akshobhya_j Editor & Promoter
@asphytheghoul, @Anirudh , thank you for your proposal submission to The Fifth Elephant Open Source AI Hackathon. The proposal addresses the need for a more efficient and versatile modeling strategy for Large Language Models (LLMs) to adapt to different languages and domains. The implementation of Mixture of Experts (MoE) to create domain-adaptive pretrained models for specific languages demonstrates an innovative approach to enhancing model performance. This submission needs to be updated based on the following considerations.
Technical Suggestions
Closing Thoughts
The proposal presents an ambitious and innovative approach to addressing the adaptability and performance of LLMs across diverse languages and domains. Enhancing the transparency and depth of technical methodologies, thorough validation of extensions and frameworks, and a holistic approach to ethical considerations and deployment will be pivotal in realizing the potential of this groundbreaking initiative. We look forward to witnessing the outcome of this promising endeavor.
→ Utilize the available platforms such as The Fifth Elephant WhatsApp group to engage with mentors and seek guidance on technical and implementation aspects of your project.