Fine-tune Mixtral 8x7B (MoE) on Custom Data - Step by Step Guide 

Thank you for sharing, this is very helpful! Looking forward to the next videos!

kudos, really simple and direct explaination.

Awesome content as usual! Thanks!

very nicely explained!

🎯 Key Takeaways for quick navigation: 00:00 🚀 *Introduction to Fine-Tuning Mixtral 87B Model* - Overview of the video's purpose: fine-tuning Mixtral 87B model from Mistral AI on a custom dataset. - Mention of the popularity and potential of Mixtral 87B as a mixture of experts model. - Emphasis on practical considerations for fine-tuning, such as VRAM requirements and dataset details. 01:28 🛠️ *Installing Required Packages and Data Set Overview* - Installation of necessary packages: Transformers, TRL, accelerate, P torch bits, and bytes. - Discussion on using the Mosaic ML Instruct with 3 datasets for fine-tuning. - Overview of the dataset structure, splits, and sources. 03:45 📝 *Formatting Data for Fine-Tuning Mixtral 87B* - Explanation of the prompt template for fine-tuning, specific to Mixtral 87B Instruct version. - Discussion on rearranging data to make it more challenging by creating instructions from provided text. - Demonstration of a function to reformat the initial data into the desired prompt template. 06:28 🧩 *Loading Base Model and Configuring for Fine-Tuning* - Acknowledgment of the source for the notebook and clarification that the base version is used. - Setting configurations, loading the model, and tokenizer, along with using Flash attention. - Explanation of the importance of setting up configurations for a smooth fine-tuning process. 08:18 🔄 *Checking Base Model Responses Before Fine-Tuning* - Use of a function to check responses from the base model before any fine-tuning. - Illustration of the base model behavior in generating responses to a given prompt. - Recognition that the base model tends to follow next word prediction rather than explicit instructions. 10:06 📏 *Determining Max Sequence Length for Fine-Tuning* - Explanation of the importance of max sequence length in fine-tuning Mixtral 87B. - Presentation of a code snippet to analyze the distribution of sequence lengths in the dataset. - Emphasis on selecting a max sequence length that covers the majority of examples. 12:20 🧠 *Adding Adapters with Lura for Fine-Tuning* - Overview of the Mixtral 87B architecture, focusing on linear layers for adding adapters. - Introduction to Lura configuration for attaching adapters to specific layers. - Demonstration of setting hyperparameters and using the TRL package for supervised fine-tuning. 14:36 🚥 *Setting Up Trainer and Initiating Fine-Tuning* - Verification of multiple GPUs for parallelization during model training. - Definition of output directory and selection of training epochs or steps. - Importance of configuring the trainer, including considerations for max sequence length. 16:50 📈 *Analyzing Fine-Tuning Results and Storing Model* - Presentation of training and validation loss graphs, indicating a gradual decrease. - Acknowledgment of the need for potential longer training for better model performance. - Demonstration of storing the fine-tuned model weights locally and pushing to Hugging Face repository. 17:46 🔄 *Testing Fine-Tuned Model Responses* - Utilization of the fine-tuned model to generate responses to a given prompt. - Comparison of responses before and after fine-tuning, showcasing improved adherence to instructions. - Acknowledgment that further training could enhance the model's performance. Made with HARPA AI

Thanks for the video 😃 i just have a question , is it possible to use the model through an API and also provide the source files for the data with the response ?

can you also make a video on fine-tuning multimodal models like llava, cog-vlm

Great video just have one question can we use the fine-tuned model as a pickle file?

Thanks for the tag @Prompt Engineering! What else is your audience requesting the most these days? Would love to find ways to create some value for them together!

Can you make this for home computer use in terms of my personal data and tech it to use tools on your system and online

3:37 format 4:15 follow a different format 4:26 Indicate the end of user input 4:33 special token Indicate the end of model response 4:39 you need to provide your data in this format 5:08 def create_prompt 5:31 System message 6:16 Load our based model

Thanks for the guide! How to continue fine-tuning process such as in this case? Can you load previous work (Lora) and carry on, or do you need to restart?

Could you make a tutorial teaching how to convert a model to ggml format?

Can I use this to train a model to answer questions from a list of pdfs?

IM sceptical this actually is effectively training mixtral MoE model and not making it worse!

Great

How can I use a generative model to manipulate content of my website Ex. Showing response from my site based on prompt given by the user

Thanks, what is the cost to do this? Server cost?

What were your VM instance specs. It is struggling with an A100?

why are you using packing in the SFTTrainer if you just said that you're going to pad the examples?

So with two 3090s this should work? And what about using multiple different gpus for training? Like I have one 3090ti 24g and one 4060 8g

Awesome man, any idea of how to get this running on a colab gpu or inference cost down?

That's a great video. Thanks for sharing. After pushing the model to hugging face, how to host it independently on runpod using VLLM ? When I try to do that, it gives me error. Tried searching a lot of videos and articles. But of no use so far.

I've noticed that Mixtral 8x7b-instruct ( and other mistral models ) constantly repeat part of the system prompt. Have you noticed this / found a fix for it?

how do you format a prompt that has multiple requests and responses within the same context???????

Could you please share the requirement.txt, i am having version conflicts despite using A100 GPU!

requires colab pro to work?

Great video! Could you please consider training and deploying it in Sagemaker?

Are you finetuning the mixtral instruct version they just released or base model??

at 5:58, Why is the sample["response"] given as the input and sample["prompt"] is given as response

Hi, thanks for this step by step guide, but in case we want LLM to learn something new about our domain (let's say it will be book Lord of the Rings) and we later want to ask our model open questions about this book (like 'where Frodo gets his sword?') what should we do? We definetely cannot prepare dataset in form of QnA, so it should self-supervised training. But I never saw examples of doing this and I can't image how it supposed to be done? Is it even possible? Looks like we should start from base model, fine-tune it somehow with our book, and later we should apply fine-tuning for instruct on top of it, right? But in this case someone still should prepare this QnA? I'm frustrated.

❤👍🏼

can you finetune Uncensored Models of this with gemini pro ai ?

Can't we train the quantized version in a smaller GPU instead of training the full model?

Can you clairfy, 1 epoch would be one run of the full data (34333 steps of your trimmed data) Why would you run this 2 epochs, does going over the data twice improve it? Also how did you determine 32 was a good batch size for this data size? (this is about 0.9% of the data?)

i wish someone made a video from collecting data example pdf, conver that to working dataset tha can be used to train model, everyone is using huggingface models and just retrain another llm

Thank you share such wonderfull video! Waiting for a video that discuss about finetuning. So that we can use higher than 32k token. Have you or any person worked with the folloing? 0) How did we measure performance after fine tuning? Did they perform well? Perplexity? 1) Json files? Creating graphs to store the context? 2) and or Large csv/sql file? As llama code sql code is not working well 3) Any image/diffusion models? Appreciate it!

Hi, I am trying to build an organisation level AI trained on my company data I would to know how can I create dataset for my data to be trained on mistral AI I was unable to find any tutorial on how to create a dataset for large data

So can't we run in colab or kaggle notebook?

How do I fine tune without prompt and instruction? I basically want the model to "know" about a thousand very recent web articles.

please let me know how to create a fixed forms with the below structures with special command to LLM: Give me score out of 4 for (based on the TOEFL rubric) without any explanation, just display the score. General Description: Topic Development: Language Use: Delivery: Overall Score: Identify the number of grammatical and vocabulary errors, providing a sentence-by-sentence breakdown. 'Sentence 1: Errors: Grammar: Vocabulary: Recommend effective academic vocabulary and grammar:' 'Sentence 2: Errors: Grammar: Vocabulary: Recommend effective academic vocabulary and grammar:' .......

I think you can rent an H100 for $5/hour. So this would cost about $7

64 gb vram kaha se laaoge pata nahi kaunse dataset par fine tune kiya hai bhai kisi kaam ka nahi hai ye video tere paise to view se aa gaye humare paise kaise banege

🥹‼️I am student.. who has no budget at all..but intrested in training any of the llm with my own dataset What are the cost effective ways?

How much VRAM is necessary?