The legend returns, Always excited for your videos. I am an international student at Shanghai Jiao Tong daxue. Your videos have given me a very strong foundation of transformers. Much blessings your way
I humbly request you to make videos on how to build a career in machine learning and AI. I am a huge fan of your videos and i thank you for all the knowledge that you have shared
Hi! I will for sure make a video in the future about my personal journey. I hope that can help more people in navigating their own journeys. Have a nice day!
I believe the most evident insight of DPO is to change a RL problem to an equivalent MLE, while the optimal reward model is guarranteed by the human input as definition. That's the meat. But the efficiency depends still on the human annotater's consistency.
Great explanation but I have some doubts, please help 36:50 in Ldpo π* was replaced with π theta... why π theta is considered as optimal policy? 44:13 You said "each hidden state contains information about itself and all the tokens that comes before it", but this is applicable only to decoder part of the transformer. So this transformer layer is actually a decoder layer? like GPT
Amazing video. Would it be possible for you to explain video-transformers or potential guesses at how Sora works? Another exciting idea is explaining GFlowNets
Hello Umar, Great as usual, however why do you say at 46:11, that you need to sum log probabilities up? The objective function is the expectation of logarithm of the difference of two weighted log probabilities ratios. I don't get what do you want to sum up exactly? Thank you
Thanks for the great video! Very intuitive explanation and particular thanks for the code examples. Question: at 37:41, how do we know that the solving the optimization problem will yield the pi_*? Is there a guaranteed unique solution?
Please check the paper I linked in the description for a complete derivation of the formula. It is also done in the DPO paper, but in my opinion the other paper is better suited for this particular derivation.
I enjoy your videos umar on my phone while commuting or sitting in a coffe. Only the small fint on a phone is tiring me ..if you make them a bit bigger that will be better
Thanks for the video. Do you know any way on how we can create a dataset for DPO training. I currently have only question, answer pairs. Is it fine if i take y_w as answer and y_l as some random text(which would obviously have lower preference than answer) and then train it?
Hi Umar, If I use LoRA for fine-tuning a chat model with DPO loss, what should I use as a reference model? - The chat model applied LoRA - Or the chat model itself without LoRA?
Considering LoRA is just a way to "store" fine-tuned weights with a smaller computation/memory footprint, the model WITHOUT LoRA should be used as the reference model.
@@umarjamilai With my limited GPU, I can only fine-tune by combining a 4-bit-quantized model + LoRA. Surprisingly, using just the 4-bit model leads to NaN weight updates after one batch. But once LoRA is added, my loss updates smoothly without any problems.
Thank you SO much for the quick answer and your excellent video. I did get the hang of DPO loss and be able to implement DPO loss + training loop with vanilla PyTorch code.
According to the Hugging Face library, yes, looks like you need a dataset with prompt and two answers, one is called the "chosen" one and the other is the "rejected" one. I'm pretty sure there are ways to convert more than two preferences into a dataset of two preferences.
@@umarjamilai thank you! Yes of course. I am just wondering why it wouldn’t help to have more than 1 rejected for 1 accepted. I guess the formula does not consider this case but may add value.
