I never thought I'd say this, but I'm actually excited to learn about efficient training methods for deep learning models on consumer GPUs. Who knew running out of GPU memory could be so... enlightening? Thanks for explaining LoRA and Galore in a way that doesn't make my brain hurt (too much). Now, if you'll excuse me, I have some large language models to train or at least, try not to run out of GPU memory
Congrats on your PhD! Could you use 'r' as a hyperparameter during pretraining as well? e.g. start pretraining with low r and gradually increase it as more precision is needed? I don't think it could do that much since gains are already very high at the start.
My editor uses Adobe Premiere Pro for editing (that is where Ms. Coffee Bean is cut in ). For the visuals, I use Powerpoint. The "morph" transition is really useful.
One of the selling points of Lora is to be able to mix and match the A and B matrices for different fine-tuning runs, without having to keep the weights of the full model if they are available elsewhere. Here it seems you have to save the entire model, so this is a big tradeoff compared to Lora and derivatives.
Can info shed during the lossy compression process be set aside in a non memory fashion for retrieval? Thinking out loud. Not always helpful. But state mapping would be interesting in the training process as well as post.
As far as i get it, i determine the gradients G and and also a low rank component P. That component allows me to „shrink“ the gradient matrix G i calculate at every step to down to R before applying it to W. So i do not save compute while calculating the gradients, but while applying and saving (as momentum or such) them?
I imagine that they would do the SVD on the CPU as it would take way too much ram to do it on the GPU or maybe they do it layer by layer and discard the intermediate buffers. Anyway it seems like a great idea. ReLoRA achieved something similar but it required an small amount of pre training and I expect slower convergency for ReLoRA because each time a new LoRA is initialized the weights need to be moved around a lot while using SVD they would be roughly where they need to be minimizing the error in the main matrix.
Holy frick what a perfectly concise video on galore! There is a GitHub implementation of the research from the paper, they are currently working on a multigpu implementation. I too am curious how well things scale up to modern and larger llms, and have a multigpu rig I want to test it out on.
The size of a LoRA (on disk) is a fraction of the size of the model it's applied to. GaLore looses this benefit by updating the weights. If you have use for many fine tunes of a single model, two GaLore fine-tunes will take more space than ten LoRAs (depending on rank, to be fair). I assume they don't mention this very significant tradeoff, as you don't mention it in the video. That seems like a dishonest comparison, if that's the case.
Fair. It's just that HDD / SSD storage is not considered a bottleneck, while the size of the GPUs surely is. The first is cheap and abundant (terabytes), the second one is very expensive and limited (tens of gigabytes).
@@AICoffeeBreak Minor nitpick: It also allows you to fit multiple LoRa's in GPU memory at the same time and thus run inference on different finetunes at the same time P.S. Congrats on the PhD!
One important thing to note is that while this technique is more memory efficient it's also 17% slower in the setup the authors use. That's a pretty big deal, especially for pretraining.
@@tornyu Sure but it's still 17% more compute and that means 17% more money. With LLMs costing millions to train from scratch 17% more money is a big ask for pretraining.
I ran into trouble trying to Fine Tune a Swin model with LoRA. That type of model isn’t supported yet for LoRA. I wonder if it’ll be the same for GaLoRA
The authors say LoRA is about low rank weight updates, which is a bad idea since all weight updates are not always low rank. But low rank gradients are a better alternative. My question is that the only difference between weight update matrices and gradient matrices is the multiplication of learning rate i.e. weight update matrix = learning rate * gradients Isn't it? So how come weight update matrices are not always low rank, but gradient matrices are? PS: congratulations on your defense :)
Thank you! The trick is not that gradient matrices ARE low rank, but that the training process *converges* with low rank gradient matrices too, and this is what the authors also prove theoretically. Think of it this way: You want to move in Manhattan from A to B, but you can only do low-rank updates, meaning that you can only move up-down, left-right (in a subspace of the low-rank gradient update), but not also diagonally. Eventually, you can get to any B, by moving once left and once up, instead of once diagonally.
@@AICoffeeBreak thank you so much for replying :) big fan of your channel you said, "the training process converges with low rank gradient matrices" does that also mean that the training process converges with low rank weight updates?
Great to see you're following up! No, it does not mean that the training convergence is guaranteed by low-rank *weight* updates. Convergence refers to finding the set of weights that minimize the loss. If the correct updates are restricted due to low-rank weights, the optimal weights for convergence might never be found. However, similar to how movement in different directions can collectively bring you to your desired location, low-rank *gradient* updates can still lead to convergence.
