This was a really good video. It really helped me understand this diffusion concept that I didn't know about. Your videos are underrated, but I have no doubt they will gain traction over time.
Congratulations on the video. I've always had doubts about whether Stable Diffusion is the same thing as Latent Diffusion. Now with your explanation I understand that they are the same thing.
Thank you so much Gabriel! I wanted to understand the intuition behind Latent Diffusion, and watching your video saved me tons of time from actually reading through the paper.
These videos are criminally underrated! This one, ViT, Attention and LoRA have helped me so much with my learning! As a Compsci student majoring in AI, going from learning lectures and reading books, to reading, understanding and implementing the actual papers is a big leap, and you've made that leap a lot more simpler and digestible. Thank you so much, please never stop this series!
Thank you! Glad you're finding my videos helpful. Hopping from lectures to papers is difficult as the style is very different and I'm happy to see my videos are helping with that. I'm also learning as I create each video, so not planning on stopping this series any time soon!
Thank you for the video, it is the first video I found about training AE in LDMS, and I think this part is the hard part to understand the whole model, thanks for your explanation, it is very easy to understand. One thing I would like to add is that the AE in the paper is based on VQ-VAE, so L_rec uses perceptual loss and L_adv is a patch-based adversarial objective. Anyway, I hope you will continue to work on this series!
Hello, thanks for the explanations! just a few words on the greek letters. its "psi" not "phi" here, and the "rho"_theta you mention is actually a "tau"
Does the text emb serve as a label in this model? For example, i put a pic of peguin and describe it "peguin". The model learns to match pic and text and reduce loss
Thank you very much for an amazing explanation! one question tho, on minute 22:00 when explaining the Autoencoder loss function, you activate the log function on the output of the discriminator, isn't that a bit problematic ( since log is not defined in the case that the discriminator predicts "fake")?
You're right! When the discriminator predicts "fake" (zero), then this value would be undefined. However, it's very unlikely for the discriminator to pick an extreme of 100% real or 100% fake. In the case of the discriminator predicting 100% fake (maybe due to quantization rounding), resulting in the log of 0, we can just add a small value such as 1e-5 to prevent the value from being undefined.
How do you know the "epsilon" ? Meaning for the second training step, where you are doing MSE ( noise, predicted noise), how do you know the "noise" before hand? Is it coming through a function "P" ? Also what does it mean to train the diffusion layers? Are diffusion layers also like convolution?
Our noisy data, x_t, is computed by a function like (for simplification) x_t = t*x_0 + (1-t)*ε For the ε value, we sample from a Gaussian which is the same size as our image. If our image was 3x256x256, we would sample 3*256*256 values from a Gaussian. This is the tensor that is both added to out data to create x_t and is the tensor being compared in the MSE loss. For each training step and for each image, we resample the epsilon value so the model learns to model the Gaussian prior distribution. By train diffusion layers, we just train a U-net. It's very arbitrary how you construct the U-net, but generally, this is made of resnet blocks with cross-attention conditioning. Hope this helps!
Glad you're enjoying the videos! I haven't heard of RWKV models before, but it looks pretty interesting. I'm going to a look at them and I think I'll be able to create a video on them!
Was working on reading the code and formulating a video. I didn't realize they published a paper recently! Yannic made a video on it here which is much better than what I could do! Check it out here: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-x8pW19wKfXQ.html
Really nice Video. I have a doubt about the latent loss, 1) Are they trying to fake the encoder-decoder part and the real input real?? 2) if the above assumption is true then they need to put plus sign for 2nd term in the loss function (i.e output of discriminator with encoder-decoder output as input)
You can actually have the model predict either the data or noise and it mathematically doesn't matter. Since the data x_t is an interpolation at any time t, and we know the interpolating function, we can equivalently get the predicted data if the model predicts the noise. Same thing with the model predicting the data instead, we can still extract the predicted noise. In practice, the model has to learn a different function, so a model that predicts noise may somehow do better than a model that predicts data. All the papers I've seen have the model predict the noise rather than the data, so I'm guessing that's the better way to go, or maybe it doesn't really matter. Would be something to experiment with if implementing your own DM.
