Also, are we using latent space as gradients here, as we are subtracting gradients from the latent, which we typically do from weights in conventional NN ?
How do we decide the scaling factor in VAE part i.e. 0.18215, any hint on how to decide it ? I did try changing and could see the different output, but what's a good way to choose ?
This is useful but I wish you went into more detail here and there. Is some CLIP or similar model included in the stable diffusion implementation? If so, are precomputed weights of the CLIP model used to calculate noise_prediction in each step? I.e. we pass the current noisy image (in a latent space) and the text embedding to CLIP and then calculate the gradient for each voxel of the image so that something (semantic similarity?) is maximized? I wish you would say what happens during training of the mode and what then happens during inference :).
I'm surprised how... complexity(?) raised up. It's second day and I only on 4th minute, spent 30 minutes debugging my coding-along session (I wrote rand_like instead of randn_like and my parrot photo went green instead of grambled)