One year after I last completed this course, I’m revisiting it again to watch specific portions to refresh my understanding. These lectures have been immensely helpful both professionally and academically. Prof YLC tutorial on energy based learning (2006) provided me with an entirely new perspective on why certain loss functions work well and why some don’t.
At 1:03:00 I don't understand how z and z_bar can be made similar using the D function. Are they distributions and you're applying KL divergence? I mean if z is a latent variable, how is it's value know for comparison?
When using methods like Contrastive Divergence (or GANs, for that matters), how can we make sure that the low energy points founds aren't actually legit?
We don't. In the end, the learning will stabilize when the low-energy contrastive points are on the data manifold: the loss will push down and push up on them equally hard.
I have a question - when we are doing transfer learning using a model trained on imagenet(1000 classes) and using that model on a smaller multiclass classification problem, unless we are careful prolonged training with relaxation of early layers will distort the features learnt for our new task. Now, if we intend to use these features for other purpose like interpretability then we will get distorted features which are becoming degenerate as we keep training. How do we circumvent this problem besides using a very small learning rate and freezing the earlier layers?
A simple trick is to use an L2 regularizer that attracts the weight vector W towards the solution obtained from ImageNet pre-training W0: R(W) = alpha*||W-W0||^2 The system will then try to adapt to the new dataset while minimizing the deviation from the initial configuration. A slightly refined version is to make alpha a diagonal matrix, so that lower layers are more regularized than higher layers.
I am following all the lectures...really enjoyed every video, I lost many time in this video.. is there any alternative source to learn the same content? Thank you, sir, for uploading the course and open-sourcing the notebooks
All the videos are transcribed in English and translated in several languages. Have you checked the video description here at all? There's *plenty* of material I've been uploading and have linked there.
I have a question - can we take the produced filters or kernels of AE ( auto-encoder ) or VAE and use them as a filter in ConvNet and start training instead of training from scratch or from pre-trained weights like imagnet ?
Of course, training an AE is one possiblity to learn feature extractors useful for later tasks. Recently, we've seen that joint embedding techniques are also effective strategies to perform self-learning.
