Thanks Alfredo and Jiachen the video really helps people understand the VICreg I've just got a perhaps very simple question to ask about VICReg and JEPA. I've read "A Path Towards Autonomous Machine Intelligence" by Prof. Yann LeCun and saw that VICReg is introduced in the section of Joint Embedding Predictive Architecture (JEPA). Since JEPA defined in the paper contains latent variable z, while VICReg seems not to have latent variable z involved, may I confirm whether latent variable z is necessary in JEPA framework? Or it is just an optional component?
How do you measure information content, like from the backbone embedding and projector embedding when you said projector removed some information ? Thanks
Depends on the amount and the type of data you have. Say you want to work with radiographs. Pretraining on ImageNet is not really going to work, since the data is rather different. Then, you may want to pretrain on annotated radiographs, but those annotations are very expensive. So, in these cases SSL is going to be superior.
@@alfcnz Thanks. Actually I understand that when annotated dataset is scarce SSL is the best way to pre-train our network. Ok say for example I want to train a person detection model, and first I want to pre-train my backbone network. Let us assume, we have 1000 annotated person dataset. Now, assume I prepare two different backbone models, 1) pre-train on the 1000 annotated person dataset 2) pre-train using SSL on 1000 person data (with out using annotation). My assumption is that SSL based trained backbone will provide better generalization (I remember seeing something like the grad-cam or heatmap from the DINO by facebook where they showed SSL based training provided high quality pre-training). I might be wrong. In summary, my query is that, given similar amount of data, will SSL be better that SL?
