What Is Self-Supervised Learning and Why Care? 

Something that you find interesting.

thanks 🤩 haha yeah, really appreciate people noticing the details! ☺️

So happy to hear it helped 🤩

💛 Really happy to hear that! ☺️

I‘m really glad to hear it could help you! 🤩☺️

Glad you enjoyed it!

I'm so glad it was helpful!

Thank you! Really glad you enjoyed it :)

Thank you!! 🤩☺️

Awesome, thank you!

Do you have a specific paper in mind? :)

@@borismeinardus Not really. I was just wondering if ssl can be used to predict future rewards or sth...

@@binig.4591 I see. Well, in a sense, RL already is similar to SSL. The agent performs actions in it‘s environment and autonomously generates it’s own training data. I’m no expert in RL, but if I happen to learn more about it or find someone to talk to that is more educated in that direction, I will try to work on a video :)

I‘m really happy to hear that ☺️

Glad it was helpful!

(Not thinking too long about the idea) I have two thoughts: - how do you want to use the BERT encoder? In the classical encoder-decoder transformer? Or do you want to use it in between the tokenizer and the actual input into the decoder? - If you pretrain the BERT model separately from the AR decoder and just use it as a frozen module, the decoder will not instantly understand the embeddings produced by the encoder. You will have to either fine-tune the encoder and decoder together, to align their representations, or freeze them and train some sort of connection module. This connection module can be a simple linear layer, an MLP, or something more complex. Of course, there are more things to consider, but those are the basic things you might need to think about. I hope this makes sense and helps!

Glad you liked it! 😊