10L - Self-supervised learning in computer vision 

🥳🥳🥳

💪🏻💪🏻💪🏻

🔥🔥🔥

🥳🥳🥳

😍😍😍

🥳🥳🥳

Prego 😇😇😇

🤓🤓🤓

You're very welcome! 😇😇😇

😎😎😎

🤗🤗🤗

You're welcome 🤗🤗🤗

😻😻😻

You're welcome 😺😺😺

🤓🤓🤓

basically agree :)

🤓🤓🤓

😅😅😅

@@alfcnz I enjoyed this session, very good content. thanks for organizing it.

I'm aiming now at two videos per week. Haha, sorry 😅😅😅

So it will be very, very, very difficult for me to read everything, but I will try. Thank you for videos, Alfredo :)

Thanks for the comment! I agree that evaluating representations with linear classifiers is not sufficient. Like you suggest, there are many different ways to evaluate them, and each of them tests different aspects of representations. Depending on the comparison/final application, the methodology for evaluating them will change.

How to implement it? Do you have any use case? I understand the rationale but don't understand how to use something like Silhouette score over here.

​@@khushpatelmd great questions. A simplified example: consider a binary classification problem where the model outputs a single number (the truth is either 0 or 1). Suppose we want to evaluate the amount of information captured by an embedding relative to this downstream prediction task. Option 1: We could measure the mean squared error of a best-fit linear function over the embedded data. In effect, this measures how "linearly separable" our embedded data is for this classification problem. Option 2: You compute the average distance to the nearest point of a different class (for all points) minus the average distance to the nearest point of the same class. (similar to the concept behind Silhouette scores, which answer the question "how near is each point to its own cluster relative to other clusters?") Now imagine that the embedding has data placed perfectly in a separated "three stripe" pattern, where the left stripe is all 0's, the middle stripe is all 1's, and the right stripe is all 0's. The pure linear evaluation (option 1) will tell us that the embedding yields about ~66% accuracy (not so good). However, a nearness approach (option 2) would tell us that the embedding is very good and yields all nearest points in the same class (distance to other class - distance to same class >> 0). Realistically option 2 is correct here, because there is a very simple 2-hidden-node MLP that can *perfectly* capture the binary classification problem given this embedding. I realize that some people might say, "well option 2 is irrelevant if you always know you're going to use a linear last layer." But that's against the point. In general we are trying to evaluate how representative the newly learned geometry is for downstream tasks. Restricting ourselves to only linearly-good geometries for evaluation is unnecessary and can be misleading. In the end most people care how difficult it would be to take an embedding and train a new accurate model given the embedding. I assume few people will arbitrarily restrict themselves to linear models in practice.

@@tchlux Thanks a lot Thomas. This is so clearly explained by you.

🤞🏻🤞🏻🤞🏻

Automatic captions should be enabled by default. I'll check this later if and why this is not working. Thanks for the feedback. 🙏🏻🙏🏻🙏🏻

I'm in touch with RU-vid support team. They have identified the issue and are currently working on it. I'll let you know when there is any update. Thank you for your patience. 😇😇😇

@@alfcnz THANK YOU VERY MUCH!! I really appreciate the length that you're going through just to make sure the auto caption is on 😭 Once again, thank you very much!

😇😇😇

They replied and… I'm losing my patience. RU-vid support is not cooperating. I'm escalating this soon. I'm not sure what part of “feed the audio stream to your text-to-speech model” is hard to comprehend.

Yup, we can indeed use the other techniques, where the positive pairs are defined by the adjacency matrix (connectivity defined by the graph). For the question about whether FB has tried these, I'll let Adam reply. (Let me ping him.)

Hi Chuan-Chih, thanks for watching my talk! I don't know of anyone at Facebook who has applied these unsupervised methods to the problem of learning node features for graphs. The graph embeddings problem is a little different than learning unsupervised image features so I don't immediately see how these methods would apply, but I wouldn't be surprised if there was a way! In the type of unsupervised learning described in this talk, you are learning a function f that converts a high-dimensional feature vector x_i into low-dimensional semantic feature z_i. In the graph embedding setting, the nodes don't have input features - you *learn the input features* in order to approximate the adjacency matrix. There are probably ways to apply these methods if you think of the one-hot edge list (aka each node's row of the adjacency matrix) as the features, but I haven't thought about it. Maybe a better place to start is the graph neural network setting where nodes *do* have input features and you're learning a function f that combines the features over the graph neighborhood to predict some supervised labels. I haven't seen any work on unsupervised graph neural networks, but there probably is some and some of these same approaches may work well!

Indeed he does. He's been present to all my lessons! 🐻🐻🐻

Why would you want to use a transformer “within the batch dimension” (whatever this means)? Can you clarify what you're trying to do? 🤔🤔🤔

​@@alfcnz I sent an email to you. tnx

I don't have the bandwidth to reply to emails, I'm sorry. I haven't checked them in a few months by now, I think.