How AI 'Understands' Images (CLIP) - Computerphile 

As people have correctly noted: When I talk about the way we train at 9:50, I should say we maximise the similarity on the diagonal, not the distance :) Brain failed me!

thank you for "if you want to unlock your face with a phone".. i needed that in my life

The legend has it that the series will only end when the last sheet of continuous printing paper has been written on.

This guy is one of the best teachers I have ever seen.

Dr. Pound's videos are on another level! He explains things with such passion and such clarity rarely found on the web! Cheers

Thanks for taking us to Pound town. Great explanation!

9:45 - wasn't it supposed to be "minimize the distance on diagonal, maximize elsewhere"?

That cat got progressively more turtle-like with each drawing.

The biggest problem with unlocking a face with your phone is that you'll laugh too hard to hear the video for a minute or so.

"There's a lot of stuff on the internet, not all of it good, I should add" - Dr Mike Pound, 2024

These guys are so watchable, and somehow they make an inherently inaccessible subject interesting and easy to follow.

The man, the myth, the legend, Dr. Pound. The best lecturer on Computerphile.

Excellent. Could listen to him all day and even understand stuff.

A very important bit that was skipped over is how you get an LLM to talk about an image (multimodal LLM)! After you got your embedding from the vision encoder you train a simple projection layer that aligns the image embedding with the semantic space of the LLM. You train the projection layer so that the embedding of the vision encoder produces the desired text output describing the image (and or executing the instructions in the image+prompt). You basically project the "thoughts" of the part that sees (the vision encoder) into the part that speaks (the massive LLM).

6:09 "There isn't red cats" Mike is hilarious and a great teacher lol

I'm a simple guy. I see a Mike Pound video, I click

I like how he just sit in an office chair and talk. Simple but high quality talk

I love these encoder models. And I have seen these methods implemented in practice, usually as part of a recommender system handling unstructured freetext queries. Embeddings are so cool.

Love seeing AI problems explained on fanfold paper. Classy!

Awesome explanation

Computerphile and Dr. Pound ♥️✨ I've been learning AI myself these past few months so this is just wonderful. Thanks a ton! :)

i just feel 10 times smarter after watching any computerphile video

The way he explains is very interesting.

Really nice to explanation 👍👍

This video was super helpful. I just wish it came out with the stable diffusion video. Was quite confused how we went from text to meaning.

Great video!

At 9:30, doesn't a distance of zero mean the image and caption have the same "meaning", therefore, shouldn't we want to minimize the diagonal, and maximize the rest?

Thanks Mike, nice clear explanation. You can still get that printer paper!? Haven't seen that since my Dad worked as a mainframe engineer for ICL in the 80s!

i'm an it-guy & programmer, but this is kinda hard to understand. thanks for the video, gives a little idea about the concepts!

Brilliant mike !

Mike is the best

When they say "high dimensional" in the vector context, I like to imagine it like an RPG character stat sheet, as each independent stat on that sheet can be considered its own dimension.

Just writing this to crystallize my understanding: (and for others to check me for accuracy) So by circumventing the idea of trying to instill “true” understanding (which is a hard incompatibility in this context, due to our semantics); On a high level it’s substituting case specific discrepancies (like how a digital image is made of pixels, so only pixel related properties are important: like color and position) and filtering against them, because it happens to be easier to tell what something isn’t than what it is in this case (like there are WAAAY more cases where a random group of pixels isn’t an image of a cat, so your sample size for correction is also WAAY bigger.) And if you control for the specific property that disqualifies the entity (in this case, of the medium: discrete discrepancies), as he stated with the “ ‘predisposed noise’ subtraction to recreate a clean image’“ training, you can be even more efficient and effective by starting with already relevant cases. Once again because a smattering of colors is not a cat so it’s easier to go ahead and assume your images will already be in some assortment of colors similar to a cat to train on versus the near infinite combinations of random color pixel images. And then in terms of the issue of accuracy through specificity versus scalability, it was just easier to use the huge sample size as a tool to approximate accuracy between the embedded images and texts because as a sample size increases, precision also roughly increases given a rule, (in crude terms). And that it’s also a way to circumvent “ mass hard coding” associations to approximate “meaning” because the system doesn’t even have to deal directly with the user inputs in the first place, just their association value within the embedded bank. I think that’s a clever use of the properties of a system as limitations to solve for our human “black box” results. Because the two methods, organic and mathematical, converge due to a common factor: The fact that digital images in terms of relevance to people are also useful approximations, because we literally can only care about how close an “image” is to something we know, not if it actually is or not, which is why we don’t get tripped up over individual pixels in determining the shape of a cat in the average Google search. So in the same way by relying on pixel resolution and accuracy as variables you can quantify the properties so a computer can calculate a useable result. That’s so cool!

4:35 "There is a lot of stuff on the internet, not all of it good." Today I learned 😀 6:05 I enjoyed that you mentioned the issues of red/black cats and the problem of cat-egorization Video was helpful, explained well, thanks

14:36 Why can't you just train a network that would decode the vector in the embedded space back into text (being either fixed sized or using a recurrent neural network)? Wouldn't it be as simple as training a decoder and encoder in parallel and using the text input of the encoder as the expected output in the decoder?

For using CLIP as a classifier: couldn't you train a decoder network at the same time as you train CLIP, such that you now have a network that can take image embeddings and produce semantically similar text, i.e. captions? That way you don't have to guess-and-check every class one-by-one? Anyway, I can't believe CLIP has only existed for 3 years... despite the accelerating pace of AI progress, we really are still in the nascent stages of generalized generative AI, aren't we?

Can AI say "I don't know what I'm looking at"? Is there a limit to how much it can recognize parts of an image?

Amazing

This video satisfies my ai scepticism.

@dr Pound: sorry if this is off topic here but, i wonder if the problem of hallucinations in AI comes from us not treating the 'i don't know what I'm looking at ' answer of a model, as a very negative outcome? If it was treated by us as a valid neutral answer, could it reduce the rate if hallucinations?

i was curious, when you are showing the paper from above, are you transforming the original footage?

Still have to manually generate the sentences for the embedding. It’s just a convoluted way to let a computer generate its own lookup table.

3:55 As someone born in the former GDR, I find it cute to label a Trabi as "a car"...

Please explain 9:2, do you mean that the embeddings laying on the diagonal will receive higher score?

Personally I woulda just did the colour comparison by putting the 24bit RGB integer colour into a double (the 64bit fpn type) and divided one by the other. If the result is greater than 0.01 or less than -0.01 then they're not close enough to deem the same overall colour and thus not part of the same facing of a shape. **Edit:** When searching for images it might be better to use simple line path (both a 2d and 3d one) matching the given text of what to search for and compare the shapes identified in the images to those 2 paths. If at least 20% of the line paths matches a shape in the image set then it likely contains that what was searching for. Similarly when generating images the line paths should then traced for producing each image then layered on to one image. Finally for identifying shapes in a given image you just iterate through all stored line paths. I believe this is how our brains conceptualise shapes in the 1st place given how our brains have nowhere to draw shapes to compare to. Instead they just have connections between...cells? neurons? Someone will correct me. Anyways they just have connections between what are effectively physical functions that equate to something like this in C: int neuron( float connections[CHAR_BIT * sizeof(uint)] ); Which tells me the same subshapes share neurons for comparisons which means a bigger shape will likely be just something initial nueron to visit, how many neurons to vist, and what angle to direct the path at to identify the next neuron to visit. In other words every subshape would be able to revisit a previous subshapes neruon/function. There might be an extra value or 2 but I'm no neural expert so a rough guess should be accurate enough to get the ball rolling.

approx how many samples do i need when i just want to do image categorisation (but with multiple categories per image)

Computerphile, This is great! I liked it and subscribed!

So the Visio-Text encoder is dependent on the quality of the captioned photo dataset? If so, where do you find quality datasets ?

At 9:48, did Mike use 'maximize' and 'minimize' wrongly? The distances in the diagonal should be minimal right? EDIT: I saw Mike's pinned comment only after posting this :P

6:30 if for LLM you have nodes of meaning then you could look fof "nodes of meaning" in description and make classes based on those "nodes", if you are able to represent every language based on same "nodes of meaning" that is even better to translate text from one language to another then average translator that is not LLM, then you should be able to use it also for clasification.

A good edging session (for my brain)

When will wee see the multimodal LLMs be able to answer a question with a generated image. Could be how do you wire an electric socket, and it would generate either a diagram or illustration of the wire colors and position. Should be able to do this but it can't yet. Next would be a functional use of SORA rendering a video how you install a starter motor in a Honda.

Quick question from someone outside computer science: Does the model actually instill "meaning" into the embedded space? What I mean is: Is the Angel between "black car" and "Red car" smaller than "black car" and "bus" and that is smaller than "black car" and "tree"?

there is stray audio in the background that you can faintly hear at 0:05

I didn't read the cited paper, but if I understood correctly, the 5 billion images need to be labeled for the training step?

Is there a specific reason why this process would have to be single-directional? There doesn't seem to be much difference in principle to an autoencoder. I assume this isn't about whether it would work or not, but rather about approximating this behavior to navigate around impossible amounts of training, or am I mistaken in this assumption?

How do you go from embedings to text of something never been see. before?

Once you have a clip, can't you train a diffusion on pure images, just by putting an image into clip, and training the diffusion to output the same image?

so the length of the clip array training the defusion would have 400 million entries ? so it defines a "corner" of the space we have spanned by the 400 million fotos and foto descriptions ?

Heyo, a Pound dogette here!

There is a paper on The Gradient about reverting embeddings to text ("Do text embeddings perfectly encode text?")

Clip is fantastic. It can be used as a 'zero-shot' classifier. It's both effective and easy to use.

Poundtown.. sweet...

9:48 I didn't quite get how do similar embeddings end up close to each other if we maximize the distances to all other embeddings in the batch? Wouldn't two images of dogs in the same batch will be pulled further away just like an image of a dog and a cat would? Explain like Dr. Pound please.

Oh look, a Mike!

So, if we want to break AI, we just have to pollute the internet with a couple billion pictures of red cats with the caption “blue dog”.

Stable Diffusion 3 = potential topic Optimum workflow strategies using Control Nets, LORAS, VEAs etc....?

How can AI determine the "importance" of parts of an image? why would it output "people in front of boat" instead of "boat behind people" or "boat surrounded by people"? Or maybe the image is a grid of square white cells. One cell then get its color progressively darken to black. Would the AI describe these transitioning images differently?

This is as simple as it’s ingenious and it wouldn’t be possible without the internet being what it is.

Awesome! I always wondered how the hell does the AI “gets” that an image is made with a certain type of lens or film stock. Or how the hell AI generates objects that were never filmed in a way, say, The Matrix filmed on fisheye and Panavision in the 1950s.

Amazing. Imagine the day when AI is able to detect digital image steganography. Not by vision primarily, but by bit inspection.... iterating over the bytes and spitting out the hidden data. I think we're still years away from that though.

Great explainer. Appreciated. I hope someone will cover AI music next

I'm curios about a thing. If you have a bunch of millions of photos of cats and dogs and they are also correctly labeled (with descriptions) but all these photos have the cats and dogs in the bottom half of the image, will the transformer be able to correctly classify them after training if they are put in the upper half of the image? (or images are rotated, color changed, filtered, etc..).

Please make a Playlist only about GenAI or a separate AIphile channel. I care only about genAI.

5.02 Never seen a "boat wearing a red jumper" before lol

I'm an artist and I'm very worried about my art being used to train an AI model. What can I do to prevent that? Any tips?

One day, we'll give these models some high resolution images and comprehensive explanations and their minds will be blown! It's astonishing how good even a basic perceptron can be given 28x28 pixel images!

10:37 "If you want to unlock a face with your phone". Ha ha ha!

9:48 Been a while since I did machine learning class... Anyone care to tell me why I'm thinking of PCA? What's the connection?

But confusing to say that you want to maximise the distances on the diagonal. Of course you can define things however you want but usually you'd say you want to maximise the cosine similarity and thus minimise the cosine distance on the diagonal.

Do any of you ever get this weird feeling that you need to buy Mike a beer? Or perhaps, a substantial yet unknown factor of beers?

Can't you do something analogous to stable diffusion for text classification? Get the image embedding, and then start with random noisy text, and iteratively refine it in the direction of the image's embedding to get a progressively more accurate description of the image.

I wish I could hear Professor Brailsford's thoughts on AI these days man

Question 🤚 Up until recently all images of a cat on internet were photos of real cars and the system could use them in training. But now more and more cat images are AI generated. If future systems use generated images in training it will be like a blind leading a blind. More and more distortion will be added. Or? Can that be avoided?

I thought diffusion models had VAE and not ViT Correct me if I m wrong

Difficult to see how convergence can be ensured. Lots of very different functions can be closely mapped over certain controlled ranges, but then are wildly different outside those ranges. What I have missed in many AI discussions is these concepts of validity matching and range identities to ensure that there's some degree of controlled convergence. Maybe this is just a human fear of the unknown.

Mike explains very well, however it is very noticeable that the concept is not easily explained in 20 minutes. There a are a lot of cliffhangers and each step in the process requires its own iteration of 20 min video, probably at least 2 levels deep to really understand what's going on.

Each time I watch one of these videos, I could might as well watch Apocalypto without subtitles. My brain is not made for this 🤓

The answer given for the "we feed the embedding into the denoise process" still felt a little hand-wavey to me as someone who would like to understand better, but overall good video.

13:54 you actually probably can, at least to an extent there’s been some recent research on the idea of going backwards from embeddings to text, maybe look at the paper “Text Embeddings Reveal (Almost) As Much As Text” (Morris et al) the same thing has been done with images from a CNN, see “Inverting Visual Representations with Convolutional Networks” (Dosovitsky et al) neither of these are with CLIP models so maybe future research? (not that it’d produce better images than a diffusion model)

The exchange at 6:50 made me laugh out loud and I had to show that part of the video to the people around me😆😆

'and we want an image of foggonstilz' me: wat 'we want to pass the text of farngunstills' me: u wot m8

"a mike"

Where did they get those images from? Any copyright infringement?

Why is there always so much mess and clutter in the background of these videos? Do you film them in abandoned buildings?

Crypto Bull run is making waves everywhere and I have no idea on how it works. What is the best step to get started please,,

AI art is great.

At 16:20 the 'cat' looks more like a shower head.

[15:00] It's supposed to have a tail

Stable Diffusion needs a CEO BTW ....just saying ... 😅