How AI Image Generators Work (Stable Diffusion / Dall-E) - Computerphile 

If it needs other people's intellectual property to work, then it is a legal concern for those whose work it being exploited without consent. This is why Getty images is suing, and why Adobe is building their AI Firefly off of licensed work, and making guidelines to compensate those who stock are being used by their AI.

@@Isaac-wr8et Why do ignorant people like you comment? This is literally an explanation of how these types of AI work, no different than explaining other AI models. Nothing about this is abstract mumbo jumbo BS, you just don't understand it, lmao.

@@Isaac-wr8et salty credit obsessed artist spotted

@@youssefabusamra3142 the way I see it, people aren't happy when their private information or content is downloaded/stolen without permission by the government or AI. The source that these AIs are using too "learn", is doing just that.

@@Isaac-wr8et "the government or AI" my brother in christ they uploaded these photos for public recognition agreeing to the site's terms and conditions what the AI is doing is the equivalent of "looking" at these photos and recognizing features and patterns. If you're so dead set on the stealing narrative then pick a real painting that was used for training and try to "steal" it by recreating it with prompts

Great point. Yes I skipped over this mainly for the sake of the length of the video. This also explains the slightly odd brown noise we see in the video, which is actually a low noise latent passed back through the VAE decoder.

@@michaelpound9891 if you tell AI to run for zero steps you can look at the noise.

I once used an overtrained network to store several images, then manipulated the low dimensional part to give some trippy image tweening (a few inputs reperesenting this is image 1,2,3,... etc. and then getting the in-between images). Unfortunately very low resolution, and took ages, guess a web-browser isn't the place for running neural nets...

@@threeMetreJim That’s always a fascinating experiment with VAEs. Encode two items to two latent points, take the midpoint of the two latent points, and then decode that latent midpoint to see what the resulting item is. I tried this with music and it was interesting to hear a transition from Beethoven to Schubert.

fatter and older

☝☝

There r videos that do that. Also there r alot of papers explaining the system in detail.

@@nevokrien95 Can you provide links to those videos please?

@@rayankhan12 sure i will send it in parts here is some code in pytorch (i personally know only tensorflow but i still got the gist of how i would go about doing it)

@@rayankhan12 this is abit mathematicly involved. Key point it shows is that the limiting behivior is not what makes these work. They r simply autoencoders with some extra whistles

@@thebirdhasbeencharged I don't understand why people answer questions they don't know the answer to. He's asking how the diffusion model which starts from purely random noise, uses the text embedding generated from clip to guide the diffusion. "A.I. is just fancy pattern matching" is about as unhelpful an answer as you could imagine.

I'm a little unclear on that myself, but the best understanding I can manage is that the CLIP (language model) embeddings must be included in the diffusion network's training. So while it's learning how to predict the noise on a picture of, say, a bunny, it's also given the text description of the bunny, which means it's learning how the descriptions affect the noise at the same time as it's learning how the underlying picture does. I think. As I said, not 100% clear on that, so don't take my word for it 😅

They asked humans, "Is this a frog? Yes or no." They took that data to develop an AI that could be asked, "Is this a frog? Yes or no." They did the same with "stilts". They did the same with "on". The difference being that they used a variety of known "objects" to determine whether they were "on" something or not. They also probably classified "on" as a verb, rather than a noun. This makes it a union of two objects. A union associated with "proximity" or something like that. Like he said at the end, they need an intact frog and intact stilts as a requirement in the "frog on stlits" image. So they look like "frog feet with proximity to stilt objects" etc. I would assume human objects on stilts strongly guided their classification of frog objects on stilts.

@@dialecticalmonist3405 I guess that's a decent high-level explanation, but I would clarify that it's actually not using any kind of classification system. Classifiers are an entirely different family of neural networks. The guidance system used here is a transformer-based language model, which is less like asking "is this a frog (y/n)?" and more like asking "here's an image, describe what it is".

@@generichuman_ haha I'm assuming they deleted their unhelpful message, as I don't see it. Great call out. =)

It's strikingly similar, except a sculptor starts with a goal image in mind, but AI image generation doesn't; it just has general "knowledge" of "associations" between the words of the prompt and parts of images.

Yep! It's a bit like apophenia, like looking at random clouds and seeing coherent shapes in them, but with some priming about what you "should" be seeing :)

"I saw the angel in the marble and carved until I set him free. ” - Michelangelo

No, that's not how it works at all. His explanation is highly inaccurate and misleading, which is throwing you off. Try reading the actual papers on the subject, or going through the code.

No, that's not how it works at all. His explanation is highly inaccurate and misleading, which is throwing you off. Try reading the actual papers on the subject, or going through the code.

Don't worry friend, just do what I do: 1. Assume everything on the internet is fake (including other people) 2. Retreat from society into a cave 3. Starve to death It's kinda like Plato's Cave, but in reverse. Anyway, it's a pretty solid solution 🙂

In fact it is already being done for artists, the LAION database should not be used for commercial use, and many IAs are actually using it in that way, not to mention that this database has images protected by copyright, so sell or publishing these resulting images is a clear violation of copyright

@@Hagaren333 Begun, the data wars have

We are getting ever closer to living in a dystopian Cyberpunk universe.

Just abolish capitalism, then.

And with the intention to guess how much noise a picture would have only by text description as an input haha

Right, it sounds like a completely non-intuitive way of going about it, and yet, that's what ended up working. They must have iterated on a gazillion different ideas before they landed on this one.

The idea of adding something, do a transformation and substract to get only the transformation of the data or the something is actually quite common in math and control theory. Real hit from AI was to get "the" transformation from data, something and output into an algorithm. This general function of transformation is what allows the image generation. We give it data or something that is slightly off, and amplify the error by the transformation. 😅 it's some weird combination of the chicken egg paradox and a rock paper scissor but with data, something, algorithm and output.

It's why science works better by being public and not subject to short/mid-term revenue. There were already teams training models to undo noise, there was already GPT to interpret text, there was already a database of millions of text-to-image pairings and there were already models trying to feed text to image-based neural networks. Kinda like smartphones, all it took is someone putting the pieces in the right order for the right purpose to make something more useful than the sum of its parts.

@@Alex-ye8qp To me this is the most bizarre part. How can a network even be trained to do that? So utterly bizarre to me. You're telling me "green cow grazing on mars" has a deterministic noise profile??

I watched Dr Mike Pound's video on Convolutional Neural Networks when it first came out and it got me into machine learning. Now I'm doing undergrad computer vision research with CNNs. It's honestly kind of crazy to think about how much this channel has affected my life.

@@emmafountain2059 Well played! Are you enjoying yourself doing the computer vision research?

@@emmafountain2059 yup. I started watching this channel when I was a bored IT audit intern who hated doing work papers. I literally sat in the bathroom or went on walks outside and just watched. Now I’m a pentester. The reach of high quality RU-vid channels like Computerphile are hard to measure but I don’t think I’m unique.

@@emmafountain2059 That's amazing! Wish you the best!

This is basically what CLIP does. CLIP learns from a massive amount of image-description pairs using GPT-style (Transformers) encoding so that it can map texts and images. CLIP data are not classification labels. Then the difference between the texts and the generated images can be calculated and minimized.

Key word is embeddings. Initial feature space of text has two bad properties: it has big dimensionality (each token is it's own dimension essentially) and sparsity. By using Transformers you compress representation of this object in more compact and dense form, so it's easier to work with.

yes they used image-text pairs dataset (LAION) to train the guided diffusion model

@@tristanstevens6162 That dataset accounts for understanding text too? Like, if you have sets of images of cats and sets of images of frogs and very few sets of different animals being "fused" (e.g. maybe one capybara that looks like a dog), how would the neural network get to the interpretation of what a cat-frog means as an image? Is LAION that big? Or does the GPT neural network somehow bridge that gap?

@@ekki1993 a lot of the job of "understanding" is being done by the embedding network, which was trained on a very large corpus of words. So while the training set for stable diffusion might not have any examples of a frog-bunny fusion, CLIP is able to take the phrase "frog-bunny fusion" and transform it into a vector that encodes something about the meaning of the phrase. Stable diffusion was trained conditioned on this embedding, so it generally has learned to take concepts from this embedding and include them in the image. The hope is that stable diffusion is able to generalize across all concepts that can be represented by the embedding, so that even if it hasn't seen this specific thing before, it has seen similar stuff and is able to still produce a reasonable image that matches the concepts in the embedding.

@@DontThinkSo11 Thanks for the answer!

@@ekki1993 It's possible for all/any of the components to contribute to the result working - like, even if it hasn't seen any pictures by an artist x, "style of artist x" may still work as a prompt if it's seen text describing them. This is an issue for artists who've been complaining that image generators can reproduce their style in some ways. It means that nothing can be done to prevent this; a base model might still understand them if the images aren't in the set. "Worse", fine-tuning seems to work well enough that you can add in new concepts and styles at home even if they're not in the model originally.

Same, I understood the noise subtraction bit, but I can't quite understand how the subtraction can lead to a picture of a frog, was the IA trained with "words vs images"? So it can relate what a frog would look like. Also, what the initial input picture (12:30) looks like? Is it just random generated noise?

​@@skirtsonsale yes a labeled dataset (image-text pairs, LAION dataset) was used to train the network. That is why it is called guided diffusion. The text guides the diffusion process not to a random image, but conditioned on the text (again the pairs were used for training). During training, it sample randoms noise from a random t according to the noise schedule (such that during training it is learned for all t). The input image on 12:30 is such image corrupted using noise from a random t. So somewhere between noise and an image.

@@skirtsonsale The concept of transformation in graphics will help you understand this.

@@tristanstevens6162 But why doesn't that just produce some incohesive amalgamation of the training data? How does it know to specifically put the bunny ears on the frog's head? Is that where the magic of having a large amount training data comes in, in that it better understands the correlation between the label and the image?

@@jonatansexdoer96 Yep. CLIP is the language model used in these, and it's seen enough examples of things labeled "bunny" that look different from each other to abstract the idea of where bunny ears are located in any given underlying image.

Gaming laptop with a 3060 (6GB) here working great for Stable Diffusion, I'm using the Automatic1111 web UI distro bundle which made setup incredibly easy. I'm still learning how to use it to get "what I mean" results, but it is quite amazing.

I've ran it on a 2060 fine, though I think its the lowest supported card

I'm using a 3060 Ti. But inference is *always* more efficient than training, so my hardware can't handle Dreambooth, and certainly would never come close to handling a full initial training.

The biggest limit is vram not gpu power as much, on a rtx 3000 series gpu 2-10s for 512^2 20 steps and 15-60 secs for 1024^2 image. Slow down starts due to vram not being able to hold everything. So you can get silly things like the 3060 being better then a 3080ti for some uses of SD.

@@asdf30111 yeah VRAM is always the main issue with AI. Gotta store huge matrices in memory. I wonder if, going forward, hardware providers will bump up VRAM on their high end consumer cards due to increasing consumer demand... Or perhaps decently sized tensor cores will become more commonplace

perfect explanations

nice

RU-vid has a function of automatic subtitles.

Try turning up the CFG scale. In general, higher values for that make the result match your prompt more closely, though it also reduces variation. If you're using the WebUI version, it also supports term emphasis, so you can try putting the word "stilts" in ((double parentheses)) to indicate that they should have higher weight in the result.

The network is basically implicitly imbued with this understanding through the training process: For the training process of these conditional models, you take a labeled dataset {(image, label)}, and the label is a natural-language description text *describing that image* . For a particular data sample, the network will then see a noisy version of this image, along with the embedding of the description of the original clean image. The training loss then rewards the network for reproducing this image given these two bits of information, and punishes it for reproducing *any* other image. I hope it's clear that the training will therefore steer the network into a direction of 'understanding' these descriptions in some abstract way. That this understanding of the input text is *abstract* is actually what you hope allows the model to extrapolate, so it can generate new images conditioned on new texts not seen during training. That this extrapolation works ridiculously well is one of the amazing things about these models.