🤔 Ok, but what IS ControlNet?

Подписаться 14 тыс.

Просмотров 45 тыс.

50% 1

A high level overview of the excellent ControlNet research paper which has been used recently to grant stable diffusion users highly fine grained control over the image generation process.
Discord: / discord
======= Links =======
ControNet paper: arxiv.org/abs/...
Huggingface ControlNet models: huggingface.co...
Huggingface Depth-2-img models: huggingface.co...
Controlnet github: github.com/lll...
======= Music =======
From RU-vid Audio Library:
Organic Guitar House - Dyalla
======= Media =======
Thumbnail Source: / controlnet_is_awesome_...
Anime Motion Tracking: • Animation with Control...
Carpark Lady: / controlnet_ebsynth

Наука

Опубликовано:

2 окт 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 141

@studiokaralis3622 Год назад

as someone who is not actually involved with machine learning, I really enjoy your videos. They ensure that I stay up to date and fundamentally understand the subject. Thank you very much & keep going!

@michaelleue7594 Год назад

The fourth little pig said, "I'm going to build my house out of A100s and the big bad wolf won't ever get in because then he wouldn't be able to train his image models anymore." The big bad wolf said, "I'll huff and I'll puff and I'll just use 4090s to train hypernetworks instead, and I'll blow your house down!" Took him a while, though.

@bundo-san Год назад

Laughed at 20:30 😆. Thanks for the explanation, I've always wondered how ControlNet worked like black magic!

@GerwaldJensRadsma Год назад

What is koiboi? How does koiboi work? Why should we care? A: because he makes awesome videos about the underlying stuff not many talk about... Tnx!

@lewingtonn Год назад

Lel

@autonomousreviews2521 Год назад

Fantastic as usual :) Thank you for that delightful detailed explanation!

@guavaberry Год назад

controlnet's outta control is what it is lol you blink and in an instant there's 5 new updates and diff applications of it

@lewingtonn Год назад

yeah I'm like... way behind ngl

@guavaberry Год назад

@@lewingtonn its so hard! lol i literally you cant keep up cause I dont have a fast computer (mac m1 mbpro with 8gb ram) but videos like yours do help ppl like me since we cant do as much testing as we'd like etc

@FunwithBlender Год назад

oh boy, always koi great video

@mrrevengaa 2 месяца назад

Amazing stuff, great explanation way better than many tutorials. Thank you for your work and time.

@4.0.4 Год назад

The fact it was trained in under a week on a single consumer-grade GPU is massive. I wonder, could ControlNet be applied to LLMs like Llama/Alpaca/etc?

@shin-ishikiri-no Год назад

You are a little bit of a totalitarian (based on another video I saw), but you're good at making educational videos on this topic. Kudos.

@DJVARAO Год назад

Thank you for the explanation. I wonder is something like this can be done with a LORA type of training. I mean, maybe we can do customized control nets? And maybe eg subject training can be more efficiently done with higher accuracy?

@user-gu9vf3cc4u Год назад

Great video as always, just if you can be a bit more dramatic on camera that would be perfect.

@lewingtonn Год назад

-_-

@Bzoargh Год назад

A really big thank you to you for all the effort you put into this. It is common to learn about the "how 2" but very rare to learn the "how" as well. Since AI and its influences will change SO much in the next few years and beyond, I am very glad you included a very, very important part of this powerful technology: basic understanding. It allows others to make appropriate assessments. And since AI can be used to change, reshape, or distort just about anything, it's hugely important, and (unfortunately) often overlooked, to also understand how all of this is possible. The video taught me a lot. I enjoy it, it's interesting, and most of all it's responsible the way you do it. So thank you! :D

@8pm662 Год назад

Great work! But we as community really should team up and buy him a more decent mic 😁

@lewingtonn Год назад

lol I'll order a good one off amazon next time I do a shop

@Jianju69 Год назад

A great mid-level explanation. Very helpful for those versed in machine-learning to bridge the gap.

@ThePixelkd Год назад

Thank you! You are an fn rockstar

@ShawnFumo Год назад

Thanks for the video! Another cool aspect besides being able to apply multiple controlnet models at the same time, is you can apply them to variations of the original locked model. There’s all these merges and mixes of 1.5, but they are close enough to the original that these external networks work with them as well.

@ShawnFumo Год назад

With all this modularity, you can actually have the case of using say Realistic Vision (which lists 10 other variations of SD1.5 that it merged together), put three modes of ControlNet on that, as well as Latent Couple to split the image into regions, and Composable LoRA so you can have different LoRAs apply in those diff regions.

@Gilotopia Год назад

Could this also be used to optimize the training for a new base model?

@lewingtonn Год назад

Literally yes!!

@OlimSaidov Год назад

Excellent work, keep doing it ❤

@chrisharden7033 8 месяцев назад

So entertaining and insightful, thank you koiboi. Really enjoyed it!

@spider853 Год назад

You mentioned LORA as full training architecture, but isn't it external? 17:25

@haohaocreates 4 месяца назад

this is such a great video, would you be able to make a video on IPAdapters?

@asquidsink Год назад

best explanation i've heard - thanks for taking the time to put this together!

@Laszer271 Год назад

Great explanation. A few questions came to my mind after watching this: 1. Will it also work without any new controlling variable like a depth map? I think the method should still work and it could be used for transfer learning instead of traditional fine-tuning. 2. Will Controlnet without the new controlling variable work the same as Hypernetwork you described in the previous video? For me it doesn't seem like this is the case as I understood that hypernetwork outputs additional layers that dynamically change the structure of the original network and controlnet outputs activations that are then summed up with activations of the original network. 3. If it's really more efficient than other methods of fine-tuning models then could we use it for other tasks? Like instead of fine-tuning transformer or CNN we could also add a controlnet to them making the training process (hopefully) faster. 4. Are there tests of how many training examples are needed to train this controlnet? I understood that it was trained for a pretty long time on a pretty big dataset but it seems like it could also work for small datasets. LoRA, hypernetwork, dreambooth and textual inversion work for small datasets and controlnet doesn't seem like that much different of an idea compared to those methods.

@Mimeniia Год назад

Awe my ma se kind, thanks for the deep dive.

@msampson3d Год назад

Another amazing video! Love seeing these in my feed. Entertaining and informative

@ajudicator Год назад

Love this style of video. Please keep deep diving into papers in this way!

@derekghh 4 месяца назад

I've been programming for 30 years. This AI brings me back to the early days of computer, when hacking was so much fun.

@RexelBartolome Год назад

you are doing (a dabbing) Gods work, thank you for this video

@KainSpero Год назад

Wonderful video! You do such a great job breaking down these concepts.

@YesWeKenny Год назад

great vid thanks! 20:30 killed me lmao

@swannschilling474 Год назад

The boy is back!! 😁

@zhenli8674 Год назад

This is so cool! The main model seems to be an artist who is good at drawing everything, and control net seems to be the expertise in drawing a cat. So when you only train the control net, you are training the expertise in drawing a cat. But if you don't freeze the main model and train them all, you are training the artist to be someone who can only draw good cats, and that's overfitting.

@poliestotico 10 месяцев назад

this seems to be the same approach the LLMS are taking now, AGENTS or SWARMS are the future of General Purpose LLMS, same thing with image creation, split the smaller tasks and only require them to activate when needed

@itsBish2 Год назад

Thanks for your videos, helps me stay updated here. Really good work!

@xhinker Год назад

Well done, thanks for creating this video!

@TeodorAngelov Год назад

Using the full ControlNet seems to be somehow merging the output from Deliberate(my original choice) and SD15(the downloaded controlnet). Is ControlNet a type of Lora?

@desir.ivanova2625 Год назад

> Is ControlNet a type of LoRA? My thought exactly. ControlNet might not require the low rank decomposition, but at a high level it's the same idea, or am I missing something?

@KtosZPlanetyZiemia Год назад

do you have those C part 4tries somewhere? I would like to watch this, as it probably have a lot of nice info indeed (some other side of camera series)? Newcomer, enjoying your work!

@pm12321 Год назад

Excellent ELI5 explanation and overview, thank you for taking the time to create the video, and for your use of many visuals and diagrams. I'm a visual person, and it helps me understand things better, and to see (literally) how all the different parts relate and work together. I'd never be able to visualize and understand it like this just by reading the paper like you have. You should make a version of the dabbing image with Jabba the Hutt, and label it "Jabba Dab-a-Do!". I know he's not Fred Flintstone...so maybe that joke wouldn't work, unless you put Jabba next to Fred, who's looking at him with an expression of "Wtf dude?" Anyways thanks for this video, I like your style of explaining complex topics in simple terms for simpletons like me.

@CMak3r Год назад

Pretrained half Unet external network, bonded with original SD model, also CN is more efficient and easier to train than having a model trained with additional depth channel, got it. But how exactly CN is intervenes SD layers? Does it affecting latent space noise, or what?

@lyricsdepicted5628 Год назад

You are so gifted explaining AI stuff. I'd need your systematic way of explaining in the LLM/NLP/GPT realm. My AI life started with Stable Diffusion, but I don't see a way to make a living with images. More lucrative way seems to be helping companies to "integrate" their domain knowledge into a LLM to build knowledge systems. With your explanation through comparison of fine-tuning strategies (DB, TI, LoRa, HN) in mind, I get confused in the text-only realm. I see zero shot prompts, few shots, "context from a chat history", embeddings of mini-, midi-, maxi-text corpora - sometimes stored locally in a json, sometimes in a vector store, sometimes elsewhere, elsehow. Everything is somehow called fine-tuning embeddings without a clear demarcation. I'd need a systematics to all the possibilities.

@kaineis Год назад

This video is just perfect. I really love your style of making videos. That makes it really fun to watch :)

@Wingularity Год назад

are you available for consulation?

@timeless3d858 Год назад

Thanks for the information. I wonder how many other specific factors there are to train for. I'd love one for lighting.

@robrever Месяц назад

Is there a way to run Automatic1111 or ComfyUI locally with ControlNet while abstracting the Stable Diffusion layer using an API to Hugging Face? The idea is to run the user interface (Automatic1111 or ComfyUI) and ControlNet locally on my machine while offloading the heavy lifting (the actual image generation by Stable Diffusion) to an API like Hugging Face. I just want to benefit from the flexibility and control offered by ControlNet while not being limited by local hardware for the image generation process.

@JustinMcAlpine Год назад

Love this so much! I actually ended up reading this paper two nights ago. Very easy read. Love your insight about it. Do you have a discord or patreon?

@pipinstallyp Год назад

s8rVscu2pM , hope this helps!

@pipinstallyp Год назад

Hopefully comment stays up. Doing my best here to keep the invites up. I don't see my previous comments mentioning that.

@lewingtonn Год назад

I DO have a discord (forgot to link actually)! But fair warning: I don't really know how to manage discords properly. discord.gg/CNTQPUqK I've been obsessing over this paper for like 3 days now, I'm thinking of maybe doing like, an ablation study to determine how useful zero convolutions actually are, cuz like, wut???

@lewingtonn Год назад

@@pipinstallyp you're getting shadowbanned by Neal Mohan bro.. watch out D:

@pipinstallyp Год назад

@@lewingtonn It's about time Neal captures my soul eternally into an NFT and I become forever a citizen of RU-vid's hivemind.

@ДаниилРабинович-б9п Год назад

is it possible to train several submodels and add the outputs from each one? Because if yes, that would be something you can't really do with normal fine tuning, I think.

@AlistairKarim Год назад

Already done through having multiple networks working inside automatic1111. Avaible in setting -> controlnet. Actually, that is best way to do it. Canny + Hed models is a nice balance between coherence and speed. Try out smaller weights - about 0.6 per controlnet.

@_hkrm_ Год назад

I'm new to Stable Diffusion, only been using it for a few days and it was fun. now, I'm only 10 minutes into the video but I wanna make sure I'm understanding it correctly. basically, there are 2 people working on a task. person 1 doing the main task, person 2 helping. we feed them both with a task. the first guy start doing the job, but then the 2nd guy stopped the 1st guy and be like "wait a minute. I know how to do this. let me handle it and then I'll teach you later." 2nd guy: "so here's how you do it. this and that, blablablabla, done" both done with the job and we get the result exactly how we want. if we don't have the 2nd guy, the 1st wouldn't've known how to do our specific job, because he wasn't trained to do that. am I correct at this point?

@fritt_wastaken Год назад

I mean, have you ever seen a real brain that is just a big blob of neurons without structure that performs all the tasks? Me neither. Modularity is almost always better for training, we've seen it time and time again. Yet everyone still trying to make versatile blobs

@lewingtonn Год назад

bro pleeeease hit me with some link supporting that? I swear I have had the opposite experience, where the blobs perform better

@AC-zv3fx Год назад

SD or any singular model do pretty well, it is not like they're blobs. Neurons are self-organizing, afaik, one big model is too and it has latent space, which is more or less similar organization and differentiates it from just a blob. It is not that the output is better with the ControlNet but it is just easier to control. For me, having some kind of modules is like giving hints, but knowledge obtained without any hints is very often much more valuable. Basically the fact that the model came up with something itself, demonstrates its capabilities. It is like comparing a regular guy with a calculator and a genius human that itself is calculator, the guy would be restrained by his instrument, but a genius would be so comfortable with his knowledge, that he could look outside the box. I mean, for example, nobody hardcoded dall-e 1 back then into how to open photoshop and flip images, but it could flip and rotate them itself, which indicated its amazing capabilities. However, it'd be an interesting idea to do to teach a model how to use tools, like gpt models now are being taught how to google. I also just feel like there's no magic to using tools, it's easier to imagine an AI with godlike intelligence, inner structure of which would be incomprehendable to a human, than an AI of the same scale and power but that uses tools that are created and used by humans. Like, we already don't know what exactly are inner workings of SD, but if we knew, then it wouldn't be as great performing.

@LeKhang98 Год назад

Awesome and useful tyvm. But I wonder why there is no powerful open-source AI like chatGPT? I mean something like SD but for language. Is language harder to train? ChatGPT is great but it is like Midjourney. They don't give users as much control over the result as SD does.

@MrCheeze Год назад

When you did the comparison of depth2img and controlnet, I assume you were using the usual SD 1.5 controlnet, right? So it's not _entirely_ a fair comparison of the techniques, because depth2img is based on SD 2.0 (which is worse than 1.5). There have been some SD 2.X controlnet models released recently, though. I don't know whether that would be enough to make a completely fair comparison against the 2.0-depth2img model possible, though.

@dibbidydoo4318 Год назад

The depth map of 2.1 is only 64 pixel resolution whereas the ControlNet depth is 512 pixels which means that controlnet depth map are of a higher quality.

@lewingtonn Год назад

I'm not 100% sure but I thiiiink the controlnet paper authors condensed the depth map down to 64x64 using another small neural network... which is better than just resizing the depth map of course but still

@MrCheeze Год назад

@@dibbidydoo4318 I also expect the controlnet results to be better for theoretical reasons. But would be nice to see it shown in practice too.

@siskavard 9 дней назад

Dude, you're awesome, thanks for this

@NimbusAbi Год назад

i am very new to this whole diffusion topic, but isnt depth2img possible in stable diffusion 2? edit: i realized you probably refer to 1.5 only, correct?

@blasttrash Год назад

21:15 controlnet images look really good, but they do not look like jesus. On the other hand, depth2image doesnt look like a dab but images look like jesus. So I guess depth2image follows your prompt more closly while controlnet focuses more on depth map(or canny or whatever you input) instead of prompt. Is this correct?

@totheknee Год назад

19:40 - "I just downloaded this image of a gentleman dabbing, from the Internet." LOLLOOLLLOL, this made my day! Understatement of the month.

@kevinlu8685 Год назад

I'm wondering how many image do you use to train your ControlNet to produce dabbing result, or you just use the hugging face ControlNet pretrain model?

@therealgamer8150 Год назад

I wonder if this approach is generally true? Maybe the secret to general AI isn't one huge network, but hundreds of smaller, more tightly trained models for specific tasks all connected together in a clever way

@jayalmeida4887 Год назад

SO we don't need to download the 45GB to use controlnet if the External networks are included in A1111?

@sergetheijspartner2005 4 месяца назад

I would say it is the difference between Straciatella icecream and vanilla ice with chocolate sprinkles, both are vanilla icecream but one type straciatella you trained the whole icecream to be different and whatever you do you will allways get straciatella and it would be a hell of a job to only get vanilla out of that box, where as regular vanilla, you can add the chocolate sprinkles and add it to where it is satisfactory to you or just omit them according to your taste, but it's a lot faster, because you just add a bit to the big model without losing the big model, whereas in straicatella you change the model to something completely different and you can't finetune it anymore

@ArtiIntel-wl7su 10 месяцев назад

Man, every controlNet model has been created for SDXL except the one most useful to me; normal maps! 😔

@zoranvelijevic7871 Год назад

well it looks the authorq of controlnet went to school and studied design patterns : decorator, visitor, chain of responsibility are so obvious here.

@Zoronoa01 Месяц назад

Amazing video, I hope you make more of videos like this!

@hleet Год назад

Okay ... so what's depth2image ? xD

@reespawson Год назад

This is so good man. Keep these coming. Are you a researcher or what? What's your Twitter?

@gabrielbirnbaum9593 Год назад

would love to join your discord but invite is expired? Also, do you have Patreon?

@burnystabbyguy Год назад

is there any reason you couldn't apply this control net strategy to work for language models?

@andredeloach8068 9 месяцев назад

Thank you 🙂

@yoavco99 Год назад

Please make a video about LoCon and also the vae revolution.

@lewingtonn Год назад

Bro please link me, never heard of this

@yoavco99 Год назад

@@lewingtonn cant link here. It deletes my comment

@jonm6834 5 месяцев назад

I appreciate the technical explanation but, in all honesty, I boiled it down after a single picture: ControlNet prioritizes image-to-image for denoising the subject, and text-to-image fills in the details.

@HRZN-xj9um Год назад

I love these educational videos.

@Li.Siyuan Год назад

I applaud your idea of answering the question of 'What is xxx"? I was then completely confused by the example used of "Jesus dabbing". WTF is 'Dabbing'? I'm afraid I only speak English and not the gobbledegook that 'dabbing' seems to describe. Could somebody please enlighten me in words of one syllable or less, please?

@robpannoni9260 Год назад

Love your videos. There's not a lot of info on SD for people who are technical and want details, but who are not AI experts or hardcore programmers. You've found the sweet spot. Your explanations have helped me quickly make a great leap forward in understanding and using SD. Thanks for putting these together.

@Clabear 4 месяца назад

Love your video! great explanation!

@loctite417 Год назад

I've been having fun using controlnet. Pretty interesting that we dont actually know why something is better. It's like throwing at a wall and seeing what sticks

@philippk5115 Год назад

Awesome video, accurate, but still it made me giggle at times xD

@Grumpy-Fallboy Год назад

I like your sense of humor, keep these videos coming 😁

@tlatosmd7074 Год назад

The first 10 minutes: Basically saying that ControlNet is about making and training an external expert, like a hypernetwork "contract lawyer". So if I get this video right, ChatGTP urgently requires a hypernetwork "contract lawyer" or external sub-routine for simple math, Excel-style table sorting, and coding. C. 10:00 to 15:00: What one may understand after studying ML at university for about 50 years. 15:00 to 19:20: Pretty much regurgitating the simple fact that what makes ControlNet special is the external contract lawyer. 19:20 to 22:20: A few priddy pictures where the guy's going absolutely bonkers about a supposed difference between the outputs from Depth2img and ControlNet I can't see at all, outside of Depth2img obviously having been told or trained to use a fresco style, whereas ControlNet was told to or trained to partly also use a photo style. It seems like he's even ignoring those Depth2img results where they have the same shirt folds and cherrypicking those few ControlNet results that are pretty much the same. 22:20 to 23:00: Explaining that Depth2img needs huge industrial GPUs worth $15-30,000 a piece, whereas ControlNet can be used with consumer GPUs. 23:00 to end: More gushing about a quality difference I can't see. I mean, it's cool that unlike Depth2img, you can do it at home, but as said, I don't see the actual visual difference.

@tlatosmd7074 Год назад

So, bottom line of the entire 25-minutes video: ControlNet means you train external little sub-routines. That's why you can do it at home, the sub-routines are much smaller, and you don't have to be a millionaire to use it.

@ahmadzaka9885 5 месяцев назад

Man this video is so helpful. Ty

@evvbmstu Год назад

Almost erotic schemes man, appreciate it!

@Welsed Год назад

I love how these researchers basically applied layers or rather layering processes to a neural network. Like, they basically are breaking down what a designer would do manually but they have to do EVERYTHING from scratch. LOL. What a round about way to develop a tool for people to be lazy.

@cdreid9999 Год назад

er...it is more like. multiple ai's working together coopeeatively. Like an arvhitect who has an engineer to design the building, an artist to draw the concept, a landscaper to design the landscape etc

@kiktamo Год назад

I can't say much from a technical perspective since there's a lot about the overall technology that I haven't had time to read up on. However, the logic behind it kind of makes sense to me after all by freezing the main model and training the secondary one you're limiting the scope of what has to be learned. Similarly to me it almost seems like the method is less learn something new and more find the difference between what it already knows and what it's being given.