How to AI Animate. AnimateDiff in ComfyUI Tutorial.

Подписаться 136 тыс.

Просмотров 151 тыс.

50% 1

Text2Video and Video2Video AI Animations in this AnimateDiff Tutorial for ComfyUI.
Install Local ComfyUI • How to install and use...
Use Cloud ComfyUI www.thinkdiffusion.com
Workflows civitai.com/articles/2379/gui...
AnimateDiff Models huggingface.co/guoyww/animate...
ControlNet Models huggingface.co/lllyasviel/Con...
FFmpeg Guide www.wikihow.com/Install-FFmpe...
FFmpeg Download www.gyan.dev/ffmpeg/builds/
Prompt styles for Stable diffusion a1111 & Vlad/SD.Next: / sebs-hilis-79649068
ComfyUI workflow for 1.5 models: / comfyui-1-5-86145057
ComfyUI Workflow for SDXL: / comfyui-workflow-86104919
Get early access to videos and help me, support me on Patreon / sebastiankamph
Chat with me in our community discord: / discord
My Weekly AI Art Challenges • Let's AI Paint - Weekl...
My Stable diffusion workflow to Perfect Images • Revealing my Workflow ...
ControlNet tutorial and install guide • NEW ControlNet for Sta...
Famous Scenes Remade by ControlNet AI • Famous Scenes Remade b...
CHAPTERS
0:00 AI Animate with AnimateDiff
0:42 Download workflows
1:18 How to use AnimateDiff - Text2Video, Settings Guide & Easy Cloud Solution
11:05 Video2Video + Local Install
22:44 Prompt travel/scheduling
25:48 How to install ffmpeg

Хобби

Опубликовано:

28 май 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 184

@ElHongoVerde 6 месяцев назад

Friday and new video from Seb. Nothing could go wrong today!

@sebastiankamph 6 месяцев назад

And it's even an extra long one to celebrate friday! 😊🌟

@dustinjohnsenmedia1889 6 месяцев назад

Great video! Now I just figure out how to tack on an upscaler at the end

@chipko 5 месяцев назад

This is fantastic.. well produced, well explained and so informative. Thank you.

@sebastiankamph 5 месяцев назад

Glad you enjoyed it!

@IAmTheAntiArt 6 месяцев назад

My first T2V SDXL AnimateDiff with HotShotXL vid is rendering right now! This vid got me started with animate diff in comfy 6 days ago. Thanks!

@Mowgi 6 месяцев назад

Mate, you have great timing. I've just recently installed ComfyUI and started getting into animatediff, but I really haven't understood consistent animations, just the morphin stuff so far.

@sebastiankamph 6 месяцев назад

Great to hear! Hope the video helped you, let me know how it goes 😊🌟

@cosmiccarp5030 6 месяцев назад

Sweet, I literally just sat down to learn comfy and animatediff. Your timing is impeccable sir! Comfy UI is worth it everyone! I'm really liking it. Especially the screen shot workflows!

@sebastiankamph 6 месяцев назад

It really is very convenient. For workflows like these, you don't even have to learn Comfy! 😊

@cosmiccarp5030 6 месяцев назад

Yes, but with relaxing tutorials like yours, it is a pleasure to learn comfy! Cheers!@@sebastiankamph

@Paperclown 4 месяца назад

@@sebastiankamph but you said you would cover txt2video with local later in video and sections dont show that. im not watching a feature length movie just to skip over solutions for people who dont even own a computer ! you dont cover the process of setting up locally for txt2video nodes

@defy_norms Месяц назад

My gosh, dude you got a subscriber!! You are amazing, keep it up. This helps small RU-vid channels like us immensely

@sebastiankamph Месяц назад

Thanks for the sub! Happy to help :)

@core-f 6 месяцев назад

Correct me if I am wrong, but I think the frames you use for prompt traveling are not the starting frames for the given prompt, except frame "0". It's the frame where the new prompt is completed. That's why you have a rather long winter in the default example, because it transitions from frame 50 to 75 into winter and then has an extra 25 frames for the winter prompt. For an even distribution you'd need the frames 0, 33, 66 and 100.

@joonienyc Месяц назад

bro , the best way to experiment with ur basic txt2vid and vid2vid ! thank u

@sebastiankamph Месяц назад

Glad you liked it!

@PushedToInsanity 6 месяцев назад

Great tutorial as usual. Appreciate all the examples you've give, by far my favorite RU-vidr when it comes to AI art. Do you offer personalized lessons?

@sebastiankamph 6 месяцев назад

Thank you, how very kind! Yes, I do. You can write to me in Discord. See link in channel or video description. 🥰

@Cu-gp4fy 6 месяцев назад

Dude your examples are 🔥🔥🔥

@sebastiankamph 6 месяцев назад

Boom, thank you! 😊🌟

@sharpenednoodles 6 месяцев назад

Finally, an animation tutorial from the king

@ShoreAllan 6 месяцев назад

Hello Sebastian, many greetings from Berlin and thank you for your work! What do I have to do to extend your workflow with a face swap from a photo - i.e. video-and-picture-to-video!

@KDawg5000 6 месяцев назад

Very nice tutorial! Just a heads up to those who want to use SDXL models; you can use them, but you have to use the Hot Shot motion module, and set the resolution smaller.

@kiksu1 6 месяцев назад

SDXL beta model for AnimateDiff was also released just today morning 👍🏻

@KDawg5000 6 месяцев назад

@@kiksu1 Wow, nice. I'll have to check it out now. EDIT: I just tried it out in A1111, but not good results. Every frame is a completely separate image, unrelated to the previous frame. Hmmm. Well at least it tried instead of just crashing.

@kiksu1 6 месяцев назад

@@KDawg5000 Oh yeah, sadly it doesn't do great yet. Tested that too but I'm not convinced yet. Hotshot worked much better.

@stonewhite3576 6 месяцев назад

you're an awesome dude! congratulations 🎉👏

@dejuak 3 месяца назад

Hey really nice video i have watched it like 10 times on the last month. I have a question, is there a way to only animate the character but keeping the background static? Would be really awesome

@clovernacknime6984 6 месяцев назад

7:00 I think you mean convergence. Divergence means that a sampler (function) never settles down to (approaches) a particular image (or other output), while convergence means it will.

@sebastiankamph 6 месяцев назад

Right! I wonder what made my brain mix that up 😅

@alpineuniverse 6 месяцев назад

Just commented the same thing and realized someone must have caught it too :)

@choboruin 2 месяца назад

First guide that actually let me generate something. Wasted 8 hours today using this and I just got a video after another 30 minutes and its just a puddle LOL. Appreciate the tutorial though! Great content.

@_Al00f_ 2 месяца назад

I'm glad you had some luck.

@YazzCasillas 6 месяцев назад

Very nice tutorial, thanks! Is it already possible to start from an image?

@hleet 5 месяцев назад

It's very well explained ! Thank you

@sebastiankamph 5 месяцев назад

Glad it was helpful! ☺️

@gorgep1242 5 месяцев назад

Hey Sebastian. Nice video. Have you planned to make an AnimateDiff video for A1111?

@hesamarmaghani791 6 месяцев назад

thanks man This was very helpful

@ronnykhalil 6 месяцев назад

as always, love yo videos! thanks so much

@sebastiankamph 6 месяцев назад

You are so welcome! Glad to have had you aboard for such a long time Ronny! 😊🌟

@Kontaktfilms 4 месяца назад

Sebastian, I'm a fan of your work and tutorials - I followed along for the Video2Video workflow, all my nodes and models are there, I see my original frames and lineart renders in preview... then when I render the Queue the image under Video Combine is just black. I had the same problem with another workflow. Any idea what could be causing this? I'm on a Apple M1 Ultra. Thank you.

@erdbeerbus 6 месяцев назад

excellent, thank you. where can I get your basic vid2vid workflow, please ... which patreon mode is (if) needed? thx in advance :)

@digidope 6 месяцев назад

Context window: That is the fps that the model is trained for and you should NOT change it with animateDiff. If you use other model than AnimateDiff like HotShot then HS is trained with 12fps and then you need to change that value.

@EarthWalkerOne 6 месяцев назад

Not fps, but the number of frames that animatediff chunks together for temporal consistency. It is motion model dependent though, so increasing the value higher than what the motion model was trained for will give you worse results.

@vaderragex90 6 месяцев назад

Very nice tutorials, these are so great! Ich liebe es!

@brandonlane 4 месяца назад

Thank you! 😊

@myghail Месяц назад

Idk how many times I heard u say "I prefer 2m karras" in the last couple of days. :D Not that I'm complaining! Thanks for the videos, they are incredible - first time I'm actually getting all this and I've tried many times.

@sebastiankamph Месяц назад

People want to know what I use :D Some just want the answers and not the whys or hows. Glad you're enjoying the content

@myghail Месяц назад

@@sebastiankamph I understand completely + in all honesty I woulda forgotten by now which ain't good considering 2m karras does work the best definitely

@weijiayao9118 3 месяца назад

I like your online version demo, but can I use local checkpoint models, it seems there are only limited selections.

@masoud.art.videos 16 дней назад

Great guide, thank you! first timer using comfyUI here, and one thing is strange to me is processing 10 frames for me takes 40 seconds but 100 frames almost an hour! I have a decent gpu(64gb), which makes me think something is not correctly set up. Any ideas?

@bdwedgeofanimotion4106 2 месяца назад

that was nice and need to absorb

@Clupea101 6 месяцев назад

Great stuff, easy learniing

@sebastiankamph 6 месяцев назад

Glad you think so! 😊

@journeysinthedeep 4 месяца назад

Awesome tutorial bro!! Only issue im running into is my video combine vhs output format is only allowing me to select from webp and gif. I cant choose h.264. Any ideas my guy??

@Tedrer 4 месяца назад

Hi. There is some changes, animatediff combine is deprecated. it is the same to use video combine node?

@francsharma7276 4 месяца назад

great video i done at 3070ti 8gb laptop, 480p resolution, hope i can resize it, love u bro

@nicocro00 5 месяцев назад

very nice! what's your setup for the local install? What machine / GPUs are you using?

@sebastiankamph 5 месяцев назад

Rtx 4090 currently. Previously RTX 3080.

@dougiejones628 5 месяцев назад

Very cool, is there a way to use an image as a reference instead of a prompt? I have a CG animated image sequence for the input video. And I've enhanced one frame from that sequence and would love to apply the style of that frame to the entire video. Is that possible? Thanks!

@sebastiankamph 5 месяцев назад

IPadapter or pix2pix

@CrudelyMade Месяц назад

do you have a document or video which summarizes the differences between the sampler like euler, karra, etc.. you mentioned some of the differences between the models in this video, and it was a great explanation. would love to see a 'breakdown' of the rest. :-)

@sebastiankamph Месяц назад

Yes, check my beginner's guides on Stable diffusion where I show a1111. I go through samplers there.

@leegregory5617 6 месяцев назад

Can you do this starting with a still image, rather than a video?

@Caret-ws1wo 4 месяца назад

Any plans to make a video like this for a1111?

@triangulummapping4516 2 месяца назад

Im trying video2video, the background of the result is completely static, just the character is moving fine, is there a way to also give some movement and evolution to the background?

@LaloHao 5 месяцев назад

What GPU were you using in local generation?

@marcus_ohreallyus 6 месяцев назад

Thanks for all of your great vids. Can you help with a technical problem that happens when I use AnimateDiff in A1111? I always run out of memory with a CUDA error halfway through every AnimateDiff generation, but I have a 4090 with 24GB vram. I've tried almost every bat file argument suggested on reddit, but the error still happens.

@sebastiankamph 6 месяцев назад

Hey, glad you're liking the videos! What settings are you using? Mainly number of frames and resolution. Are you using xformers or sdp?

@marcus_ohreallyus 6 месяцев назад

@@sebastiankamph I have xformers on. Just the default frames, I think its 16. And I don't do any upscale...resolution is 512 by 768. I don't think it truly is a memory error, because I know that people with much lower ram do this with no problems. When I look at VRAM usage on my GPU monitor, there's plenty of RAM during the generation. I thought maybe there was a setting you know about to make the error stop.

@sebastiankamph 6 месяцев назад

@@marcus_ohreallyus that's weird. I would try it in Comfy and see if you get similar results there.

@ShawnTWhitney 5 месяцев назад

Great tutorial but I'm having an install problem with the ComfyUI Manager in Google Colab b/c there's a conflict between two component versions. Can anyone provide some insight into how to fix this?: "Detected that PyTorch and torchvision were compiled with different CUDA major versions. PyTorch has CUDA Version=12.1 and torchvision has CUDA Version=11.8. Please reinstall the torchvision that matches your PyTorch install."

@EmilyNilsen 6 месяцев назад

I think I'm hooked on your pappa jokes.

@sebastiankamph 6 месяцев назад

Lovely, aren't they? Everyone needs more dad jokes in their lives 😊💫

@SebAnt 6 месяцев назад

Thanks! I had no desire to try animations but got sucked into the rabbit hole and tried all 3 examples you taught. Your teaching is crystal clear 🙏🏼 I did stumble into some error messages that my path was too long, and I had to modify the registry to enable long pathnames. Last week I learned how to use Reactor to do face swap in A1111. If it’s possible to use that within ComfyUI for videos too, it would be awesome if you could do a tutorial on that .

@sebastiankamph 6 месяцев назад

I'm happy you keep enjoying the videos, Sebastian! And I very much appreciate your support, it is a great help towards me creating these videos. There are face swappers for video, yes. I will for sure get a tutorial done on one of them soon. That's a great idea 😊🌟

@SebAnt 6 месяцев назад

@@sebastiankamph I just saw a video where someone did face swap in SD using Reactor. Can this be done in ComfyUI? ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-6dY_1TfzZTw.htmlsi=XusTReirIFc0nE8u

@KINGLIFERISM 6 месяцев назад

Seond on that@@sebastiankamph But consider this idea (I am not good at comfyUi but have a grasp on a concept that I am not able to do myself) I was wonder since roop has a graphic limitation due to the model it uses why not do ip-adaptor + face in the workflow then face merge/blend into roop this way you get a higher resolution face and then that image goes into the video generation process.... let me know what you think.

@kietzi 2 месяца назад

wanted to try the v2v in thinkdiffusion, but i have just errors there and dont know how to load uploaded img-seqences.. so i stay at rendering over night, to see some results at my own machine :P

@Inner-Reflections-AI 6 месяцев назад

Nice! Thanks for highlighting my workflows! I have learnt a lot from your videos!

@sebastiankamph 6 месяцев назад

I'm happy to hear that! Comfy is probably where I've played the least so far, but for some tasks it feels like a necessity :) Keep up the good work. Maybe an img2video workflow next?

@alpineuniverse 6 месяцев назад

Great tutorial! Thank you for doing this. Quick correction: I think you mean "convergent" instead of "divergent". We want our sampler to converge on an image, not diverge from it. Cheers!

@sebastiankamph 6 месяцев назад

You're absolutely right, thank you!

@Disco_Tek 6 месяцев назад

I'm using SDXL and Hotshot. I'm not generating from empty latents though and am running multiple controlnets and temporal net... but the results are extremely good.

@ufukzayim6689 6 месяцев назад

Do you have same or similar tutorial also for A1111 please

@Shingo_AI_Art 6 месяцев назад

That thing about divergency with samplers was interesting, it would be nice to have a video reviewing all of them

@user-pc7ef5sb6x 6 месяцев назад

I noticed a difference in quality between Auto1111 and Comfy Animatediff. Animations are less jittery in Auto1111. Not sure if it's something in the way Auto1111 processes the images.

@sebastiankamph 6 месяцев назад

Interesting! I haven't done a comparison between the two. Let me know if you find out more about that.

@art3112 6 месяцев назад

@@sebastiankamph I would be interested in a quality comparison between Comfy and A111 too.

@KINGLIFERISM 6 месяцев назад

What you did that many do not... was you made errors and corrected them. That is a sign of a person that understands user experience. That is also a person of empathy. You sir... are a nice person.

@sebastiankamph 6 месяцев назад

That's very kind of you, thank you very much! You're the real mvp 😊💫

@A_G420 6 месяцев назад

I may give Think Diffusion a shot. Running a 6800 on Arch Linux for the basic txt2vid on your tutorial took 22 minutes to generate. No controlnet for Comfy on Think Diffusion kind of sucks

@sebastiankamph 6 месяцев назад

Works well for me. What are you missing?

@A_G420 6 месяцев назад

From looking at the site it appears that way. I'll do a free trial today & mess around with it. Thanks@@sebastiankamph

@zzx2879 5 месяцев назад

hi i have a question, when i apply the control net, i get an error message as followed, would you recommend a fix? Thanks Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_CUDA_addmm)

@zzx2879 5 месяцев назад

im applying the same control net as you, just using a different checkpoint (photon v1)

@Eightysixstudios 5 месяцев назад

getting this error message when trying to follow: Error occurred when executing KSampler: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_CUDA_addmm)

@juschu85 6 месяцев назад

7:09 I think converge is the word you're looking for.

@dragongaiden1992 14 дней назад

Friend, it is functional, but my result is nothing like my input video, it is completely different, it should be said that I am using checkpoint xl and model xl from animatediff, at night I hope to continue testing, thanks for the video, it takes me about 15 or 20 min to render 50 frames locally

@SLAMINGKICKS 2 месяца назад

what does this mean ? Motion module 'mm_sd_v15_v2.ckpt' is intended for SD1.5 models, but the provided model is type SDXL.

@sebastiankamph 2 месяца назад

There are two types of models (well more, but simplified). 1.5 and SDXL. Most stuff works best with 1.5 models. So you need to swap your SDXL model into an 1.5 one (the main model you have selected).

@icchansan 6 месяцев назад

Looks amazing tuto but I'm getting mass errors with missing nodes, seems likes it manager get it from the repo but still getting errors with vhs videocombine

@sebastiankamph 6 месяцев назад

All nodes in those workflows should be able to be installed with the manager. Let me know if they won't. If vhs videocombine gives you errors, try the regular videocombine. Just double click and type combine to see the ones you have available.

@ohheyvoid 3 месяца назад

that joke got me! :D

@MrRomanrin 5 месяцев назад

ok WHERE ARE THE WORKFLOWS PICTURES ?

@khanhtrong7761 6 месяцев назад

Between ComfyUI and SD1111 WebUI, which one gives better result?

@sebastiankamph 6 месяцев назад

For animatediff, I find that there are more options with comfyui

@kevinbrennan4066 6 месяцев назад

Thanks for this great video!

@sebastiankamph 6 месяцев назад

Glad you liked it! 🌟😊

@EonsAway 6 месяцев назад

Hiya, How do you get the preview image in the KSampler?

@sebastiankamph 6 месяцев назад

If you click manager and check top left, there's a preview dropdown there.

@user-cf4on1qy8n 5 месяцев назад

anyone knows why my mac m1(sonoma) generates black images when using animatediff through comfyui?

@tambe2182 5 месяцев назад

[Errno 2] No such file or directory getting this error, what to do ?

@mr-s23 4 месяца назад

Can I change the setting to more than 4 seconds?

@raizer8358 5 месяцев назад

can i still use the txt2vid workflows with 6 Gb of VRAM? will it crash or only run slower?

@sebastiankamph 5 месяцев назад

Might break due to out of memory, but test it and see.

@LianParma 6 месяцев назад

I've tried to run this and it just won't work. Even tried a new install of ComfyUI from scratch and even more errors pop up when it gets to the KSampler. Is it broken for someone else or it's just for me? :D

@ElevateMeEM Месяц назад

will this work on imac?

@stereotyp9991 6 месяцев назад

servus! wo kommst du her?

@user-xj3fg3bd3d 24 дня назад

what should I do if there's a message that some node types were not found

@sebastiankamph 18 дней назад

Did you go into the manager and try installing missing custom nodes?

@koningsbruggen 6 месяцев назад

I love these tutorials

@sebastiankamph 6 месяцев назад

I love seeing you in the comments again!

@kkryptokayden4653 6 месяцев назад

is it possible to swap faces as well in vid to vid maybe adding a reactor node or roop node?

@diklashain Месяц назад

Hey, I keep on getting the same error message: "Error occurred when executing KSampler: module 'comfy.sample' has no attribute 'prepare_mask'" It happens with all the workflows, and I can't find solutions in google... Could you please help me?

@MrRomanrin 5 месяцев назад

i am searching for a Img2Img + ANIMATION work flow

@havelicricket 4 месяца назад

(pls help me) Error occurred when executing KSampler: 'NoneType' object has no attribute 'shape'

@MatichekYoutube 6 месяцев назад

and for the final - we can do LCM insane speeds and AnimateDiff? Letsgooo?

@sebastiankamph 6 месяцев назад

Yasss! 😁

@MatichekYoutube 6 месяцев назад

@@sebastiankamph i can not imagine you would be able to do like 5min video in 10 min rendering time :)

@techviking23 6 месяцев назад

WOWWWW

@sebastiankamph 6 месяцев назад

Now go make some fantastic animations!

@PlayerGamesOtaku 5 месяцев назад

Hi, why didn't you show many nodes how to install them? I can't find: number of frames and many other nodes, I don't keep them and I can't find them

@sebastiankamph 5 месяцев назад

Use the manager, see comfy install guide

@RenoRivsan 3 месяца назад

Where do I add controlnet?

@AshChoudhary 6 месяцев назад

How can we add Lora in this?

@dkontey6421 6 месяцев назад

where is this video to video workflow?

@samsilva7209 2 месяца назад

when animatediff is completely black, what happened wrong? I'm on a m1 max 64

@choboruin 2 месяца назад

It took me 40 minutes with 3080 ryzen 5900 to generate a puddle of mud. What kind of PC are u using? seems ur rendering fast LOL..

@_Al00f_ 2 месяца назад

I'm glad I'm not the only one who feels like they wasted their time. Followed everything to a T and I'm STILL missing the input box. Going through ALL THIS BS for a 5 sec looping video? Uh no thanks. I've watched way too many videos on this topic. Comfy-my arse. I don't see a point in asking questions. I'll get no answers! Anyway, good luck to you.

@mikerotchburns1622 3 месяца назад

the lineart preprocessor is bugged or something, its getting hung up on that. updating the nodes does nothing. its not shown as a missing node. changing to open pose does nothing

@Suketh 2 месяца назад

try to "Enable win32 long paths" to 1 in your Registry Editor

@MilesBellas 6 месяцев назад

A new animateDiff playlist next?

@sebastiankamph 6 месяцев назад

That sounds like a fantastic idea!

@JohnLeeKim 5 месяцев назад

Woooh it's working!

@uncleben2019 3 месяца назад

Where do I get woman.mp4 (or women in general?)

@rooqueen6259 25 дней назад

Guys who have come across the fact that the loading 2 new models stops at 0% or I also had an example - the loading 3 new models is 9% and no longer continues. What is the problem? :c

@abdullahxie 6 месяцев назад

i am getting error of VHS_Videocombine from a week can you help me ?

@sebastiankamph 6 месяцев назад

Make sure to update all

@abdullahxie 6 месяцев назад

@@sebastiankamph still not working, may be it’s bcz of ffmpeg version ?

@Full_Zeb 5 месяцев назад

My comfy wont let me install any nodes

@felipealmeida5880 6 месяцев назад

It's cool, but the animations are not consistent, they are short and if you don't have a very good PC, it takes a long time to generate just a few seconds

@skybeast2738 3 месяца назад

True, although with controlnet animations can be quite consistent

@dhanang 2 месяца назад

AI is still at very initial stage. Considering what these programs can do like 1 year ago, this is a leap.

@ThatGuyNamedBender 2 месяца назад

I guarantee it takes less than it takes to render a 2 second clip in blender. Man script kiddies are too damn bitchy nowadays. Like bro you're already generating art with text, not time or skill deal with the limitations 🤣🤣🤣

@XobyThePoet 3 месяца назад

I wish I could get this to work. I am envious of those who can do this. I have wasted money on getting this shit to work and I am fucking done with it. Who do I have to pay to just set this up on my damn computer for me?