ComfyUI AnimateDiff Prompt Travel: ControlNets and Video to Video!!! 

Tested on AMD RX6800XT (Ubuntu22.04 + ROCm 5.7). it works flawlessly and stay close to 12Go of VRAM. Really helpful, thanks a lot.

Nicely Done!

Hard! Thank you very much for paying for it all the way! It's a pity that I brushed it here before going to bed and have to wait until tomorrow to practice.

Bro we on this journey together. Keep goin!

Converting the pretext, and how to do that completely blew my mind and opened doors to understanding what I could do. Thank you.

yea baby (edit: this is straight up the most valuable 10 minutes ive watched on RU-vid in a while, exactly the signal I needed amidst all the noise regarding comfy and diff. You explained it really well and clear. Thank ye kindly!

Nice job on these of late. In general I have a hard time watching video tutorials w/ people on screen talking - but you're hitting all the right notes on these so far.......Haven't -wanted- to bother w/ Comfy, but have definitely admired the generations some have been sharing. Thanks for making well timed / friendly tutorials. Stick w/ it and you'll def build a good/active channel. Thanks!

Love your vids bro!!! I know it takes a lot of work to makes these, really appreciate your efforts. 🙌

damn, glad i saw your channel recommended on reddit. fantastic video - calm, concise and well made!

great video

Very helpful and approachable tutorial. Thanks!

Exciting! I hope this helps me figure out how to do a much longer vid2vid without running out of vram

Thank you. Excellent tutorial :) Keep them coming, subbed!

I was waiting for this one! Thanks mate & keep 'em coming :)

straight to the point and clear, nice to follow,no music is perfect. i can choose my own background if needed. thanks a lot you just made me a fortune haha

Thanks for the video, very helpful. Well done😍

awesome, thank you. Im really appreciated

Awesome! Thanks for sharing the resources.

thanks for the tutorial, man! 🙌

Thanks worked finally❤❤

good stuff

this is it!!

Thanks! Straight to the point!

that my friend is a very nice video, thanks a lot, I'll follow your stuff

this is sick, nice work dude. sub'd

Whoa! This is awesome, thanks for sharing your workflow. I haven’t used ComfyUI - just been in A1111. Can you recommend tutorials for Comfy? Or any you’ve made that’ll be a solid start to start learning this method? Thank you!!

Good work! Thanks! 👏

This video will help many creators. Please have a place to pay for coffee.

Great!!!

cool vid, you'll might the best animatediff channel now, what's coming next?

Really good tutorial,thank you!

perfect tuto ! Thanks a lot !

very helpful as always thanks. Is there a way to make a "preview" video frame node so that you can view the progress of the render before it is completed? This way one could cancel the render if it looks terrible or not the way you want it without wasting render time. This was one of the nice things about deforum that saved me a lot of time

What was the size of the video in the end? Because you showed 1024 576 in the beginning. Is that the resolution in the end as well? Also how do you load other formats of video, I only have webp and gif?

nice tutorial, have u found a way that u can upload a 3 min video in one piece into the VHS load video node?

Great Video my friend!! Elliryk 😉

I use run diffusion and can't load your workflow in json( can you upload the pic too maybe that can fix it?

Thanks for ur sharing. I really learn a lot. Quick question... how do I change model's cloth and keep the new cloth can prerform consistently. I type in sweater, but some frame will change sweater into tank top...

I get an error at the Ksampler processing node! "AttributeError: 'NoneType' object has no attribute 'shape'"

error KSampler mat1 and mat2 shapes cannot be multiplied (2464x2048 and 768x320)

This is so fucking cool and it worked with no issues! Thanks!

Awesome! I have one question, how can I install / connect ffmpeg to comfyUI. I get this error "[AnimateDiffEvo] - WARNING - ffmpeg could not be found. Outputs that require it have been disabled". Thank you very much!

followed this all the way through and at the end my video output was just black. any suggestions?

I don't know how to fix this problem. 'ControlNet' object has no attribute 'latent_format'

crushing bro

Error occurred when executing CheckpointLoaderSimpleWithNoiseSelect ...

the ebsynth of comfyui

really helpful tutorial, thx

Awesome video! Do you know how you can combine openpose and depth and lineart together to improve the results?

Great tutorial bro ! Could you make a video about morphing videos with Ipadapters?

what do i have to do if i want to change the look of it ? since yours is closer to anime style i want to make it to realism or sci-fi etc

Im using Ksampler, but it takes 9 minutes for 1 of the 25 samples. Which is obviously extremely slow. Im working on a macbook m1 Max, do you have any tips on making it quicker?

can you show how to remove animatediff from this workflow... i dont want my video to change style

Starting to learn ComfyUI, when I click on Queue Prompt, my computer takes a long time to process the KSampler node. I'm using a MacBook Pro 14 M2 PRO, is it normal for it to take so long?

thanks! got it working after a few tries, but i realize the prompts are not really working (at least I don't see them "travelling"), it seems the whole prompts are taken into consideration instead. do you have similar issues? i see that the default workflow has 4 prompts, but in your generated video at least it traveled from green lush to wintery storm, whereas mine always started with wintery storm, and remained like that throughout the video.

Top!

Do u know what happen to this error? Error occurred when executing CheckpointLoaderSimpleWithNoiseSelect: 'model.diffusion_model.input_blocks.0.0.weight' File "D:\AI\ComfyUI_windows_portable\ComfyUI\execution.py", line 151, in recursive_execute output_data, output_ui = get_output_data(obj, input_data_all) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\AI\ComfyUI_windows_portable\ComfyUI\execution.py", line 81, in get_output_data return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\AI\ComfyUI_windows_portable\ComfyUI\execution.py", line 74, in map_node_over_list results.append(getattr(obj, func)(**slice_dict(input_data_all, i))) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\AI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-AnimateDiff-Evolved\animatediff odes_extras.py", line 52, in load_checkpoint out = load_checkpoint_guess_config(ckpt_path, output_vae=True, output_clip=True, embedding_directory=folder_paths.get_folder_paths("embeddings")) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\AI\ComfyUI_windows_portable\ComfyUI\comfy\sd.py", line 511, in load_checkpoint_guess_config model_config = model_detection.model_config_from_unet(sd, diffusion_model_prefix) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\AI\ComfyUI_windows_portable\ComfyUI\comfy\model_detection.py", line 239, in model_config_from_unet unet_config = detect_unet_config(state_dict, unet_key_prefix) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\AI\ComfyUI_windows_portable\ComfyUI\comfy\model_detection.py", line 120, in detect_unet_config model_channels = state_dict['{}input_blocks.0.0.weight'.format(key_prefix)].shape[0] ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

I don't see a link to download the workflow

i really appreciate for this guide. this will help me alot! however i have 1 problem maybe you can help me with i have done everything right. i see frames from video and i see controlnet output with lines. however after ksampler my gif and image outputs are all blackscreen. what do you think my problem could be.

Hey man - love the tutorials!! Where do you add different video creation formats - I only have gif and webp - Thx

Thanks! I downloaded this workflow but the output only has formats for image/gif, or image/webp, even though I am inputting video. There is no video/h264 setting in the dropdown. Any idea how I can add that?

Wow. Huge thanks for this! Is it possible to use a SD XL model for generating a painting style? I’d like to use this one and I don’t know if it’s possible with your workflow. Btw, many thanks !!

The worklflow file is giving me an error: TypeError: Cannot read properties of undefined (reading '0')

i love the output

Will it work for a macbook pro M1 with 16gb or ram??

Weird im getting preview images from the upscaler node and the lineart images from the controlnet, but im not getting any actual output results.

Nice, may have to try this after work. is it the same process if you want to use more uptodate models (cant go back to 1.5 after using SDXL lol)

I don't have the option for mp4 output do you have any idea why ?

is there wa way to point to different directories so we don't have to re'download models, lors and others files.

ive set everything up, but for some reason i get error at ksampler step "Error occurred when executing KSampler: 'ModuleList' object has no attribute '1'"

Can you make a tutorial on how to upscale the render ?

Thank you for this tuttorial! I'm using colab version and i get tottally black result pictures and video, could you give me a hint how can i fixe it? thx But most of the time i get this issue ..SD model must be either SD1.5-based for AnimateDiff or SDXL-based for HotShotXL. Need help..=\

How would you go around adding a Lora (not a motion one) to this workflow? I tried adding a Load Lora after Load checkpoint, but I'm getting black images.

Hey man, great tutorial ! I have an error message that pops out however, it says "Control type ControlNet may not support required features for sliding context window; use Control objects from Kosinkadink/Advanced-ControlNet nodes." which is weird since I have Kosinkadink's model installed. Have you experienced this error as well ?

You made a video (I think) about unlimited animatediff length animations. How would we incorporate that into this workflow so we can go past the 120 frame limit?

Great video! Just you know: the models in huggingface are free to download, no need to open any account

Thank you so much for sharing your workflow! Is there a way to bypass the VAE nodes in order to use it with Models that don't require a VAE (such as Dreamshaper, EpicRealism)?

hey, why i got noise with artefact on my output? i follow you

have you tried this workflow + IP adapter, it will increase memory significantly

can you do cli prompt in comfyui? great tutorial btw!

Can you use RunPod or does it have to be local?

how to apply more than 1 controlnets?

does this work with AMD RX 6700??

The Node 'realistic lineart' node is always missing when loading your CSV. Any tips of what I could have missed? I am warned about "LineArtPreprocessor" missing and then in the install manager I only see Fannovel16s which is already installed.

i upgraded to 3090ti 24gig.how much cpu ram i need to do video to video SD? I have 32gig

Yesterday all is ok and today I have this error: Error occurred when executing KSampler: unsupported operand type(s) for //: 'int' and 'NoneType' File "C:\Users\lenin\Downloads\ComfyUI_windows_portable\ComfyUI\execution.py", line 153, in recursive_execute output_data, output_ui = get_output_data(obj, input_data_all) Please help me, learn this is really frustrating many times, but Im love it!!!

Hello! Used your tutorial and workflow but duno why, my video is crap :) The background is modified and is cool, but my face is still like original video with some modified colors. If I want to make my face a robot for example, just won't work... With openpose instead of lineart, i got great results but is missing mouth movement when I speak If I use same prompt in img2img, results are amazing

How are you only using 11GB of VRAM? Mine goes above 24GB and has to use non-GPU RAM...

bro it's been a week, where's some new vids, eagerly waiting

Can this alsowork with automatic1111?

hey nice video 🫶 where to get ttn text node?

Can we do image to image??

Is anyone using AI to generate workflow for Comfy UI?　Please let me know if you know of any useful links.

Awesome tutorial. I'm using the Controlnet set for the next one. Here's my latest video: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-xqRcZ2RGS6s.html

please tell me why I got this error?? when I launch COMFYUI... D:\comfuUI\ComfyUI>python main.py ** ComfyUI start up time: 2023-10-17 05:30:32.177484 Prestartup times for custom nodes: 0.0 seconds: D:\comfuUI\ComfyUI\custom_nodes\ComfyUI-Manager Traceback (most recent call last): File "D:\comfuUI\ComfyUI\main.py", line 69, in import comfy.utils File "D:\comfuUI\ComfyUI\comfy\utils.py", line 1, in import torch ModuleNotFoundError: No module named 'torch'

My original input video was 7 seconds and the output video I got is slower which is 12 seconds. Do you know why?

Hey when i drag from the pre text input im not getting the ttn text node option. What am i missing?