How is your performance with the low-VRAM GGUF quantized models? UPDATE 2: Speed increased x2 with flux1-DEV/SCHNELL-Q5_K_S.gguf in comparison to the original models (tested on AMD GPU). Important, you have to start with runtime parameter ‘--force-fp32’ and although this parameter speeds up the quantized models, it slows down the original ones! T5 text encoder model has currently zero influence on my machine's performance, so I choose T5xxl_fp16. UPDATE 1: Quantized t5 text encoders are available from city96 huggingface.co/city96/t5-v1_1-xxl-encoder-gguf/tree/main. Many different sizes are available, choose Q5_K_M or larger. Place in your ‘clip’ folder. Update the GGUF node (do a 'git pull' in this node's directory), most probably you have to update ComfyUI as described in my vid, too. Replace the clip loader in your model with the new 'DualCLIPLoader (GGUF) to be found in 'Add Node->bootleg'.
this is the 1st tutorial on flux that worked for me, on my potato M1 8G in comfyui-CPU with GGUFflux-schnell and t5xxl-fp8, got images with 768x768 15min( better then nothing) lol.
This use slightly less vram (q8 vs fp8) and (f16 vs fp16) but it's not faster, as long as you don't overpass your vram pool, the speed will remain the same. Same thing for schnell, q4 render in 4seconds on a 4090, just like fp16. Unified or "baked" BNB NF4 flux models was way faster to load, but not compatible with LoRA and now considered as Deprecated.
Not sure I will try some loras this evening and in the comments it said nf4 would work with Loras,in forge, although I ve heard otherwise before. EDIT: Some work, some don't support is still in progress (experimental state), I would say it remains unclear how well nf4 will stand his ground and if it will be working with all LORAS. For low VRAM /RAM constellations without LORA it's still an option I'd say, though I cant really check for myself as I'm on the higher end of RAM/VRAM constellations.. at github you find the forge/nf4 Lora-Support thread, I cannot link here it seems.
NF4 seems to be faster and high quality. I tested these Q models and others took me 6-8 mins to generate 1 image, running rtx 3060 12Gb for a test performance. Looking forward for NF4 with lora compatibility.
@@Markgen2024 Interesting. There are different opinions circulating online at the moment, I'm excited to see where the journey goes. May I ask which resolution you have used for your tests and how long the generation takes with the original Flux models when the Q models need 6-8 mins?
I get "RuntimeError: mat1 and mat2 shapes cannot be multiplied (1x3072 and 1320x18432) mat1 and mat2 shapes cannot be multiplied (1x3072 and 1320x18432)" when trying to use these new GGUF models with Forge UI. Does it even work with Forge?
Thank you so much for this video! I only just got started with flux today, but I am already generating images with it successfully after following your guide. I am able to generate images with flux-schnell-Q4_0 in ca. 20 seconds on my RTX 3070, despite it having only 8 GB of ram!
Thank you very much for your feedback and for sharing this detailed information. I'm happy to read that you can use the quantized models with such a great performance!
@@rifatshahariyar 4GB is not much. Start with Q2_K and parameters --lowvram or --novram in order to check if it works. Then move on to next bigger models, like Q4_K_S and Q5_K_S. Both have much higher quality than Q2_K.
hi! im lost in the Zluda step, im not using comfyui portable, i have cloned comfyui-zluda from patientx, in wich folder need to pip install gguf? thx for ur videos
im on AMD, using Zluda and ComfyUI is up to date and i can see flux support in the patch notes inside comfy manager but it cannot get the dualclip into flux mode is there an extra step required that i could have missed ?
We're talking about starting with Flux and not already using the quantized models, right? Please see my vid on how to install Flux on ComfyUI: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-52YAQZ-1nOA.html
dude can u please tell me how u got FLUX working with AMD GPU, ive been dying to get it working with my 7900 XTX, is there a tutorial u followed? please let me know
@@NextTechandAI works now for some reason my comfymanager wasnt able to update comfyui to the newest version reinstalled everything and now it works thanks
How much do you "lose" using these models? Like my GPU can handle the full dev model, but its slow. Would using these models be faster but at the same quality or do you lose noticeable quality? Also what do the different Q* files mean?
To be honest, I didn't do any scientifically based research, I just compared the results visually. I generated many images with a resolution of 1280x720 and some 1024x1024. The quality of Q5_K_S was completely sufficient for me and could hardly be distinguished from the FP16 original. With higher resolutions (FLUX allows 2 megapixels) there are clearly noticeable differences and I had to use FP16 or Q8_0. However, I couldn't find any relevant difference in quality between these two. Regarding quantization, I have included a link in the description.
In general, yes. I usually move the folder outside of my ComfyUI folder structure and restart. If everything works as expected then the moved folder can be deleted.
Hi I'm new to all this - I don't see bootleg within the add node dropdown? Little help... Ah I got it the cmd didn't install on the first try for some reason :P
I am struggling a bit with Flux. I have a GeForce 3080 Ti, which is nothing to be scoffed at, and driver version 560 is installed on Windows. I tried a bunch of different workflows with dev FP8, and all of them are super slow. I only have 64 gigs of DDR5 RAM. But I haven't read anywhere that it should be a problem.
Your specs are perfectly fine if the resolution is not too high. Which resolution are you using and what generation times do you achieve? Anyhow, I suggest using the quantized models or for high quality the FP16 model, I do not use the FP8 anymore. The suggested --lowvram might slow down your machine, you have to try with different resolutions.
Definitely Q5_K_S, I have had a very good experience with it. Fast loading times with good quality. For high resolutions you could also try Q8_0, but that's not a must and it already has 12GB.
NF4 > GGUF. GGUF is slower due to being compressed and NF4 was optimized for speed. As a trainer I wish either I could use to train with and this is hideously slow being forced to BS1 on a 4090.
Please follow my guide for Comfy & Flux here: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-52YAQZ-1nOA.html. You only have to download the T5 as described there, you can select the (most probably only installed) default model for Clip.
@@iDeker Still according to my vids about 1. Zluda on SD.next and 2. ComfyUI with Zluda (based on SD.next). As far as I know lshqqytiger's fork of Zluda is still accessible.
hi! have a 6700xt 12gb vram comfyui with zluda and stable diffusion 1.0 xl when put 1024 x 1024 crash the app,the terminal show a message CUDA out of memory tried to allocate 2.50 GiB sorry my question is not from this video T_T thx for all your excellent videos.
Thanks for your feedback. Even with my 16GB VRAM, SDXL was often at its limit. You might try adding the runtime parameters --lowvram --use-split-cross-attention. Anyhow, I would use e.g. epicrealism with 512x768 and upscale according to my upscale-vid. But Flux is currently a better option, maybe one of the quantized models fits your VRAM 😉
About 355 seconds running "flux1-dev-Q4_K_S" in ComfyUI on a Mac Studio (96GB/38-Core GPU). So, still unusable for me, but par for the course because Apple doesn't care about MPS and open source.
You have to be doing something terribly wrong because I can run that same quant on forge with a 16GB RAM M1 Macbook Pro and get 90-120 sec generations.... You should be getting around 30-45 sec with a Q4.
@@jumbomeatloaf Now, there was a positive report from an owner of a 2050 with 4GB. You might try with option --lowvram or even --noram. Will be very slow in case it works.
Weeeelll i got a geforce 1080 potato graphics card... which ... only to an extend can use SDXL models but starts to struggle as soon as i integrate controlnets... fortunately we gotin our company A4000 graphic cards for our professional work.. But would i get this running also at home?
