Phi-3 Medium - Microsoft's Open-Source Model is Ready For Action! 

Of note, I recently watched a video by the AnythingLLM chap and he said he was using llama3 8B but emphasised that, for good results, you needed to download the Q8 model, not the Q4 as Ollama defaults to. Myself, I use Q4 on my inference server for larger models but my workstation is faster and runs Q6 at acceptable speed. He said if he was running llama3 70B, he'd download Q4 and "have a good time", but for smaller models where they're less capable, you want to limit compression. He also said it's a "use case science" which makes me think you have to test out what works for you. The Q4 model I have on my server is based on Mixtral 8x7B and, for my use case, is proving to be better than GPT4o, which is stunning. What's amazing is that, for my core business stuff, I still haven't found anything better than Mixtral 8x7B for balance of speed and performance.

Yeah, a video would be great! I read papers about this a year ago, and the drop to 8-bit is very minimal, drop to 4-bit is reasonable, and 3-bit or 2-bit quantization and things get much worse. Of course there are different ways to perform quantization, so this may have improved. I've tried comparing 16-bit and 4-bit models, and usually the difference is much much less than an 8B parameter model and a 32B parameter model. This is probably why NVIDIA's newest GPU's support 4-bit quantization, and I tend to run everything using ollama's default 4-bit quantization, though for Llama-3 70b or mixtral 8x22b this is excruciatingly slow on my laptop with 16GB of VRAM and 64GB of RAM. I rented a machine with 4x4090's for a couple hours and they ran reasonably well with 4-bit quantization, but 10% as fast as groq (note the "q").

@@ts757arse Not sure "Have a good time" is an objective measure of efficacy though. I'd say given the type of technology the results are very sensitive to use case.

@@ts757arsethanks for your testing and advice. I am now experimenting with Mixtral7B 4K. This is all a bit new to me, but it looks great so far

@@mickelodiansurname9578 nope, it's not particularly empirical. I think he was making the point that you're messing around at that point and making so many compromises that it's a bit of a laugh. Or he might have been saying with such a large model, the compression has less impact. Regardless, I've found llama3 to be *awful* when running quantised models and I simply don't bother with it at the moment. Given his advice, I'm going to try the 8B Q8 model as a core model for a new project but I'm also building it to easily move over to Mixtral if needed. I tend to run a few models doing a few tasks at the same time, passing the tasks between them and so on. Helps having a server to run one model on and a workstation with many cores and all the RAM. What I'm seeing at the moment is a lot of models aceing benchmarks, but then being utterly dogshit in real world use.

It would be hard to compare unless it was something like "Write an answer with 7 words" and the number "7" was randomized.

Yes. Exactly.

Yeah Q4 is usually pretty balls. Every time I check quant benchmarks q6_k seems to be the sweet spot.

Yeah. It would be great to have a clear overview always on how much vram each model needs and if quantized model is used: how much it is gimped compared to full model.

Bro what is quantization

@@braineaterzombie3981Reducing the numerical precision of the weights from 16-bit (usually) to just 4bit or even less with some function. Is like rounding the values.

@@Nik.leonard oh ok . Thanks for information

It's a tricky one as some models perform terribly at Q4 but others are great. I think stepping up the quantisation if he gets weirdness like the CUINT issue would make sense, as it'd show if it's a model problem or not. Ollama defaulting to Q4 blindly is kind of annoying and it's not immediately obvious how to get the different compression levels. LM Studio is great for this.

@@ts757arse Exactly my point. I know Ollama default on Q4 so it's better to use one level of quantization. I don't like when people test the unquantized versions because most people would not run them, but a Q8 is a good level.

@@abdelhakkhalil7684 Just been reading someone else saying Q8 is hardly distinguishable from the 16bit models. Interestingly, LM studio makes it seem as though Q8 is a legacy standard and not worth using? I'd prefer ollama to make it clearer how to get the other quants. It's fine when you know, but I've literally just figured it out and can finally stop doing it myself!

I once had an LLM write me a contract where the first letter of every line spelt "DONT BE A CUN" (no "I", additional "T"). Got it first time and I sent it to my client.

I cackled when he said that 🤣🤣

@@rocketPower047 haha yeah I like his side note comments like that lol. The second he started spelling it out CU..I…NT I thought exactly what he said in real time

Because that's how most people will use it I'd guess. Myself, I'd not be watching a video about unquantised models as they'd not be of any relevance. I think he should, when he finds this kind of issue, try Q6 or even Q8.

For sure

Do research. Your machine can easily run q8 or fp16 model with full gpu offload with super fast speed using cuda

What did you think? Felt underwhelming for me

It's a dumb question. A dead killer is still a killer

It is available on hf

just use llama 3 8b it's a lot better

@@adamstewarton in gguf format? I don't believe so...

@@lalalalelelele7961 there isn't gguf for it yet. I thought you were asking for the released model.

ollama run phi3:14b-medium-128k-instruct-q4_K_M

@@elecronic does that adjust the default context size?

@@rousabout7578 exactly - and it's not only in english

He did say he was using a Quantized model and thru Ollama, which I swear has inferior quantization method or process

Framework? Config? System prompt? Parameters?

@@AizenAwakened yeah right

​@@Cine95 Try saying something helpful, like what software you're using to get better results

8b*