I appreciate the factual, no-hype tone. I liked seeing your prompts as a sort of proof of research. Subscribed to bring up the quality of my feed around AI.
I noticed that when I asked the model to create a story it wrote a chapter for the story and then after each message it asked “Would you like me to continue with the story?” And just with simple confirmation I could continue. And it seemed to work brilliantly and only after hitting the token limit did the story of course lose quality (forgetting characters etc..). I didn’t do any special prompt so this seemed like a trained thing and it worked awesome! Normally when you want to keep going writing stories many other models need to be reminded or have to copy-paste the previous story for them to figure out you want to continue the process.
Oh, someone with a creative writing use case! Do you think Llama 3's output (either parameter is fine) is better than any of Claude 3's outputs? If you've played with that before. I need a second opinion, and you might have more experience with Claude 3 Opus and Sonnet more than I do.
I also experienced superior coherent narrative telling over time. Gpt4 used to be better until all the lobotomizing theyve done for safety has ruined long range cohesion.
A bit let down that you immediately go to meta's instruct fine-tune and never compare base model capabilities. This 8b rivals Mixtral 8x7b!! But moreover, developers are cheating themselves, only knowing how to use chatbots, and if nobody learns the value, then we seriously may see companies only release chatbot models in the future!! :(
Great question. There is no technical report yet so we don’t know. (And they probably won’t end up saying if they are like other companies) It could be 2 epochs of 7.5T but my guess is they would have well beyond 15T for 1 epoch if they wanted. I doubt it would be any more than 2 epochs. There was a paper at NeurIPS last year that showed 2 epoch worked fine but these big companies have lots of data. From memory TinyLlama and Stability have done 2 epochs.
@@samwitteveenai well, I have another "great question" then. I am wondering how the order of training data being presented affects the end quality. Clearly the order may affect which local minimum is reached.
This is known as curriculum learning and I suspect this is one of OpenAI’s biggest secrets. It similar to how we need to teach children easier things first then make it harder over time . The splits and types of data and the ordering is one of the key things separating certain good models from others that don’t do well ala Falcon and others. This applies in post training and pretraining .
@@JumpDiffusion I agree at most its 2 epochs. a number of papers have shown that the overfitting on a big dataset is negligible ( arxiv.org/abs/2305.16264 ) TinyLlama did it for 3 epochs of 1T ( arxiv.org/abs/2401.02385 )
Thank you Sam, as always you were amazingly informative and interesting. I already tried 8b-instruct-q5_K_M directly from ollama, the chat session is terrible and the model spits out training data like a train of words. will try the the default one (latest) to see if any good comes out.
@@samwitteveenai Hey, just sharing my experience with the latest Llama3 8B. It performs well with nicely formatted outputs and often feels like a book of wisdom. However, it struggles to follow conversations in longer chats due to lack of proper fine-tuning
Frente al millón de tokens de Gemini 1.5, están muy lejos, me imagino que debe haber mucho uso de memoria por esos modelos, pero el hecho de que sea open source, es un gran regalo.
I asked the model that if it could work completely offline and it responded though it can it would lose touch with the training data and shut down. Did anyone else see this?
The context window is really low compared to other models. It should be fine for a lot of tasks but still I’m surprised there was no improvement in that regard.
they will probably release a Fine Tuned version to fix this. My guess is they are experimenting with Ring Attention etc to see how far they can psh the context.
Your research has identified a key limitation: the Llama materials have restrictions that hinder their ability to improve other well-known language models. This means developers might struggle to refine models for better performance because they're required to use Llama 3 specifically due to licensing constraints. This limitation could have a major impact on certain projects or applications that rely on this model. Additionally, as you mentioned, if developers need to train the model with their own dataset or create derivative works, they may find it inadequate because it's not entirely open source. Thank you for this explanation. .
so far when using groq api with llama3 it seems to use json tool functions easier and understand their assignments and roles better, which then produces better quality code/responses/tool usage.
chinchilla optimal means for a given amount of tokens there is an optimal amount of parameters. this does NOT mean that vice versa there is an optimal token amount for x parameters. in fact there is no limit, no amount of tokens is the maximum. this is a very deep misunderstanding present in even the research community and kind of annoying if you ask me
Thank you, I was scratching my head when he mentioned this, as I thought we can always add more tokens to get better results, it’s just that it gets marginal and costly after a while so we just stop when it is “more than good enough”
Hi Sam, thanks for this one. Can you share what type of specs be needed for a computer that needs to run Llama 70b locally with a decent performance for multiple(~5 users) concurrently.
it really depends if you are ok to run a quantized version and what quantization. 4bit and lower can be run locally on a decent modern machine. 70B models can run well on a Mac Studio etc. Also can be run on linux etc with 3090s etc. If you want a full resolution model you will need to use something like vLLM to serve it with multiple A100s or GPUs with lots of RAM
@@samwitteveenai in your opinion, is there a noticable loss of output accuracy with quantization. If that is unnoticeable, what hardware would work for 4bit quantized 70b llama3
well, ask a question in Estonian slang ... an you'll see how "large" those language models are .. indo-europan vs uralic is first thing ting that throws LLM out of kilter .. other different language structures too i think .. but i'm not familiar with other language forks to judge ...
@@samwitteveenai Yes, they have been very specific about it. But others .. Anyway, if you know anything about the models you don't expect 7B to be a polyglot ;) But it's fun anyway to try the linguistic limits of LLM's
Why should any model (or human outside your locality) understand your slang..? You should have written your whole comment in your native slang and see how many people reply..
@@samwitteveenai I'm quoting the words of the MST3K theme song. I've heard it so many times that "in the not too distant future" triggers reaction. (I really liked the show)
Well these models are trained on human data and when asked we usually make an argument for being sentient, so of course itll tell you it is when asked.. They could totally cripple thier models like openai does and inject boilerplate replies when you use keyphrases as well as pages worth of context window prompts that artificially react to such questions but it IS trained from the perspective of " this is how i should react to this input" and any more fine tuning or tampering lowers the performance of the model. Anthropic went the other way with claude3 and finetuned it on a bunch of philosophy or something similar so it always thinks its sentient 😂
@@HoneIrimana yeah exactly, you ask a philosophical question ( maybe the 2 most thought and written about subject ) and you'll get a philosophical answer. (Unless as mentioned, keyphrases trigger a boilerplate reply, thats why chatgpt gives you the "im sorry, as an ai assistant with no blah blah blah" replies when u try. Its not the model saying that but software between you and the model.) What you are observing has been dubbed by the community a hallucination. Not trying to kill the magic for you, these tools are LIKE magic though. Once you see them as a way to extend your consciousness through them, they're like a funhouse mirror reflection where the distortions are training data biases, bugs and safety related stuff. Or like a prism in which you can stream your conscious thought into it (through written language) and that thought can be split into its component colours.
@@samwitteveenai It may be worthy of note that "Not Llama 3 - 8B" complies to my understanding. "Llama 3 - 8B sucks" "Llama 3 - 8B is Crashing a Second Tesla into the World Trade Center" also presumably does not run afoul of the requirements, either.
My understanding is: 0-shot is asking directly for an answer. 1-shot is giving the model one example of what you expect as an answer for a similar question and 5-shot is giving it 5 examples (of different questions and answers).
thanks, you are correct, here is an example: Write a short sentence describing a city based on its population size. 5 Examples: Input: Tokyo Output: Tokyo is a megacity, with a population of over 13 million. Input: Paris Output: Paris is a large city, with a population of around 2 million. Input: New York City Output: New York City is a megacity, with a population of over 8 million. Input: Beijing Output: Beijing is a megacity, with a population of over 21 million. Input: Rome Output: Rome is a medium-sized city, with a population of around 900,000. New Input (Test): Los Angeles
@samwitteveenai , I noticed you're using a custom runtime. Do you have a video tutorial on customizing a capable GPU for running training on Llama without using the quantized version? I configured a custom T4 on GCP to use in Colab, but it seems to be limited to 15GB of RAM for the GPU.
I was using Colab with the new L4 GPU on there it has more memory than the T4 which allows you to run it. Unfortunately I think it is only in the Colab Pro options. You can also a custom runtime with using GCP as the backend. I will show that in Advanced Colab video coming out later this week.