Thanks, this is a wise take! I agree, that we now can make progress on some old problems will not mean we are running out of problems, but that we are pushing the boundary of science to discover new problems.
This is how humans solve problems. We come up with a million dumb ideas, think a bit, and discard the 99.999% that are obviously wrong, and then verify the ones that aren’t. Every once in a while, we find a gem.
So... if we are back to brute force, what could be the implications of IBM's quantum roadmap they just released? I said right from the start, the hallucinations aren't a bug they are a feature. It's not a coincidence people who think very abstractly are more likely to get their wires crossed about things. You're never going to come up with a completely novel good idea without trying out 1000 ideas that don't work in the process. That's pretty much a law of nature in human intelligence, no reason why it wouldn't be a law of AI. You'll be able to make an AI that can always give you the right answer but has zero imagination and can do nothing original. And you can make an AI that will be able to come up with new ideas but 99.999 % of them will be bad.
Love mathematics and science. . . I remember my maths teacher when we worried so much about how solving problems, and he said "remember every problem contains its own solutions". How true. Cheers from Sydney '
given the correct level of abstraction its simple and seeing it alone in isolation one would probably say 'thats not creativity', but then dropped back down into reality you would be amazed at how creative is!
I keep saying that hallucinations are the best feature of the LLMs. They are exactly the last missing piece of puzzle I needed to automate discovery and invention.
I asked GPT4 for creating a prompt for FunSearch this is what I got: "You are tasked with a challenging problem: [describe the specific problem or task succinctly, including any key details or constraints]. Your objective as a pre-trained large language model is to generate innovative and diverse solutions for this issue. Your outputs will be evaluated based on [define the evaluation criteria briefly, linking it directly to the problem’s objectives]. It is essential that your suggestions are innovative and feasible within the specified constraints. This process will be iterative. Based on the evaluation of your initial suggestions, the problem statement will be refined, and you will be asked to generate further ideas. This iterative process aims to enhance the quality of the solutions progressively. You are expected to utilize your extensive training and capability to generate a wide range of potential solutions. Approach this challenge with creativity, considering various perspectives. Begin with your first set of ideas and solutions for the stated problem" Usually during brainstorming of ideas additionaly I ask for randomization of presented solutions. LMMs have a bias toward the first answer - and assess it the highest. @@AICoffeeBreak
This sounds broadly like brainstorming; you intentionally suspend criticism at first and just try to come up with many possibilities, no matter how off-the-wall they may be, valuing instead their volume and diversity, and only then in a second step you evaluate their quality. Wonder if other approaches to systematic thinking one can find in popular literature might have some lessions for ML research as well. I think I've heard of this kind of overgenerate + rerank approaches in NN papers before, though not specifically the added step of then prompting with the best ones and iterating the whole process again in that manner.
Thank you for covering this. This is a great overview about something I was curious about. I hope Ms. Coffee Bean didn't expand her mind too much with hallucinations.
Interesting times are coming in most fields, I think if we learn to use the advantages and understand our new roles, this cooperation could help us mitigate many shortcomings. You wouldn't be afraid of a powerful ally now, would you?
I think an excellent point was made, that from the brute force of learning we experience emergent knowledge that is not always a logical continuation of that learning; lateral thinking is what we're particularly good at (for now). I think offloading the voluminous learning processes to the AI, and from that experiencing our lateral aha moments, a synthesis of mathematician and AI will speed up progress. Now, I live under no delusions that this will continue as-is. AI's will eventually begin to not only make lateral inferences from unaligned learning, but they will eventually self-realize the value of lateral thinking. I hope to be retired by then, lol
Mathematicians searched for centuries for solutions of algebraic equations using radicals. They ended up with Galois Theory which explains everything but is practically of little use for solving actual algebraic equations. Maybe AI can help finding new solution formulas to algebraic equations?
I like it. It basically takes AI which is known to provide false answers and which we cannot fully understand and makes it create something that is deterministic, something that we can verify and even understand. It basically addresses the biggest drawbacks of LLMs. Future won't be asking LLM to do stuff for you, it will be asking LLM to write a program that does stuff for you. I wonder how it would do on some problems that were always thought to be unsolvable by programs like classifying Imagenet images. It was always an example of why we need AI - because otherwise we would need to code unimaginable number of rules into our algorithms. Maybe AI could also come up with that unimaginable number of rules though. Otherwise, maybe it could write a program that not necessarily solves the problem but complements the AI in solving the problem? Like an AI that creates new ways to leverage another AI or even itself to solve a specified problem. Interesting times if you have access to such compute.
@@AICoffeeBreak The emojis are brilliant! Do you have them on Discord? How come I didn't find Discord link in description on RU-vid, Patreon and your website. Is this true that you have discord and patreon benefits?
It's a selection of the fittest, but the candidate solutions do not appear to be the result of genetic evolution. Are they, or are they developed by other means? It isn't clear to me.
@@AICoffeeBreak Yeah this is typical of evolutionary algorithms from decades ago. They don’t suffer from local min/max the way that gradient methods do. They don’t even need the LLMs to “solve math”. They are just taking already existing solutions and recapitulating within LLMs paradigm to benefit from the hype. “LLM hallucinations discover new math solutions” seems a bit misleading when the LLM wasn’t what made it work, but instead the evolutionary algorithm.
I accept your point of a clickbaity title, though I argue (1) it is both the evolutionary algorithm and the hallucinations: Take the LLM outputs out of it, and you're back at a search without good solution candidates. (2) one needs catchy titles to convince the audience to watch this video instead of cute cat videos. 🐈 Once caught, the audience can be educated by the video that it is a smart search through the hallucinations and not just hallucinations. 🪤
I would like to say i bleieve hallucinations are a feature and a bug. Neither do i believe the answer to making models more accurate (this is the bug part - hallucinations can make models innacurate) is removing hallucinations, but rather, aligning them with reality / what we percieve as truthful.
Once AIs have been properly trained in mathematics they will go through it quickly solving all the problems in math remaining. Its only about following rules carefully.
Yes, there are people advocating for the term "confabulation" instead. Since LLMs are not grounded, everything they say is confabulated, it just sometimes happens it corresponds to reality.
@@Neomadra To ground would mean to have them see the same world context as you do at the moment you are asking something. A simple, short question in the form we are usually formulating to ChatGPT, such as "What should I wear this evening?" is underspecified in all aspects: where do you live and what are the customs geographically? What is culturally accepted when you mean by evening (the spanish 10PM or the German 7PM)? How do you look like and what would fit you? What I mean by not grounded is that they do not share context with you.
