I don't have time to go through the whole video at this specific moment, but it seems to me that you came to a fairly similar answer to myself: LLMs are pretty strong at presenting data and handling noisy inputs, while traditional computer programs are pretty good at doing the math (numerical instability notwithstanding). One obvious opportunity that I'm not seeing in the first ten minutes (though I'll certainly allow it's possible that I'll have egg on my face after finishing), is that this seems like an efficient way to embed agentic function calling into a model; if the steps to solve the problem contain a call to a remote system with the equation as an argument, and the remote system can solve the equation, that seems a lot like a function call to my eyes. Beyond that, there's also probably some room to reverse engineer a problem with the synthetic generator LLM based on the equation and answer, in order to encourage semantic problem solving, as seen in certain benchmarks which have word problems encoding ideas best solved with mathematics. Overall, this is a super cool project, and is probably going to be very beneficial for people doing continued pre-training or experimenting with certain ideas like grokking. I'm pretty excited to have a hack at it myself.
Absolutely spot on, I cover later in the video that the same technique can be used for function calling for complex expressions and can also be used for teaching code generation etc
When you said "math" I thought you meant symbolic math, not arithmetic. Using an LLM to do arithmetic is pointless; a calculator does a far better job.
@@BlunderMunchkin symbolic math is coming but you have to start with a foundation…. but in order to do symbolic math, the llm still needs to know how to count
