Miles Cranmer - The Next Great Scientific Theory is Hiding Inside a Neural Network (April 3, 2024) 

Another banger from youtube algorithm

It seems like very powerful idea, when AI observes the system, then learns to predict behaviour and then the rules of this predictions are used to delivery math statement. Wish the authors the best luck

00:00-Introduction 01:00-Part I 03:06-Tradititional approach to science 04:16-Era of AI (new approach) 05:46-Data to Neural Net 13:44-Neural Net to Theory 15:45-Symbolic Regression 21:45-Rediscoverying Newton's Law of gravity 23:40-Part II 25:23-Rise of foundation model paradigm 27:28-Why does this help? 31:06-Polymathic AI 37:52-Simplicity 42:09-Takeaways 42:42-Questions

Amazing talk, and great Research!

Love the definition of simplicity, I found that to be pretty insightful.

It is precisely what I'm working on for some time now, very well explained in this presentation, nice work! (the idea of pySR is outrageously elegant, I absolutely love it!)

What an amazing fck of presentation. I mean, of course the subject and research is absolutely mind-blowing, but the presentation in itself is soooo crystal clear, I will surely aim for this kind of distilled communication, thank you!!

been in the rabbit hole lately so glad this popped up you rock miles!

So here we are, you guys seems to be chosen by algorithm for us to meet here. Welcome, for some reason.

This is a very nice idea. I hope it will work! It will be very interesting to see new analytical expressions coming out of complicated phenomena.

I came here to read all the insane comments, and I’m not disappointed.

The folding analogy looks a lot like convolution. Also, the piecewise continuous construction of functions is used extensively in waveform composition in circuit analysis applications, though the notation is different, using multiplication by the unit step function u(t).

This is SO cool! My first thought was just having incredible speed once the neural net is simplified down. For systems that are heavily used, this is so important

It makes intuitive sense that a cat video is better initialization than noise. It's a real measurement of the physical world

Jesus christ, okay RU-vid I will watch this video now stop putting it in my recommendations every damn time

There are multiple different awesome ideas in this presentations. For example, an idea of having a neural net discovering new physics, or simply of being the better scientist than a human scientist. Such neural nets are on the verge of discovery or maybe in use right now. But I think the symbolic distillation in the multidimensional space is the most intriguing to me and a subject that was worked on as long as the neural networks were here. Using a genetic algorithm but also maybe another (maybe bigger?) neural network is needed for such a symbolic distillation. In a way, yes, the distillation is needed to speed up the inference process, but I can also imagine that the future AI (past the singularity) will not be using symbolic distillation. Simply, it will just create a better single model of reality in its network and such model will be enough to understand the reality around and to make (future) prediction of the behavior of the reality around.

I was wondering or missing the concept of Meta-Learning with transformers, especially because most of these physics simulations shown are quite low-dimensional. Put a ton of physics equations into a unifying language format, treat each problem as a gradient step of a transformer, and predict on new problems. In this way, your transformer has learned on other physics problems, and infers maybe the equation/solution to your problem right away. The difference to pre-training is that these tasks or problems are shown each at a time unlike the entire distribution without specification. There has been work to this on causal graphs, and low-dimensional image data of mnist, where the token size is the limitational factor of this approach, I believe.

This is a brilliant idea. I hope this goes places

All i always wanted to hear is in this video ! thanks !

Amazing lecture!!

Great presentation! My main takeaway is that we need a more unified approach to neural network models. Interoperability is important and can substitute for or even supercede the quality increase of pre-training.

Incredible lecture

Being able to derive gravity laws from raw data is a cool example. How sensitive is this process to bad data? For example, non-unique samples, imprecise measurements, missing data (poor choice of sample space), irrelevant data, biased data, etc). I would expect any attempt to derive new theories from raw data to have this sort of problem in spades.

Grate path to walk on .. wish luck to the lecturer and hiss fellow researches

Awesome explanation

Very cool visual at 28:12 - where would harmonic analysis fit?

yooo, i'm so glad i came across this! i've been thinking about how neural networks can teach us about our own thinking and pattern finding; i'm glad there is discussion about it

Nice! I interpret this as, “these are the standard models - we can use them to kind of explain why AI is growing so exponentially in languages we can’t even understand, but really - we have no idea what’s going on and this is why to complex for our linear models.”

This is the first exciting concept I’ve heard in the current AI revolution

This is actually really important

interesting! was just learning about neural networks, so this is a pretty cool application :)

This was amazing-- confirms my suspicions.

This is some ingenious work

Cool idea! Essentially, we can deduce symbolic, testable scientific theories from deep learning models using things like PySR. Making foundation models (which are trained on a wide variety of phenomena, not necessarily related to the area of application) for specific scientific application gives ANNs an advantage. Simplicity (explainability, legibility) comes from familiarity with a problem area, so we should be training models on lots of diverse examples to help them “get used” to solving these types of problems, even if the examples may seem irrelevant (cat videos & differential equations 🐈) Interesting application of explainable AI 🎉 Congratulations on your research

There's a paper on Feature Imitating Networks that's gotten a few good applications in medical classification, and subtask induction is a similar line of thought. FINs are usually used to produce low dimensional outputs, but I was thinking about using them for generative surrogate modeling. FINs can help answer the question of how to use neural networks to discover new physics. An idealized approach would turn every step of a coded simulator into something differentiable. It occurs to me that the approach of this talk, and interpretability research generally, is essentially the inverse problem of trying to get neural networks to mimic arbitrary potentially nondifferentiable data workflows.

Wow this is incredible and sort of confirms some thoughts I’ve had about neural networks and the compression of knowledge.

Briliant ideas

Training LLMs on code doesn't teach them to reason a bit better, it teaches them to reason a LOT better. It makes sense if you think about it: what do you learn when you (a human being) learn to write software? You learn a new way of thinking.

I am re-reading once again the book By David Foster Wallace History of Infinity. There he describes the book by Bacon Novum Organum. In book one there is an apt statement that I would like to paste 8. Even the effects already discovered are due to chance and experiment, rather than to the sciences. For our present sciences are nothing more than peculiar arrangements of matters already discovered, and not methods for discovery, or plans for new operations.

Fantastic work, I thought we would take AI in this direction and here we have that reality.

At 17:53, he has a plot on the right side, but he seems to attain only an expression in the variables x and y. There is no equation, so how is he even able to make a plot against those 2 variables? If you try plotting some of the given expressions by equating them to a constant (e.g. 2(x+sin(y+1.3))=3 ), you don't get anything that looks like his plot. If there is a 3rd variable (e.g. z, or something like f(x, y)), then the plot should be a 3D plot. Instead, the plot is 2D.

I can't shake the feeling that someone is going to train an AI model on a range of differently scaled phenomena (quantum mechanics, atomic physics, fluid dynamics, macro gravity / chemical / physical dynamics) and accidentally find an aligned theory of everything, and they'll only end up finding it because they noticed some weird behavior in the network while looking for something else. Truly, "the greatest discoveries are typically denoted not by 'Eureka' but by 'Hm, that's funny...' "

35:21 Good pretrained in some epochs by using Polymathics results does not mean training from scratch has a worse error. It is just a matter of time the good model will have the same quality.

Fantastic. At 55 minutes though, it is suggested that we don't have a simple concept like + built into us. Perhaps not in a blank neural net, but we for example are not born with a blank slate. It is clear that any toddler understands in some way, the concept of 'more' and 'less' even though they lack empirical understanding. With sufficiently robust generalized data sets based on physical principles, information theory as language and perhaps even the nature of emotions, given enough GPUs to sustain large inter-operational neural nets, would this not give rise to something more than the sum of it's parts?

Could you pre-train some layers (i.e. turn the standard activation functions for a few layers into pySR estimated functions) as a way to increase/change the dimensionality of the input data? Possibly could decrease the number of layers needed or time taken to train the network. If not run early training with parallel genetically pruned custom activation layers to approach the space from different paths while trying to find the minimum loss.

Regarding simplicity: I think that you are missing something important about the addition operation that makes it "simple". We are also familiar with division (the arithmetic operation) and it is also useful, but we would not say that division is "simple" in the same way the addition is simple (or we would say that addition is simpler than division, even though both are "familiar" and "useful").

Great presentation. Its marvelous to see a take on AI from a broad, scientific/mathematical perspective without too much focus on technicalities. Really exited to see how this might improve or add to our understanding of the/(this?:) ) universe.

this was very interresting

This is the reason why I like RU-vid

Wow!

The 'Avada Kedavra' potential of that pointy stick is immense. Brilliant presentation.

So am i the only one that going to point out that SORA from OAI is basically a generalization for a 3d engine that might let us preform experiments!

Slime mold is my favorite way of imagining it

Well not sure this will go anywhere except maybe modify some of our archaic equations for nonlinear terms. The problem is probably related to NP hardness and using more expansive nonlinearity methods to crack certain problems that are more specified. We will always not know what we don't know. Using more general nonlinear models was bound to greatly improve our simulations. The real question for NN is this the MOST ACCURATE or most INSIGHTFUL and BEST of nonlinear methods to do so? Somehow I doubt this, but it's certainly a nice proof of principle and place to venture off further. To put all our faith in it might be a mistake though. We might be looking at long predicted by mathematicians limits to reductionism, and our first method to not overfit billions of parameters will give us an illusion that this is the only way, and we could be looking at a modern version of epicycles. If we want to really go further we need to use such models to not just get better at copying reality, but finding general rules that allow it's consistent creation and persistence through time. Perhaps one way to do this would be to consider physical type symmetries on weights.

I didn't get what is the reason to use symbolic regression. Analytical relationships/models are not the same as symbolicly representables. "Derivability" is required.

model mining? brain digging? fascinating, i guess we gonna need some tools to uncover these gems from the nural nets or do we need to build the nets/models in a more comprehensive way?

I would love to see some breakthrough in Dark Matter regime. There is so much data regarding Dark Matter yet no theory to back it up.

Polymathic AI 🤖 is a wonderful idea 💡

I'm going to state the obvious. That is smart. Yes it draws questions about AI explainability regarding deep learning NNs but what this chap is saying is quite brilliant. For me, as long as the conventional approach is combined with the model he is propounding, there should be some excellent science out of that. Then there can be even more science when we start to understand the reasons and mechanisms by which the deep learning neural networks some humans build are doing and are capable of what they are so. Let's not miss the point of what he is saying, at least what I interpret that he is saying... The NN is finding some order through patterns, it really is those patterns that are probably most related to something interesting, ie of scientific interest, then we can sift through the rest of the noise to see if something was missed, let's say we do that if questions are presented that don't have an answer. So all in all, it is a very powerful way of cutting through the fluff. If we then want to scientifically describe the fluff itself, it is now more distinct. I think what this guy is saying is brilliant. Incidentally, I think we ultimately find out that deep learning neural networks come to sensible decisions because the have the fidelity to tap into the innate intelligence structure of reality itself, but that is a next topic, although entirely pertinent.

I see similarity with physics informed neural network especially with Sparse identification of nonlinear dynamics (SINDy)

The issue is discovering the higher-ordering principle which subsumes a continuum of self singularities and discontinuities. Linear math works well in-between the singularities, but cannot extrapolate through them, in a sense they are like mathematical worm-holes. Attempts to linearize across the discontinuities will fail. A whole harmonically-related series will only be properly understood from the perspective of a higher-ordering principle, similar to the idea of projection from a higher magnitude to a lower dimensional space, or from the idea of negative curvature. The point is the epistemological assumption of a static model is problematic, the real world has static islands which are bounded within areas of great change, and so the basic function changes completely there, that is to say, the dynamics of change themselves change. So to bridge that gap you can’t just ignore it, or flatten it, you have to seek how to remap it in such a manner that it is no longer infinite, but cyclical, as Gauss did with the complex number domain.

No feasible for UHDLSS Feature Selection.

So if you sought to get what an LLM knows out into some equations we could understand, what could they be like?

Simplicity is the absence of relative complexity.

The "folding analogy" is incorrect. That is not how composition works. It works only in this case because of the very specific nature of the "first layer"(in his example).

36:36 becoming more basically intelligent because of understanding spacio temporal connectivity. The flashing faces in peripheral vision illusion it shows us The monsters we create when we lack that.

I was pretty surprised to see this not actually purpose much of anything other than using tools to analyse patterns, three same tools that have been in use for decades. Is this a venture pitch? Throwing more processing at it helps, but doesn't "solve" anything on its own.

Yes AI is definitely faster generating random ideas, and is also quicker fitting these random ideas to a data set. It’s a very powerful tool.

Yea it’s great if ur trying to use a system In a system to garner traction

Add some thermodynamic constraints?

You might want to think about simplicity in terms of Kolmogorov complexity e.g your NN should try to emit the least complex, in the Kolmogorov sense, syntax tree. Also, I think "+" is simple because it is closed over the field of integers. I think that if your operation takes you from one domain to another its more complicated. In that way you might consider using Category Theory. You could think about penalizing models that "move' further away into other mathematical spaces from a 'base" space.

35:16 Yes, doing the model from scratch with traditional machine learning is worse compared to the pre-trained generative network, but only for the *same time frame*, if you give the traditional machine learning approach more *time*, then it can out-perform the pre-trained generative network, while the pre-trained network will just keep on spitting out the same type of results.

Here's an April fools episode idea. Tales from the dark side episode where you discuss the extremely creative solutions you had to come up with while trying to do chase films on some UFO test object. Deliver it in a straight face with digressions on how it would be cool if / or how you are waiting to see the technology make it's way into the civilian market ect . For the photos have blurry images of birds or maybe say that the reason pictures are blurry is because UFOs are just blurry. Even when in the hangar

someone so smart, only listenable at 4x

This is very trival knowledge if one has an open mind, but its great that it is now formally been empirically proven for those out there that need proofs.

I don't know about all the fancy stuff but as a programmer this makes me 30 to 50% more productive and my daughter, who is a manager, makes her about 10 to 15% more productive.

Specific density squared ; Volume Quebed ; Vice Versa und AUgmentation Cycle

Did you see the Lifestyle Trader ad? Proof that money is not just a commodity but logarithmic.

Is this folding similar to convolution?

Well more fun than AI churning out what are the most lethal substances. Modern nerve agent analog were in the outputs.

As an artist using image generation models, it's become obvious that foundational models trained on very wide content perform much better in general. It's similar to an artist drawing nudes and studying skeletons in order to draw fantasy characters better. It's also been shown that newer foundational models that have their dataset neutered do not perform as well, even though they might be higher resolution, or generate more detail. This is why I think it could be argued that training is transformative and falls under fair use. Unfortunately the marketing has been centered around making images that looked like other people's work (copying) which is a mistake. This has attracted people to file lawsuits against AI companies. This could be mitigated if AI companies worked closer with actual artists in order to better understand the creative process and how that relates to presenting the technology as a tool for artists, similar to how this presentation is illustrating how to use these tools for scientists.

Is the part in 12.40 just convolution or am I just dreaming?

it's seen it before so it pattern matches and i think it will be useful to scale up and pattern match and have other people pattern match and we can train it generally and scale up and train it generally and fluid simulation and we found matching outperform train it on more data and it does better the title of this video overdelivers

Great

Fine-tune an LLM to interpret neural nets. Iterate and maybe symbolic regression (i.e. language) will help us supercharge LLM training. But hallucinations could be a major issue...

With enough inputs you can make any curve or field match the current data. So it this even science ? I am very skeptical. It will provide very little real insight, when you have inscrutable AI model able to predi t something. It might as well be the oracle of Delphi.

This has crossed my mind and this is exciting indeed. High dimensionality patterns are often hidden but the fact that they are high dimension makes for the discovery of robust natural laws. We are in need of territory. We no no longer have to rely on empirical, philosophical or mathematical models to create natural laws. Data in high dimensionality can reveal many laws. Exciting times!

Reminds me of a sociology paper with tons of seemingly complex math that, in the end, says something like, "school bullying is exacerbated when it goes unaddressed." So what was all the math for? Credibility.

That notion of pre-trained NN's discovering new mathematical operations and generalizations is so fascinating! It's so difficult to imagine there would be huge conceptual holes in our version of mathematics, but there's no reason why they couldn't exist! They're probably already there in our foundation models, just waiting to be discovered!

Symbolic Regression is starting to catch on but, as usual, people aren't using the Algorithmic Information Criterion so they end up with unprincipled choices on the Pareto frontier between residuals and model complexity if not unprincipled choices about how to weight the complexity of various "nodes" in the model's "expression". A node's complexity is how much machine language code it takes to implement it on a CPU-only implementation. Error residuals are program literals aka "constants". I don't know how many times I'm going to have to point this out to people before it gets through to them (probably well beyond the time maggots have forgotten what I tasted like). This whole notion that "+" is just what we're used to is intellectual poison.

broadly useful algorithms across different systems = mathematics

For "+," I do think it is simple because I hypothesize that the human brain does have built-in neurons specifically for counting small numbers (usually 5-9 varying between persons), so when you are an infant, you don't actually need to learn to count objects under this number (I suspect that in certain area of the brain, likely hippocampus, there are this amount of special neurons that are served as synaptic placeholders for the visual cortex in object identification. Then, it serves as the starting point to further learn the abstract concept of "+." That is also why "+" is the first mathematical operation that most humans (if not all) learned. If nothing is built-in, I wonder if someone can teach a human multiplication without them knowing addition. This experiment would be highly unethical, tho.

Serious questions here, isn't his "folding analogy" just superposition of waves? Or I am missing something?

30 mins to say that you can fit simpler models to a neural network data-generating process, and another 30 to say that more training data (even if relegated to what we call “pretraining”) improves performance. ps: things are simple because they are ubiquitous and they are ubiquitous because it’s how the world works (law of conservation of mass and energy, i.e. addition), not because it’s “useful”

Great idea. By the way he sounds like a science-oriented version of Mark Ruffalo

Folding is an analogue to reducing dimensional complexity.

Interesting! How could neural network be an empirical finding? It’s not sth tangible that we can see or touch. It’s hard to believe that AI is developing the way fluid dynamics did.

the empirical fit part was a bit of a thinker, huh

Could these models apply compression to themselves through techniques like quantization, pruning, and knowledge distillation becoming faster and faster and smaller until AGI emerges from a phone sized device which can invent warp drive?

38:18 Why is “+” simple? Well, maybe because it is closely related to the concept of counting and “having” things. If I have two apples and I get (add) three more, I can just count the number of apples to verify that I now have five apples. That has got nothing to do with the concept of simplicity. Not sure if I even want to continue watching the whole thing…