#69 DR. THOMAS LUX - Interpolation of Sparse High-Dimensional Data [UNPLUGGED] 

I really wanted it to go on for a solid another hour! When are we having the round 2?!

Really enjoyed this episode! Interesting topics, and lots of insights

Sounds like automatic differentiation is going on the list of talk subjects lol great episode MLST, I really got a lot of geometric insights into ml out of this one.

Nice. 😆 Happy to answer questions or talk details here in the comments everyone! Also I wrote a reddit post about topics related to this talk just now, feel free to interact there too. www.reddit.com/r/MachineLearning/comments/tcott9/d_is_it_possible_for_us_to_make_fixedsize/

Extremely intriguing! Thanks for sharing!

Another thing I’d like to share here is related to the idea of “truly making progress in AI” and it dovetails on the discussion of current neural networks being “slightly conscious”. We’re missing two major paradigms altogether that are present in every known conscious being, notwithstanding panpsychism. Namely continuous closed loop autonomy of learning and predicting. We exist in a constant state of simultaneous perception and action and learning with a purpose whereas today’s function estimators are in a read only state when fed and must be fed manually by a user rather than on their own accord to pursue a natural objective. A closed loop is important for sentience, and today, feed forward passes of any model are certainly not that. The second factor is plasticity upon stimulus. A major difference between human and artificial neurons is that human neurons alter connection weights when stimulated, not just corrected or updated by targeting error. So, really we’re missing two entire paradigms!

This is what I mean when I say keep the technical depth. Good 👏 stuff 👏 I feel like y’all have been collapsing noisy dimensions down to key understanding over the last year or so, and we’re about to plateau on a shiny polished grokking that’ll be hard like diamond to crack further. Have to go looking for new gemstones after that 😅 Really neat hearing from Dr. Lux (megaman much?), this no nonsense mathematical approach is kicking butt.

Very nice episode, as usual. I have a question for Thomas: I AGREE with his thesis that the continuous subsumes the discrete and so in theory, these models “contain” the solution and they can, again in theory, get at the almost discrete solution. But, who said this is easier then formulating the discrete solution in the first place? What I mean is this: carving out the discrete the solution from the MASSIVE continuous hyperplane might be, LITERALLY, like looking for a needle in haystack, where the haystack in the continuous manifold and the discrete solution is the needle (literally). In other words, the “search” for the discrete portion might be effectively non-computable, espeically that for many discrete solutions we cannot except any "fat" - the function is strict in the sense that it will not accept any "additional" approximations. So, the fact that the continuous “includes” the discrete should not give us any comfort. I hope I can get an answer on that.

I really enjoyed this, particularly because there was collegial debate throughout, rather than flat acceptance. Dr. Duggar specifically gave appropriate pushback. It reminds me of my grad school days and that was appreciated. A few thoughts: 1) If you want to learn from the best on Delaunay triangulations and combinatorial geometry more generally, please pick up Herbert Edelsbrunner's "Geometry and topology for mesh generation" or if you have more time and mental bandwidth, "Algorithms in combinatorial geometry" (the latter crushed my mind on many occasions). 2) I realize Euclidean space is a good approximation of reality in many instances, but think we should avoid framing problems in terms of any specific metric space for as long as possible, and should certainly not start with the assumption that we're in E^n. (Unfortunately, the Delaunay guarantees are only in Euclidean space, so I suppose if using this triangulation approach, it forces the Euclidean constraint.) 3) Dr. Lux made a comment (16:54) about how the real world is 3+1 dimensions, so the 1,000,000 dimensions of our function was not a result of the inherent dimensionality of our image space but a necessary tool for sampling theory. I disagree in part. The coordinate system of the world + time may be 4 dimensions, but that does not account for anything in the world, the many possible states, or actions we can take in it. The data we're analyzing is based on sensing states of things in the world; it's not based on an empty coordinate system. My linear algebra professor once posed "How many dimensions is a tank?" I said "three" to which he replied, "Does that account for the angle or height of the barrel, whether the barrel is loaded or not, whether it's moving forward or backward, etc." That's just one thing in the world. 'dimension' and '|basis| of our coordinate system' are not equivalent. I'll concede that our function's parameter space has natural redundancy and that plays into sampling theory in part, but the world together with the stuff in it in their many states, taking many possible actions is way, way more than 4 dimensions. 4) Dr. Lux's comment about the order between positional and apositional learning was fascinating, and I'm glad Keith pushed him on this point, because I would have dismissed his seemingly arbitrary choice on architecture had he not explained further. 5) All of this made me think Stephane Mallat, who continues to dedicate time and research on understanding DNNs, would be a great guest. 6) The discussion (14:26) on data density in relation to the varying surface made me think of UMAP which uses some clever Riemann Geometry to stitch together localized metrics. Perhaps Leland McInnes would be another great guest. Really enjoying the UNPLUGGED format. Thanks!

Great episode!

Additionally, data augmentation sometimes creates new invariances. For example, masking tricks represent exogenous intentions to induce invariances not contained in the original data in the first place.

Thanks!

At 34:00, maybe I am mistaken but that sounds like a convolutional conditional neural processes model.

Bookmark for self 14:30. Optimal sampling strategy.

Please do the episode on forward gradient. It will be very interesting.

25:00 I think a transformer is a performance optimization on top of an RNN. Just my noob opinion 30:00 LOL, well yes, every single node in a network is an abstraction. Technically. Abstraction is the essence of intelligence.

First!

3rd