What is Spatial AI? "The Next Frontier of AI Architecture" 

Matt playing that video at 1.5x while I am playing this video at 1.5x

00:00 - Introduction to Spatial AI and Fei-Fei Li 00:35 - Fei-Fei Li's Contributions: ImageNet and AI Research 01:43 - Cambrian Explosion in AI: Pixels, Videos, and Language 02:09 - AI's Evolution Through Deep Learning and Data 03:12 - Early Days of Deep Learning: ImageNet and Commercial Applications 03:54 - Rise of Generative and Discriminative Models in AI 04:20 - Algorithmic Advances in Computer Vision and Language Modeling 05:16 - AI’s Power Unlocked by Large Data and Compute 06:14 - Key AI Papers: Attention and Transformers 07:28 - Breakthroughs in AI with AlexNet and GPU Power 09:57 - Supervised Learning Era and AI Data Utilization 11:22 - AI Research and Algorithmic Breakthroughs in Academia 13:54 - Evolution of Generative AI: Style Transfer and Image Generation 15:10 - Speed and Optimization in Generating AI Images 16:36 - Gradual Advancement of AI Towards AGI 18:05 - Fei-Fei Li’s North Star: Visual Intelligence and Storytelling 19:46 - Current AI Capabilities: Computing Power and Algorithm Depth 21:13 - Tesla’s Use of Real-World Data for AI Training 23:08 - Transitioning from 2D to 3D: Learning Structures in AI 24:27 - Nerf’s Breakthrough in 3D Computer Vision 25:33 - AI's Focus on Reconstruction and Generation Convergence 27:04 - AI Representation of the 3D World Through Physics and Structure 28:12 - Spatial Reasoning and Limitations of Multimodal AI Models 29:14 - Contrast Between 2D and 3D World Understanding in AI 30:12 - Processing and Representing 3D World Data in AI 31:35 - Human Perception of 3D World Through 2D Visuals 32:57 - Future Applications of AI in Virtual 3D World Creation 34:19 - 3D World Creation and its Economic Impact in Gaming 35:46 - Desire to Explore and Simulate 3D Virtual Worlds 36:58 - Impact of Spatial Intelligence on AR and VR Technologies 38:00 - Spatial Intelligence as the Operating System of 3D Future 39:02 - Ideal Hardware for Spatial AI: Glasses vs. Goggles 39:33 - Blending Digital and Physical Worlds: AR and Robots 40:06 - Conclusion: Spatial Intelligence’s Role in Robotics and AI

I watched this interview on the a16z channel. it is so amazing how much AI is making actual progress even among all the over-bloated market economics taking place right now into making anything and everything AI which is making many people wary. Well, if the internet and AI took away your critical thinking skills, AI and internet isn't responsible for it. I'm so sick of channels interviewing "CS researchers and scientists" and they spend like an hour decrying everything AI. Folks, use you own goddamned brains a little!

Please don't speed up the video. We can speed it up if we want to.

you cant watch technical interview at highh speed

Didn't feel like their call brought anything new to the table at all.

The issue for AI is having no environment therefore causing inability to access or navigate beyond calling apis. Ifigured the trick is to put a remote dpi into an app so u can display a computer environment in a window on your own desktop, then add a node interface over that and train ai on navigating the node system It works because each node is a sensor and informs the back end of pixel colour directly underneath each node in realtime and also functions like mouse clicks and keyboard inputs so the view of nodes and functions occur together, and train ai on navigating the node system with machine learning beginning with just training on a user just using the computer themselves. The ai never has to see the display because the nodes give all the info so if it can learn to navigate the nodes then it can navigate the computer without any direct connection which means jack all overhead. Not sure if people realise this but even us humans don't see the world, we see a simulation informed by signals from the senses, one type being rods and cones in our eyeballs, this node system would be the rods and cones interface that by correlating with action allows ai to link action potential with view of a digital environment. The coolest thing about this is the ai would be in an app on another computer so it could navigate the internet in the same way humans do to find the info it needs to do tasks and could build seperate apps, rebuild new versions of itself on another computer using other ai like Pythagora and test its apps by actually opening them and trying them, it could set up a system where it remotely navigates multiple remote computers at a time and trains on all of them, ai building ai, automation automating automation and automating iterations out of better interactions with manual input becoming less and less in each one The crux of this is giving ai an environment. People have tried with robots and virtual environments but why not the digital space itself, that's the first thing we need to do

So many incredible use cases for this technology. People creating their own ‘happy places” they can just put on the headset and submerge into while on a long flight.

I've done a few tiny experiments just prompting an LLM having it think of real world descriptions in terms of spatial coordinates of objects. It seemed to be a little better at solving real world 'puzzles', like what happens to a ball in a cup that gets turned upside down. Not perfect - but it did uncover some of the model's vague understanding of what a 'cup' is, for example.

OpenAI still having a context length of 128K is just a bummer. If they won't increase that, I can see them losing the high-ground regarding the reasoning models. Google's Gemini has a Context length in the range of MILLIONS and this could give them the upper hand, since reasoning models essentially use up that finite context length pretty quickly. Of course, summarizing the thought process and feeding back only the summary can reduce the amount of tokens needed, or simply making the AI think in a more efficient language that humans might not be able to understand (at the expense of transparency) could also help, but ultimately, they'll need to increase that context length. It will get ugly for them if they won't.

This 40 min long video was worth every sec of it. Thanks Matt.

fei fei is great! she have the pioneer's dna

Regarding about the image generation being more like a continuum from previous milestones but was regarded by the public like an abrupt new thing, it made me think of the quote: "It takes 10 years to become an overnight success."

Spatial AI is a thing for sure. Our world is dimensional. If an AI wants to understand everything that is happening in our world, it has to learn about all the dimensions and the physical connections. You said that, for example, Tesla has a lot of spatial data. That is correct for sure, but for quite some time we collect spatial information with our mobile devices. Think back to 2014/2015 when google showed project "tango" (the mobile AR that became ARCore in the end). What they wanted is to capture all the insides of the buildings they capture from outside with Google street maps. And we should not forget that also audio is spatial. You can capture a lot of data just by analyzing audio data (submarines do this for a long time). I am very excited about the spatial AI approach and i hope it will be available for everyone (open source). Thanks for the video, Matthew.

It seems that the practical applications of these developments are still in the process of being fully realized. I hope they are finding ways to apply these innovations beyond just gaming. With that in mind, here are a few possible areas where they might have significant impact: 1. Generating a comprehensive set of architectural and engineering designs based on site parameters and design preferences. 2. Creating 3D product designs, such as furniture or wearable technology, that adapt to environmental factors and surroundings. 3. Offering emergency assistance through augmented reality, such as using smart goggles to guide someone through landing a plane in a critical situation. 4. Enabling underwater robotic welding to facilitate complex repairs in challenging environments. 5. Utilizing autonomous drones that can navigate hostile environments and selectively target designated individuals. It might sound harsh, but it’s likely similar technology to what they would be using for shooting games.

Did you forget to link the A16z vid mate? Cant see it in the desc.

This stream of mind-blowing news just gets crazier every day... if it continues like this, you will have to switch to live broadcasting. Your insights and comments are very valuable.

How about a video series teaching us how to put our Nvidia gaming GPUs to use for AI. I know you already have tutorials on running llms locally, but I keep finding out about new features and use cases with my RTX4080, and I can't find a RU-vid channel that really focuses on getting the most out of these GPUs for AI. Chat with RTX is pretty cool for beginner's. I've moved on to OpenWebUI and ComfyUI for my llms and diffusion models. Currently trying to figure out ComfyUI properly, isntead of just using other peoples workflows and dropping my own loras in. I want to learn how to use 2-3 loras. 1 lora for creating images of myself, one for a certain style of image, and one to improve photo realism, skin tone, etc.

If we're crafting AIs in our own image, we shouldn't expect them to think outside the box we've built. Matt you should stop erpeating "real world" and wipe off that overzealous grin - this is a funeral my man. What we're teaching AI to perceive is the world we live in. Even the visual representation of local reality at the visible spectrum will be rendered differently by beings with a different central nervous system. It is okay to limit AI to human cognitive/perceptive frameworks, but don't then complain that it can't provide novel insights. This is another nail into the coffin of ever having a truly foreign intelligence, another brushstroke on the frame of the mirror we like to admire ourselves in.

An early lunch of Optimus is in my opinion also to fullfill the need of visual data of homes and what happens there, because homes are not streets.

Wearing a pony tail makes you look smarter. Just like having a british accent. I forgot, if you talk fast you are by definition a genious - so much to share with us mere mortals...

I feel like our ancient ancestors - pre-language - had a non-communicated version of language that language sits on top of but doesn't necessarily describe, even with huge language sets. We can hard code physics and things of that nature to almost literally ground AI in the real world, with being able to probe its answers for a quite material-based explanation for why it is stating something. One of the most captivating use cases for VR is just exploring and screwing around in photoreal 3d environments, "telling stories / pretending" BUT there is a huge lack of these at low price that are quality - I have to dig through about 80% detritus for the good stuff. If this suddenly wasn't the case, it would mean endless novel worlds to explore, personalized, responsive to speech, hand motions, etc. This drives mass adoption of VR, lowering the cost of high res, high FOV headsets that are comfortable, which in turn drives forward spatial AI

To go with Spatial AI, I hope they also combine it with a physical structure that not only morphs under and around you but also visually represents the Ai world from our own ocular perspective. If that can be invented in my lifetime then I will be rather pleased. I have always dreamt of having my own Holo-Deck.

That's a lot of words to say we're trying to build Star Trek's Holodeck.

I would be interested to know what the training data would be - 3D scans, dual camera or possibly control net data with depth maps and positioning info. The real challenge is as she states, a human can look at an infinite number of variations in the world and know what it is instantly (a chair for example, regardless of shape or build complexity we know is a "chair" even if we've seen one for the first time in our life, we just know it's a "chair" - for AI in 3D space this would be a challenge because it has no text cues, and manual human tagging for every object on earth is practically impossible). Tesla is limited to road data, it has no concept of objects invisible to it outside of the narrow depths of field from roads.

Fascinating. I would point out though that ADN condons are "words" in nature (not floating in the sky, but embedded in every living creature) and that they convey 3D intelligence.

verses ai is working on similar spatial projects. They created the spatial web protocol that accepted by the IEEE.

I don't understand what World Labs is up to. The concept of modeling the world and reasoning about objects, places, and interactions in 3D space and time closely relates to topology, which is the mathematical study of how spatial relationships and properties persist or change under continuous transformations, such as stretching or bending without breaking. So, they are either giving a new name to the study of topology, or they don't fully grasp what math has already covered. Additionally, using mathematical reasoning and logic, combined with Multiphysics Simulations, produces more accurate results than relying on probabilistic token-based models for representing objects.

fei fei, ilya sutskever, and alex started the ai deep learning revolution with alexnet back in 2012, they're so underrated ... to this day deep learning continues to conquer field after field, amazing

Wtf… you threw me off. Lmao Thought my youtube was playing at double the speed.

I hope they use all kinds of sensors when they build their data set, I like the idea of AI being able to estimate the temperature of things based on an image.

I am surprised they did not mention Gaussian Splatting.

Bro drank 5 red bulls before the interview😵‍💫

I already did these experiments and development in the 90's using REAL PEOPLE, as a cash Math/Physics tutor (now over 80,000 hrs). Guaranteed clients an A in Engineering Physics 1, by reverse-engineering an expert system from the Curriculum, and employing "Lazy Learning" techniques, employing the HUMAN BRAIN as my neural network. I extended the concepts to the laboratory teaching, and my class became so popular, 3 hours per week, NO HOMEWORK, that the University changed a by law to allow withdrawing from the lecture class but keeping my class. Not only did "compute" not cost me anything, people paid ME to do my compute!. Also mastered transition to Algebra 1 fluency. No one wanted it, the schools just being totally corrupted with mediocrity. So now I tutor the smart, well to do, immigrant crowd and enjoy watching the world catch up.

Meta's SceneScript is already capable of reconstructing environments and representing the layout of physical spaces. So, the question remains: what truly innovative or new approach is World Labs bringing to the table? And finally, there’s a need to reinvent Meta's Orion AR glasses and the advancements from Boston Dynamics.

This is the future in my opinion. I see this modaility taking over the 1D, 2D image modality in the near future. It'll give much more details in color and nuances to the LLM to process. It might be a little more computionally expensive but the benefits will outeeigh the risks and "costs" I think. The three modalities LLM has now will reach their thresholds at some point. It'll need to lean on more advanced technologies like this one too process more nuanced representations from the 3D or the real world. Another one is video reading. Current LLMs can't do that. It reads transcripts only not the visuals. With the camera shutter mode it's possible for this tech to read videos in 3D. Those will enrich the LLM in its representation algorithm. With a richer context, it'll reduce case of hallucinations even more. Anyways, this is exciting stuffs if you'd ask me.

Good look on this one Matt thank you!

Is everyone gonna ignore the work of Jeff Hawkins like he hasn’t been doing this for 20 years a Numenta?

iPhone have had LiDAR sensors since the iPhone 12 Pro but haven’t been utilised much, photogrammetry is super interesting

This might be exactly what you need: 1. Learn about spatial AI and its potential impact on various industries. 2. Explore resources like the a16z Podcast with Fei-Fei Li and platforms like Mammoth AI to deepen your understanding. 3. Consider the ways spatial AI can enhance robotics, augmented reality, and world generation in the near future.

The five basic human senses are Sight, Hearing, Smell, Taste and Touch. Ai will be doing that.

very interesting topic, as I work in virtual construction, obviously it goes way beyond this but just reaffirms my opinion that ppl won't be designing and coordinating much for construction for much longer.

Interestingly a paper came out just a few days ago, Robin3D: Improving 3D Large Language Model via Robust Instruction Tuning.

loving this channel. such great insight.

when you play a video at high speeds as part of a recording, you are making it hard to follow for many people. please keep it to normal speed from now on and let us pick the speed in the youtube player ourselves

Somehow we're still seeing LLMs improving a lot (say, twice a year, e.g. with 4o, 1o) with overall the same amount of data, perhaps they're already incorporating more than 1D/2D in there..

Research! Researching in 3D would be ground breaking. Everything that is important is 3D. Genetics, physics, engineering. All 3D.

whoever can crack the cheap and fast apprehension of high quality 3d structure of the real world will provide a huge advantage to many fields, this has been a huge and expensive struggle for decades now, making design, engineering and games so much more expensive.. An "unlock" like this will provide training data for calculating 3d from the billions of 2d photos, videos. Then this links up to LLMs and makes them more precise as well. Once you have a detailed and precise understanding of a 3d environment there is TONS of "free" functionality - simulate new lighting, calculate volumes, path-finding for robots, physics, etc And these tasks can be done in non-real time as needed. This leads to AI-designed automation solutions acting on the real world, deploying drones, IoT etc which then completes the loop so we get a real-time sim of the real world with fidelity proportionate to locations of interest.

except Tesla's data isn't spatial since Elon insisted cameras were enough, no?

Like I told openAI, it's a megawatt per million for your projects (mouha ha haha (the energy company rubbing its hands))

Turning on captions makes a mess when the video in the video also has them

This is the right path, Spacial intelligence with point of interest.

"around 2012 ish" Yup. But the idea didnt pop into his head.

🎯 Key points for quick navigation: 🌐 Spatial AI integrates AI with spatial data for enhanced contextual understanding. 🏙️ It enables AI systems to perceive and interact with physical spaces in real-time. 🚀 Spatial AI is considered the next frontier in AI architecture due to its applications in robotics and autonomous vehicles. 🌍 It revolutionizes industries by optimizing logistics, urban planning, and environmental monitoring. 📡 The technology enhances augmented reality experiences by overlaying digital information onto physical environments. 🛰️ Companies are investing heavily in Spatial AI for its potential to redefine navigation and mapping technologies. 🏗️ Challenges include privacy concerns and ethical implications regarding data usage in spatial mapping. 🌌 Future advancements in Spatial AI aim to create seamless interactions between digital and physical worlds. Made with HARPA AI

great commentary video thanks man

Welcome! To The World... Of tomorrow! :)

I'm now listening 3x speed of Feifeili

Gotta say im not a fan of speeded up videos. Let us decide when to speed up please. You made the content less accessible 😕🤷🏽‍♂️

actually even some of the showcased, so, somewhat cherry picked Sora videos made major spatial blunders

What's the URL to "Archive" that is mentioned in the video?

i skipped to the interview and was like wow these tech bros are definitely on adderall

On October 10th the world will see that Tesla inc. has already built a real-world AGi. You’ll see

I cant help but chuckle everytime people say about new paradigm of reasoning and chain of thought… I put together a group chat for gpt3 to discuss my prompt and witnessed the power of it a few days after chatgpt release… Am i genius or are these ‘researchers’ dumb?😅 I dont get it guys, are they slowly milking the cow that should have been a phrnomenal steak right away?

What if architecture of a model (weights interpreter) should be also dynamic and trainable via deep learning or via selection/mutation Darwin-style mechanic

While 3D can be generated, lowering the creative cost for building those environments - I’m doubtful AVP is the right device to explore those worlds. The form factor and cost is too high. (I don’t see next gen kids willing to wear those for any length of time)

I noted Justin Johnson spoke about 4D... to me, that speaks to AI dealing in a holistic view of the world, not only the basis of 3D models, but how those 3D models live in their placement within the world. But consider also, over an object's relative position in a given time in space. But I could be wrong... but that seems to be some far out Einstein Relativity thought here. What do you think?

Alexnet vs Imagenet what's the difference?

Without spatial intelligence you won't reach GAi (human level). You need the Ai to think and visualize in 3D space and time like the human mind does. Current image/video Ai generators, trained on 2D data, are pretty useless in the commercial world. You would want full control and management of your props, textures, assets, lighting, in your Scene in the x,y,z coordinates to achieve the vision you have in your head. I've spent too much time and energy trying to generate an image or animation the way I want it using inpaint, ipadapter, SAM... I'll just have to wait a bit longer.

Thanks for the video.

I love this channel!!

Very interesting, thank you.

I bet this model would be very good at the marble problem 😃

Damn ive already saw that interview.. i thought it was something new... Ok ..👍 👍

please don't set the speed for me, I do that myself.

I swear that I watched this video months ago.

They said .. plumber cannot be replaced... with spatial intelligence you can be plumber! with that device it can project AR Of pipes behind the wall because spatial intelligence able to generate 3d!, mechanic?

Mammoooot 🤣

Fascinating! 👍

Well... very informative and cool and I can see it being applied in many industries, however it feels like there is something missing. Oh yes where on earth are you going to store all that spacial data ? The next big thing would be then Storage! Not compute...

It seems to me, that this 3d dimensional architecture will turn out to be a very narrow niche approach and we will laugh at it (as well as at transformers) once agi is achieved. When a baby learns the world it only gets 2d images from two eyes, two microphones and a gazillion different touch sensors all over the body. I bet almost nothing in the brain is 3d oriented when the baby is born. AGI architecture would not necessarily incorporate 3d in hardware. I think it will be more generic. Even simpler than transformers. But will require much more compute. Imho.

This will kill TESLAS FSD Advantage

Please don't speed up videos, I can speed them up on this end if I want to.

Tack!

Basically building the Matrix

40minutes, and title is what is spatial ai, is this topic need a whole 40mimuuuuuuuutesssssss

So very fascinating 🎉

Contact lenses with AR combined with neuronal output device on your ear with mic and bone induction sound ..(since brain input is still theoreticall - and risky when it will be possible ) You would have AR and besically think what you want and it appears in AR.... ex:call somone ..time...video....see the collusem in real life and overlay as it was with gladiators and all, games like pokemon go but actually see the pokemon (Now that i think about it we might have to invent a few new names for mental problems that people will have😅)

More bubble money :) Ah it's just like watching the late 90s all over again.

I’ll pass using my data so that these people can be rich and have power😂

Normal-speed talking and thoughtful comprehension are so 'last decade'. How about compressing everything into a 1 second tik-tok ?

A huge step towards Metaverse?

Simple WE'RE F'ED - end of discussion

She sounds like she is going in the right direction

The 1.5 speed sounds terrible, like a cartoon duck or something, and earned this a thumbs down and not finishing and not sharing.🤬

Smart people

Python is dead end! One guy, which famous now, writes neural net in low-level Assembly and it's 500 times faster than Pytorch on 1 Cpu core. That insufficient power to run cumbersome high-level Python is illusion and trap! Almost a conspiracy, because everything started by leaked (not really) Meta model on which Stanford Uni by some order made Llama, on which all open models works, these require Nvidia hardware from the early start, i tested GPT2 in 2020th, it was incredibly slow already. Fundamental foundation of models are wrong, these from the start made to require horrible amounts of money & resources if we want to achieve anything useful, full rewriting into low-level languages will be more productive than trying to make this mess work in current architecture (which is valuable to Nvidia by some reason). At least we see in Python where to go, kinda a prototyping stage. Robots will able to run only on low-level architecture models, that's efficient and possible on current equipment. It will be shocking revelation that we already have too powerful hardware, we just can't write efficient software for it, like at all.

Please stop playing things fast. People are already playing things fast, so you are forcing it to be double fast. Let people control their own lives and listening preferences.

What is GodMother of AI. Can you guys stop this? AI doesn’t have god anything. Everyone in this field has helped.

Does blind people have world model ?

Helen keller