How To Build Generative AI Models Like OpenAI's Sora 

Chapters (Powered by ChapterMe) - 00:00 - Coming Up 00:49 - Intro: Generative AI for Video 01:13 - Sora Videos 05:05 - How Sora works under the hood? 08:19 - How expensive is it to generate videos vs. texts? 08:55 - How do YC companies build foundation models with just $500K? 10:01 - Demos: Infinity AI 11:23 - Sync Labs' hack to train a Lip Sync Model with a single A100 GPU 12:45 - YC deal with Azure 13:41 - How Sonauto Built a Text-to-Song Model 15:44 - Metalware: Hardware Co-Pilot 17:40 - Guide Labs: Explainable Foundation Model 18:20 - Building your own models vs. Using existing open source models 19:29 - Phind's Clever Hack: Synthetic Data 22:03 - Simulating real-world physics: Atmo (Foundational model for weather prediction) 24:21 - AI in Biology: Diffuse Bio 25:36 - Piramidal: Foundational model for the human brain 27:15 - AI in Robotics: K-Scale Labs 28:58 - DraftAid: AI Models for CAD Design 30:38 - Playground going against giants and Suhail Doshi Background 31:42 - Companies pivoting into AI 32:44 - Takeaway Message 33:20 - Outro

All you're really saying here is that people can build any foundational models as long as openai doesn't also do it. That's not very reassuring to hear. We started with words, now pictures and videos, why would anyone not expect music, robotics, hardware etc down the line?

The lipsynching on Tim Ferriss looked way off. There was a bit of an uncanny valley with the deepfake switchover as well.

I've been waiting for a new episode for weeks!! Thanks for the content guys!

20:15 I personally find the concept of synthetic data to be a fascinating spur for more neuroscientific research. People dream about what they study and are constantly reviewing problems they are working on in their head. In other words, I feel that humans use simulations in their own mind to build out the models they use to understand their world. We might be able to think of this as "generating 1000x more data" than can is directly extracted from the real world. Another example of this that was done to awesome effect is AlphaGo's self-play training.

The text/prompt for the video was quite detailed n informational. Even as a bad programmer I was able to mentally construct an algorithm for a video on the fly...maybe I have to watch this podcast more than the first 5 mins to understand why Sora etc is a big deal...

Thanks for sharing this talks!

11:40 Hindi demo is perfect. My first language is not Hindi but can definitely tell it is great translation.

Thank you so much!!!!!!!!!!!!

They didn’t source robotics papers for Sora’s architecture. They combined Diffusion Transformers (developed by Peebles) with the video diffusion methods released by Stability/Google/Meta/Nvidia.

This is awesome thanks!

I think Sora is better positioned on imagining a new world and totally a different world than to simulate our perception of what the world is and what the world was.

Are there links to the sora videos they are showing?

Great insight

Yes but how do you find the datasets to train for new foundational models? Like their EEG example - how did she acquire this data to train the models?

Alibaba is also doing some interesting things with AI video, we (open source community) have almost destructured the process.

At 4:17 to 4:20 in one of the column one ladder joint got added.

M 17:20 here seems an interesting approach… but sorry that I am new to this AI space, what does it mean by building its own foundation model but with gpt2.5. Does it mean it fine tune through gpt2.5 with its own data?

Who can share the papers which are necessary to get to a level of understanding that is actionable? As explained in the video :)

̶f̶ i̶r̶s̶t̶…. no… early adopter

An idea that anyone can take (though it might already exist): Use AI to help recreate crime scenes and make recommendations on what data might help better understand and solve cases. The ideal solution would be able to use data from other cases in order to improve recommendations.

I love this series

best episode so far.

interesting that raytracing in games might be done and games will be diffused not rendered

I appreciate the show and encouraging people to go for it, and I get hyping up the early YC-backed products, but the first couple weren't even super impressive by March 2024 standards, let alone being "the best thing" on the market. I'm not bashing any of the products and I hope they do awesome, I'm just saying these are not at all good examples of "the best we have right now", and is discouraging to hear from you guys. @ 11:42 The lip sync is completely off. This while perfecting lip sync motion was already accomplished last year. @15:40: Check out Suno AI v3. That's like GPT-4 compared to GPT-2 (what you showed here)

NICE VIDEO MY FRIENDS

Thanks for sharing! Love the content. Any great repositories for the latest academic papers/journals to read up on as mentioned near the end?

Respectfully stable diffusion is way better than anything else to act like mid journey or playground is better is to not understand the flexibility and creativity you have with stable diffusion. Stable diffusion can combine with control net there is a massive community Civitia with LoRA and textual inversion etc and there is a thousand tings you can do from deforum to you name it. Stable diffusion is the only model that can give you precision when needed if you know how to use it, yes its more complex but it is the best model

Great ❤

the lipsync has some better open source free solutions but still cool

4:47 there's tons of videos of the golden gate in 360 - gaussian splatting can do it much better 😁

it's mind blowing, how newly graduates create AI driven products without even 10 years experience and research in ML as well as without spending decent amount of cash. Fascinating !

12:40 Really burying the lede here to the question "How are YC companies able to create these models with only $500k?" We arranged for free compute with MSFT (she didn't say how much, but said hundreds of times more than they'd get otherwise)

How to work this type startup

¹1¹! It's "Sam" day in the Bay area.

I wonder what the dog thinks about him lol

why is that cup on the ground instead of being on the table?!

Inspiring to know AI barrier can be overcome without a PhD in ML/AI. Thanks guys!

Hi everyone, I'm exploring how startups are balancing AI model training with customer data privacy, especially in competitive industries where data can make a difference against competitors. If you have insights or experiences to share on anonymization techniques, federated learning, differential privacy, or service models with privacy tiers, I'd love to hear from you. Let's discuss this further and exchange strategies for responsible AI development.

Okay I am sold on Y C lol will submit my application, access to GPU's for fine tuning is valuable

What if this video also is ai generated

No way u dont kno who that is 11:53

Seeing one example of the generated spelling being correct or even a few does not mean there was any advancement in this area...

It’s just the start of the AI and a lot of Sci-Fi things becoming real.

15:04 memeworthy clip

Great ! I watched this video to relealise i need millions of free credit to build a foundational model for free

❤❤❤❤❤❤

What sora imagined was a single-laned residential street, lots more space for trees, gardening, walking and for neighborhood activities. Cars move one-way in from one direction and out in the opposite.

I hope playground wins though the more competition the better

Looking down on synthetic data makes no sense. Models like orca was built on synthetic data and it outperforms models 10x its size.

The models you showcased today aren't really "foundational models" (at least in the way the term is currently used.) and a lot of what you show isn't super new. Saying that you don't need a lot of GPU's to compete is very misleading.

Feels like I’m sitting in listening to the four smartest kids in my class XD

I hope you guys are very successful so you can buy some furniture! lol jk bro. just a prank bro.

Let’s push 100% AI to the market. There should not be single human needed for a corporation after that.

Thanks for sharing one more useless hype. You jumped from blockchain to crypto, NFT, and finally to AI. You should change your brand name in Y Hype 💩

the hell?

A guy not speaking Hindi gives his opinion on an lip sync model speaking Hindi 😂

If you really want to add value to this podcast why not add a black person to the conversation? We reccommend Christopher Lafayette. He's in the Valley and can contribute well to this conversation and draws an audience. Else, find someone else, but consider the upside to this. Thanks.

AI, even at current levels of capability, is far too dangerous to our society to be released to startups or governments or businesses or the public. We need startups to tackle the safety of these models at a more aggressive rate than capabilities advances.