AI Conquers Gravity: Robo-dog, Trained by GPT-4, Stays Balanced on Rolling, Deflating Yoga Ball 

"2022-era." Ah yes, I remember those vintage times. Life was simpler back then.

This is what exponential, compounding returns look like.

"You might have to focus." Thanks for the heads-up.

"Groomed by Gpt-4" is not something I expected to read today.

I'm reminded of the survey you quoted a while ago where AI researchers were asked which jobs would be replaced last, and the majority replied AI researchers...

A lot people are worried about jobs but little discussion on the risk of military applications. Lowering the human cost of war could see the increase military conflicts throughout the world.

we have no clue what is about to happen...

I for one welcome our ball balancing AI overlords.

Next level of AI: non-patience is all you need!

I'm at a loss for words to express my appreciation for your ability to read papers, comprehend, and draw conclusions. You've truly chosen the perfect name for your channel. :-) Your videos confirms and empower.

0:23 zooming in to the paper for two minutes was like a horror movie. After one minute I was sure something horrible will happen.

While I’m skeptical about AI “reasoning” and all that, this application is, indeed, genius. So I guess now the ultimate test of training robot skills in simulation with zero-shot success in the real world is a robot riding a bicycle? (Or maybe walking a tightrope?) _That_ I would like to see! The actual AI achievement aside, the video is a masterpiece of clear explanation! Great as always, Phil!

this helps me to envision a future where home robotics are constantly using Dr. eureka style platforms to simulate different tasks and iterate on reward functions using Three-Dimensional scans of my apartment as well as data from cameras and other sensors. maybe in the future housing could be built from the ground up to accommodate this kind of technology, for example pressure plates in the floor and that sort of thing.

GPT-4 is "fine tuned" by RL, and then the LLM updates the RL reward function. Impressive work!

I had to immediately pause the video when you said that gpt 4 trained the dog better than humans. Jaw dropping moment and this is only the beginning

You are the best 👍🏾 AI channel!

Wake up babe AI explained just uploaded

Periodic reminder that this is the best AI channel on RU-vid, by far.

After two watches, I finally understood how much I still don't quite understand. I'll have to look at Patreon after I catch up here. Thank-you P, you help keep my Gen-X brain active, I envy your enthusiasm & immense practical intelligence. _ps: DrEureka's completely different approach to humans is quite amazing._

Serve me butter

Thanks for summarising this paper man. It's maybe my favourite since the simulacra for human behaviour paper. Excellent presentation as usual.

Self-driving is far more difficult than you think. The Public will want and assume ZERO, or close to it, accidents. It will take only ONE instance of a fully laden semi driving into a stopped lane of traffic at full speed and they will all be banned.

Appreciate your style, knowledge, and effort. This makes all the value for everyone spending few minutes with your channel! What a time to be alive!

Wow I thought we'd need a lot of real world examples and data to train something like this but looks like simulating it is so good,i wonder if simulation could also work for self-driving Feels like this accelerated AI robots development a lot

Thanks for the subtitles in the interview btw !

Love your videos! Learned a lot! As always thanks for quality informative content 👍

This is incredible.

So let me get this straight; these reward-functions that are already absolutely crushing the equivalent human attempts are written by an almost 2 year old model, and furthermore, OpenAI just very recently received their first H200s so I would think even their next model won't have had time to be trained on those. And then not even that far the line Stargate is planned to come online in 4-5 years... And those are "only" the fairly predictable hardware improvements that we know are coming. Meanwhile the entire world will be working on micromisation, algorithmic optimisation, architecture improvement and model specialisation. That exponential curve is becoming clearer and clearer, and it's looking more like a vertical wall to me at this point. I've not bought fully into the intelligence explosion theory, but papers like this are rapidly convincing me. Thanks as always for bringing your thouroughly researched presentation and unique personal perspective. I think I'm even gonna read this one in full myself.

Brilliant as always, a lovely addition to a long weekend! It feels like everything is really hinging on how much smarter OpenAI’s next major model is, it’s been well over a year now and we’re still training and performing tests using GPT-4 as the SOTA model! I’m particularly excited to see some multi-modal improvements (as you already know). I also can’t help but look at all of these papers now and wonder how much better they’d be with SmartGPT!

Let's goo, best thing when AI Explained uploads. IO predictions?

The dream utopia that AI can bring I envision is no one has to work but can if it’s as a passion but we all go into a meaning economy. That’s worth fighting for.

🎯 Key Takeaways for quick navigation: 00:00 *🤖 Overview of the Eureka concept* - Training a quadruped Robo dog from simulation to reality using GP4 language model. - Language models like GP4 are more effective teachers for robots. - GP4's reward functions outperform human ones in robot training. 02:20 *🧠 Implementation details of training methods* - GP4 trained Robo dog reacts effectively to new tasks not seen in training data. - Use of domain randomization for realistic parameter ranges seen as crucial. - The process details involve isolating variables, testing viable ranges, and domain randomizations. 07:28 *🎯 GPT 4 versus human training comparison* - GPT 4 outperforms human-designed reward functions and domain randomization in robo dog tasks. - Language models like GPT 4 generate multiple reward functions simultaneously for continuous improvement. - Dr. Eureka's robot training shows significant performance improvements in forward velocity and distance travelled. 10:34 *🏋️‍♀️ Safety instructions and reward function design* - Importance of safety instructions to prevent degenerate behaviors in robot training. - Multiplicative reward functions generated by GP4 for robust training. - Glimpse into the design of safety-oriented prompts and smooth reward gradients. 13:34 *🌟 Future improvements and implications* - Discussion on enhancing the approach by incorporating vision and co-evolution. - Speculation on the potential limits and possibilities of the Eureka approach. - Predictions regarding AI's impact on physical tasks in industries and the evolution of robotics. Made with HARPA AI

Thanks! Great content, as always! 🙏🏼

I can hear the excitement in your amazing explanation. I understand. This is big.

Fascinating stuff. Thanks for breaking it down 😊 I personally can’t wait for the robot servants and realistic looking robo-dogs 😎

I was, in fact, in bewilderment, but for reasons I'm not proud of😂 I wonder how other models like Llama3 would do at this.. Magnificent work, really appreciate the jokes, and that you provide the simpleton version of all the big words too🤭🤗❤

Really impressed by Jim Fan and his team. Excited for what may be coming in the next year

one of the few content creators I always watch and like.

damn back to back uploads shit's real

This is next level.

amazing, as always

I have to say I can’t get my head around a lot of these things. Gpt-4 often just tells me “yes, that a complex task and you will need these skills. Good luck!”

Perfect timing ❤

Very cool - thank you!

This is a total WOW. Recursive improvement is a type of positive feedback cycle, right? And, unless constrained, those quickly become exponential. So, yes, rapid improvement in robotics is to be expected. Thank you very much for this and all your diligently researched videos.

As a mostly blue collar worker who mostly drives forklifts and other physical activities all day, I thought it would be at least 6-7 years before robots could compete. Did not even know that writing reward functions, testing, and iterating was such a cumbersome task, the fact that LLM's can still do the reward functions despite hallucinations and can co-evolve with the robots is truly cool and scary at the same time. Only thing standing in their way is financial, energy and regulatory constrsints. Lovely video as usual.

Infinite time in-sim is all you need! The answer was always a Hyperbolic Time Chamber ⏰🤺

Amazing work. So so cool.

Congrats on passing 250k subscribers by the way!

I starting the podcast even though I read the episode name, I think there is a coming tech business relationship transition, 2:20 . Can you trust the educator unquestionably, interesting stuff, cool dog, reacting in real time. Just thinking, sounds like the most common mode of motion will involve a wheel based platform for product distribution, and evolve niche capabilities in bipedal motion, great update, physics lesson included. The guard rails are almost transparent in the world of ai/agi. As always thank you for sharing your time and work Phillip, ✌🏻

I've always had confidence that I could keep up. Now I feel like a pair of plane brown shoes in a world of tuxedos.

While humans didn't write the reward functions, I am quite sure that there was a lot of back-and-forth with the prompts until good reward functions (and variable ranges for DR) were written for this task, this robot and this environment. In a sense, still a lot of domain knowledge, but you leverage an LLM to scale up "domain expert productivity". Maybe I am missing something, but using LLMs like this for robotics seems very hacky. A cynic would say, It's almost as if someone was trying very hard to find some way, any way, to apply LLMs to gather attention and generate hype ...and attract funding.

These RL integrations are what I have been waiting for!

It makes sense that an AI would excel at training robots, but it's still surprising. Great video! Regarding self-driving, Tesla recently saw a huge improvement once they finally adopted an end-to-end AI solution with version 12. They might actually solve it.

5:54 this is literally how I thought they would do this. I’ve had a dueling theory of extremes and outliers theory; seems to be playing out with the data and its implications on embodiment and the interpretation of physics in the real world. Awesome stuff.

Brilliant!

"The compute budget is the limit" -- there seems to be an emerging consensus around this recently

Kinda what I was imagining in my sci-fi future: like factories with "brains" or control AI and other robots transfer information all the time on what they need, repair and improve themselves and request needed materials from humans

waymo is on tracks. Tesla FSD is actually impressive in a million unique settings

12:31 tightrope walking

When talking about "testing all the scenarios" people would be well advised to consider the ways that things are counted in computer science (and much of computer science is about counting such things). Numbers get ridiculously big very quickly (ref: wheat on a chessboard).

This is THE craziest thing I've ever seen and I think I'm not exxagerating.

One of the few things touted as advancements lately which actually seems like an advancement and not just generic hype. Terminators are on the horizon!

Insane. Incredible.

I guess the obvious question is, *what does this look like when you apply it to LLMs?* Given Anthropic's interpretability research (superposition/the unit of analysis not being individual neurons but groups of neurons & monosemanticity/tying each cluster of neurons to a specific word or meaning it triggers off of)... it seems like not too soon, this might be something an LLM could do to itself to try to recursively improve itself (at which point, all bets are *off)* -- and even failing that, get an LLM to "debug" a smaller LLM could be very, very interesting. Wonder what it'd look like?

AGI anxious notification gang!

I seldomly say this, especially in the AI-Space but this is HUGE if generalizable! I mean ZERO SHOT! Ho-li! Thanks for that update!

If you could have access to the model itself, I wonder how well GPT-4 could perform as the robot-dog itself, as in adding a sort of "action" modality allowing it to see and interact in the real world or a simulation.

This is so big that I would say it's one of the major leaps towards AGI. By doing this for every single task, you can optimize physically doing anything. Or rather this is conquering the real world. This combined with Sora's capabilities is gonna get you incredibly skillful robots. You can also do something similar for non-physical problems. You can solve a lot of things. Allow a good LLM the tools science has for experiments and it would invent new science.

There goes my job security. Hard life being a yoga ball balancer

2 AI Explained videos in 3 days. LFG

i dont think you need a LLM for balancing task, plenty of negative feed back control system can already do balancing task that's hard for human. balancing an unicycle on a string, balancing a pencil on its pencil tip, jugging balls. On a more useful scale, maintain ship heading in rough storm also already use feed back control system, it can correct for both periodic era like wave or impulse error like a pocket methane spout. The difficult part is to find the correct feedback model parameter and the correct sensor data to make sure the system stay within feed back control force range. If the llm can automatically generate the correct feedback model of the physical object, then it is a game changer.

13:37 this is sounding eerily like takeoff

Presumably this works because GPT-4's training data included some robotics textbooks. While it's impressive that it can make such effective use of the material, let's not forget that that material was discovered and written down by humans.

What a time to be alive!

Love to watch your videos. One thing to note on this particular video is Tesla is leading the self driving car race. Waymo, impressive as it is, is limited to mapped areas and isn't scalable imo. Tesla is learning to how to drive using end to end NN

Eventually, it will become practical to test every single scenario. We're living in one of those scenarios now...

Wow this is crazy!

Can it be convinced to allow the high level of openly operating net piracy to continue if it were also to become disappointed about it and is it going to choose to not participate if employed so?

The first example of real synthetic co-evolution!

This is freaking remarkable.

Well I'm glad GPT-4 didn't take the robo puppy to the gravel pit after the first minor setback!

Manufacturing as a constraint - Will commonly available hardware (power tools) be robotized? What is the minimum change to a power drill, so that it can take advantage of this sort of software?

I might be wrong in the way I understood how the robot works, but the robot balancing on yoga ball is actually does not bring us to autonomous robot helpers that much. The way I understand, problem with robots is that they can't navigate around the world without understanding it. There is just too much knowledge to have about the world. This is why autonomous driving is such a big problem. But things like balancing on a yoga ball can be done literally blind. It requires only knowledge of your position and forces affecting you. Although more efficient training though simulations is definitely a gigantic advancement, I feel like the simulation is much more important than ability to balance.

I wonder how far away those robot arms etc are from being able to be used as prosthetics? Maybe each limb could have a processor that has been trained as described here? People are starting to have chips embedded, so if they could send wifi signals to the limb... anyway, just a thought. 🙂

nice, thanks! :)

super interesting

If I understand it correctly the reason LLM's are so powerful in these cases is because of how their understanding of natural language makes everything else easier? I think that there are other systems would be better apart from the practical communication benefits right?

I think that OpenAI releases GPT-5 the week of June 10, or during WWDC. Epic arms race incoming!

Wang Jo Wang dorm apartment background shows he recycles more Heineken than Budweiser. I predict in one year the US or UK military will have a biped robot that can complete a complicated dexterous task like a Wing Tsun dummy routine, or Aikido hands drill. Do you think, or have heard any students trying fingers for piano dexterity? Good report as usual. cheers

I think GPTs/LLMs may actually be better at training and operating robots than answering text questions. even though all of the first applications are computer-based, I think robots will be a huge part of their future use.

Amazing

One factor I find both impressive and confusing is the ability to apply expected response for each iteration of Simulated training in terms of physics. My understand is AI has trouble understanding and applying physics but must have understood it to improve the iterations. Another uncertainty is around the leash used on the robot. Was it a known factor and part of the simulation training? It appears the 'walker' used it to steer the dog, actually causing it to walk backwards and pull it off the deflating ball.

Humans have now found and created a process that basically equals a generalized reward function. This generalized reward process is basically AGI/ASI already. I officially call the starting gun on the singularity.

So what exactly am I supposed to retrain in now my white collar job is on the line due to AI? Is anything safe? Or should I stop worrying

Skilled Trades were always going to be the last thing replaced. However, I always figured that difference between "knowledge work" and "electricians" would be more than a few years just from the manufacturing of the robots. Reality is the "extreme mode" simulation, and it doesn't care how you THINK things work - they either do or don't work in reality. The thing is, a "handyman" bot with 3, 4, 6 or 8 arms and 2 legs ... is going to be capable of ridiculously efficient work that humans won't be able to compete with. The 2 arms, 2 legs thing is cute, but a 4-armed professional painting bot is going to get stuff done in ways no human could imagine.

Language models are just smart. It is astonishing what you can do with them, especially when they're multimodal.

Doesn't it feel like the ability to "throw out the human textbook" is tending towards the direction of an alignment blackhole? Isn't the ability to understand those local maximas in other dimensions of the calculations that makes understanding what/how the model does so hard?

Someone needs to apply this to the Robsen Transformers

The main thing preventing robots being useful at human tasks such as cooking, plumbing and all the others is cost efficiency! im not going to buy a million dollar state of the art robot chef when i could just hire a human for a fraction of the cost.