AI Learns to Escape (deep reinforcement learning) 

This 8 minute video took over 100 hours to make! Everything in the video (except for the music) was created entirely by me using Unity, so please, like and subscribe!:D If you're interested in training your own AI like Albert but don't know how, there's now a really easy way to do it! Luda, an AI lab, recently built a web app that allows you to create and train your own AI using deep reinforcement learning (just like Albert) completely for free in your browser! You build your own character (called a Mel) with lego-like building blocks then watch it train in real-time on their website in just a few minutes (really). It's an awesome project, and just like my videos, makes deep reinforcement learning so much more accessible, which is why I love it so much. This section of the comment is sponsored by Luda, but these words are entirely my own, it's an amazing project that I would have been obsessed with had they released it before I built Albert. I've genuinely been looking for a sandbox/game exactly like this since I was a kid. They're still early, but they're giving my audience first access to their closed, pre-alpha build. Make sure you check out their site and create an AI agent for yourself!:D prealpha.mels.ai Now, back to Albert: Time it Took to Train: Room 1: 10 minutes Room 2: 20 minutes Room 3: 29 minutes Room 4: 48 minutes Room 5: 5 hours 42 minutes Total Training Time: 7 hours 29 minutes *NOTE* You only see one Albert in the video, but there are actually around 50-100 copies of Albert and the room he's in behind the camera training simultaneously. This makes it so instead of me needing to go through 500 hours of footage to edit the video, I only need to go through 7. Albert was trained using reinforcement learning, meaning he was rewarded for doing things correctly (like hitting a pressure plate), and punished for doing them incorrectly (like falling off the platform or hitting a wall/obstacle). After Albert finishes each attempt, the actions he took are analyzed and the weights in the neural network (Albert's brain) are adjusted using PPO (proximal policy optimization) to prioritize the actions that lead to a positive outcome, and to avoid the actions that lead to a negative outcome. All of Albert's inputs is his 'vision', which comes from raycasts. There are a total of 21 raycasts, 7 looking down, 7 looking straight ahead and 7 above his head, all with a maximum FOV of 70 degrees to try to mimic our own vision. Each of these raycasts is responsible for 2 inputs for Albert's neural network; the distance to an object (if any), and the type of object it is (pressure plate, obstacle, ground). I also stacked Albert's vision 6 times so he can have some sort of short term memory, he can find a pressure plate in the room then take actions to get towards it even if it's no longer visible (for a little bit). If you're still reading this, you're probably really smart and want to learn more about Albert, so make sure to join my discord server I just made where we can talk more about the details of Albert's AI! discord.gg/jM2WkNuBnG :) In the first room Albert starts off randomly making moves until he accidentally hits the pressure plate to open the door, giving him a reward. This reward made the neural network controlling his actions update to try to replicate that outcome, and this continued with each pressure plate until Albert opened the door and was able to walk (on an invisible pressure plate) in the next room. Once Albert got into the next room, the same process was repeated, continuing with the same neural network that let Albert escape the previous room. For people who think I faked this: It would probably take me longer to fake this than it would to just do it for real, I used Unity's ML-Agents toolkit to make it easy, though I have experimented before with doing everything from scratch (poorly). The reason this AI is a lot smoother than others you've probably seen is because I only allow the AI to make a decision every 10 academy steps (game ticks), so when it starts to turn for example, it's forced to keep turning that way for 10 ticks. I did this because I don't like how jittery the AI looks when you let it make decisions every tick. Also, you only see one AI in the video, but behind the camera there are roughly 50-100 copies of Albert and the room he's in that train simultaneously to speed up the training process. The reason I opted for this over having them all train in the same room is because I wanted to be able to follow a single character for the sake of the video. Albert uses the same neural network the entire time, all you need to do is add "--resume" to the end of the command when training in Unity to have it continue using the same brain. Unity's ML-Agents makes AI quite easy! We're looking to hire people to help make these videos! If you're a talented Unity game developer you can apply for a full time position here forms.gle/ko54z1LQmZNUT9Vp8 And if you're a talented AI developer (ML-Agents), you can apply here! forms.gle/Uou1Vwb5Q9VccaAY7 We're looking for full time employees, but part time works too, what we're really looking for are skilled and passionate people, so feel free to apply if you're interested! :D If you have other ideas for what I should make Albert learn how to do, please let me know!:D

Sometimes he simply looks like he's celebrating without knowing what else he needs to do and I love it

The fact that he would do 180s and 360s whenever possible makes me happy

I like how in Room 3, as soon as Albert figured out he needed to jump two times to clear the room, he decided that jumping as much as possible was the best strategy for everything.

This seemed like a beautiful mix of Stanley Parable and Portal. The narrator is trying to help Albert to do what he's intended for, but gets mad when he can't. In the end, the narrator reveals that he has more plans for Albert than just a "cake". Incredible work. See you soon, Albert!

At 4:50, Albert taught us a valuable lesson in problem solving. Thanks Albert!

Albert just casually jumping on the same spot after succeeding just shows how natural this is. Made me happy for him

Albert confidently jumping off the platform to what I can only assume is the sweet release of death really speaks to me.

I love when he jumps off a platform, he spins like he's doing a 360 noscope, I know he's probably just trying to turn midair so he can keep going straight when he lands without wasting time to turn or just trying to check the entire room while he's high up but I just find it so humanizing lmao

I love how occasionally Albert will just start jumping and doing 360s in place when he does something good, I know it’s just a quirk of the AI but it feels like he is celebrating his victory

I love how Albert learned little mannerisms throughout the video, like doing 360 spin jumps or his little shimmy before jumping on platforms.

your editing and design are *fantastic*. every little joke and characterisation has me smiling, laughing and cheering for albert. i am feeling lots of good emotions about a simulated cube, and it's all thanks to the exit arrows, the lil eyes, the music cuts..

7:45 Can we appreciate that Albert learned to use the kind of setup you’d see in a speedrun to line up that jump? Jumping twice while spinning and then jumping backwards is a lot less to keep track of than manually lining it up.

Best ai video I’ve watched in a while can’t wait to see albert jumping out of a plane to try to hit a button

I love how the ai develops some "superstitions" such as that shimmy before the first jump on the final room, or the 2 standing jumps before going for the last pillar

4:09 Albert: FINALY Time: *out* Albert: *SHI-*

4:47 THATS SO TRAGIC NOO

3:19 “There’s still more you need to do” Albert: “No” *proceeds to jump into the abyss*

I love how at 4:45 Albert just jumped into the Void questioning his existence

Y'know I wonder if the spins people like so much are actually Albert trying to see as much as possible while midair?

I love when AI are just adorable

03:58 i love that he didn't just open the door but made it with style

I was surprised that I felt genuinely bad for him at the end. When the floor was getting smaller I actually was like 'oh no, he must be so scared' in my head lol.

i dont know what kind of complex code shenanigans caused this, maybe just a "what doesnt kill me doesn't get evolved out" sorta thing, but the way he really gears up to jump onto platforms (especially in room 5) is so friggin cute! And his habitual spin jumps before the last platform. I'd almost think he feels 😭💕

I love how in Room 4 Albert would sometimes pace back and forth to prepare himself for the jump

Watching albert jump off the ledge on purpose is funnier than it should be to me tbh

i love the little celebratory jumps and spins albert does when he gets something right.

Q: I realize the object of these exercises is to try to get Albert to figure things out for himself, but what happens if he were to observe a human-controlled cube completing the puzzle(s) pretty much immediately? Would he learn to imitate the human cube in short order, or would it still take a thousand iterations before he figured out the human cube's methods are likely the key to efficient success?

I love the way it decided at some point to just spin in circles anytime it jumped, I presume to keep itself upright while in the air. What a clever little guy

Albert needed a "will to live" meter because sometimes it looked like he couldn't do it anymore

This reminded me very much of the plot of Portal. It's nice to see that someone is developing the topic of AI in this regard, please continue to do what you do, thanks to such people the world becomes better and more interesting.

난죽택 왜케 귀엽냐 ㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋ 알고리즘의 선택을 받으신 제작자분 화이팅입니다...!

Those little victory jumps after pulling off a difficult leap, mid-air spins and even the way he "gets fed up" and jumps off the ledge are hilarious. My sides hurt but my day has been made. Thank you.

It would've been interesting to see the final version of Albert try the previous rooms. Check whether his general problem-solving is improving, or he's just overfitting to each room in sequence.

I'm amazed how he came up with buffers and Speedrun strats for movement like bouncing off the wall to line up the second jump in room 5 or in the same room, bouncing twice and spinning to get the correct angle but less momentum so he didn't flip over on the final platform

i love how albert got confused with the wall, goes “i see your challenge, and i propose an answer” and proceeds to yeet himself off the edge💀😂

You could throw in objects that are only responsive to the pressure plates that they are assigned. So, that way Albert has to figure out how to move the objects themselves to the pressure plates. Like in the Portal games. Very exciting seeing this. I myself messed around with creating an AI that was based off of the same functions you used.

i love how at 4:48 he gets so angry that he decides he's had enough, and jumps off

I know it’s not meant to be a comedic video, but I died at 3:50. It’s like Albert contemplated doing it and then couldn’t resist the urge 😂

The way I see it, Albert has a bright future in games journalism with his skillset.

So great watching Albert succeed after such hard work!

I'm getting so attached to Albert! Theres just something about seeing something or someone learn that makes me so happy!

I think its very interesting to see how Albert memorizes the sollution to earlier puzzles but then realizes that they dont work in this puzzle after dying, he probably felt that jumping to the backwards after hitting the button was almost always good, because he got accidentally trained to do that.

3:20 the little jumps he does when he reaches the top is so cute lol, it’s like he’s celebrating his success

I love that it has a subtitles it makes this video way more entertaining and funny you have a new subscriber congratulation

Please give us more Albert! Your videos are the best

Watching room 5 was like watching a professional game reviewer play a tutorial level.

imagine if 50 years into the future AI takes over the world and tracks down the person who made it go through this and put them in an escape room for revenge.

I don’t know why but seeing an orange cube successfully escape a room interests me and i want more

just found your channel and read your pinned comment. I've never wanted to be multilingual in my life LOL I would love to help you out. I think this is amazing and I'm enjoying watching Albert's journey. The trick for filming and for smoothness is amazing. I'm an old school programmer who is just now dipping her toes in the AI world. Loved your explanation of the vision and how you obtained short term memory. Again, I look forward to watching Albert learn, but also your explanations of how it all works. I find that just as interesting. Hoping for a "behind the mind of Albert" video or something that show how albert is set up (like the eyes and such) and how it's filmed. Incredible job for one person. Keep up the good work! I'm a big fan!

The fact that he understands his mistakes and learn from them unlike my league teammates is scary yet fascinating

AI making decisions can be ultimately amazing, or utter chaos. The visual representation just makes the chaos so much more enjoyable. Absolutely love this.

I loved how when he first needed to jump to hit the pressure plate, after he received positive reinforcement for jumping a few times, he just decided he would jump everywhere. Adorable.

Pleeeeease keep making such videos this is great ! I Love Albert

Always love the little razzle dazzle albert puts into their jumps

Albert being consumed by the abyss after trying for thousands of hours is the same as a person who studied all his life and finally graduating just to get hit by a bus before he can enjoy the fruits of his work

I'm blown away by how much I would sacrifice for Albert just because he has eyes

很高兴能看到有视频展现人工智能学习的经过

4:52 声上げて笑っちゃった

I have come to the conclusion that Albert is a genius of visual comedy and deserves an award

4:50 IDK WHY THIS IS SENDING ME

Fascinating! Amazing work.

The end is grim, Albert should've gotten in a room with a bunch of other squares so he can socialize and experience true AI love.

all i can think is that when he finally understands a part and jumps randomly hes just so happy and excited that he did it 🥺

It's interesting how when learning to jump off the platforms at 02:04 he introduced an assumption into his learnt behaviour; not only did he learn that he to jump off the platforms, he also wrongly assumed he needed to jump backwards to succeed- this behaviour is evident right up to the final room.

This is amazing! I want to learn how to do this. I have so many ideas.

albert just fills me with joy and happyness bro

Would love to see the 'trained' Albert run the courses again and the different choices Albert's algorithm would make.

Albert seems like a great friend, he's silly, always there for moral support, can't make you sad through speech and is still able to make you happy through what he does. Albert is just a great friend in general!

Cool video good work! deep learning is really hard! I try to learn C# with unity to create a AI neat ecosystem, i am just starting on the project but it's really interesting (so much thing to learn)

It’s fascinating that once a path is figured out to work, it doesn’t get optimized, it just moves on to the next challenge

I love albert’s preparation jumps in the last room before jumping to the last pillar. He’s like, “Ok, this is it! Focus, Albert!”

The 180’s and 360’s were amazing!! Also the fact that sometimes he would just give up and throw himself off the edge is hilarious and relatable

this is really inspiring! albert is my favorite ai character

You can feel the pure rage in the red text saying “why did you jump?!?!”

Very well done. I'm doing my PhD research on AI using ML-Agents to study artificial curiosity. There's a lot going on in this video worth mentioning, but most of all it's fun and I love the editing. You've done a great job to describe RL, PPO, and even how to use ML-agents. I'm looking forward to seeing more.

5:52 tactical roll

Love these!

7:21,7:47 Albert jumped straight above twice. This behavior seems unnecessary , but I feel like it is his own routine. That's very human-like. He is lovely.

This is such a nice video. It really speaks to both the potential capability of AI, and also to the way humans pack bond even to things with no actual emotion. The way AIbert spins while he jumps, "celebrates" his success, and even wiggles around to adjust position before jumping, all of these little learned actions activate the humans instinct of "this little thing is charming, I care for it." Despite knowing AIbert isn't a living thing, people still have affection for him because humans will see these little actions as "personality"! So fascinating on both ends!

What I loved most about Aibert here is how happy and sort of cocky/smug the little guy becomes when he’s figuring it out. You see him styling his jumps with spins and even doing little dances at certain points. Kind of reminds me of smash amiibo fighters learning to taunt

imagine you as Glados and Albert as test subject from portal, it just fits so well

Great video! I loved how you explained everything in such a clear and concise manner.

6:28 forever!

Man seeing this lil guy go from game journalist to semi-competent was somehow inspiring.

This is very satisfying to watch this

I like how in room 5 he just like *"I'M NOT DEALING WITH THIS"*

I love all the hilarious interpretations people have made about Albert. My favorite is when he learns to 360 and proceeds to do it for every jump. It's like he discovered call of duty.

his little jump of joy at 1:19 i love albert

3:21 I love how Albert gets too happy doing the tall platform he goes in excitement uncontrollably and then commits suicide due to happiness

It's so cool watching how ai learns

You should keep on posting you gaining super quick🎉

Man there’s no way that’s not just a person acting like a game journalist

It always fascinates me watching these as a trainee animal behaviourist because I go through the same emotions watching this as I do when training a dog that's just not quite getting it. It's amazing

When Albert starts skipping pressure plates, I was just thinking of Yoda saying "you must unlearn what you have learned"😂

Its amazing how attached we all get just by putting googly eyes on an orange cube and watching it bounce.

I love how he's jumping around at 3:19 like: 'Look! I did it! I did it!'

2:25 "why did you jump?" "BECAUSE I WAS EXCITED OKAY? ANY HUMAN WOULD DO THAT!"

I love how he seemed like he was hyping himself up for jumps

abi çok iyi yapıyorsunuz tebrikler !