Mario Bros is too easy for INSANE AI 

It may be possible that the AI is learning that coins are good naturally. That is, it might begin to "believe" that collecting coins is going to have a higher probability of receiving a higher reward.

@JoziGlitzh So from what I understood from the video, the AI is making decisions only based on the pixels that it “sees” - the pixels + the feedback from the reward function. It will try to maximize the rewards based on that (pretty much like a human). I’m guessing, then, giving rewards for coins would make things harder as you would need to gather data from the game aside from the images. If you think about it, deep down the AI probably does predicts rewards based on coin locations, as they mostly indicate the right path to ending the level.

​@@TheGhostWinner I understand, maybe if the AI received colored images it could then detect coins, though, for it not to confuse anything yellow with a coin, it'd have to detect its shape too, which could be quite complex and it may still confuse other yellow things for coins, especially if they're round. And yes, what you say is true. If it sees that the path it has to follow for it to be rewarded has coins, it would probably associate coins with playing the game correctly, however, that would take a lot of time, making it so just collecting a coin rewards it would speed up this process a lot, and as a consequence, it would probably learn quicker how to beat the level.

*proceeds to put it in a Mario Maker troll level where all coins lead to death*

@@TheGhostWinner How would retrieving the coin count from the game and using that as a reward function pose any problem? As I understand the AI is already being rewarded for increasing its x position and punished for dying. Couldn't you just drop and replace that x position with the coint count?

Thanks! Good to hear you think its improving, hoping that if I stick at it for a while and the content is good, the channel will grow!

@@aitango For sure! well let's hope first that there is a way to get the shaking to work within dolphin!

@@aitangoit definitely will!

That's a good point! I forgot to mention this in the video, but the AI isn't actually given just a single frame, but rather than last 4 (this is often called framestacking). Giving it the last 4 frames tries to rectify this problem by at least giving it some sense of motion

@@aitango What frame rate were these last 4 frames at?

That is an interesting point. I wonder if the pure reward for going right, excluding the penalty for death, is partly responsible for the slow training. For instance, I have noticed that it rarely chooses to do nothing when that might be beneficial on the rotating platforms. What it does instead is go right too much, then corrects by going left, then tries to go right again because all it learns is to go right. This process makes it more likely to die from failing at attempting a leftward correction. My first thought is to introduce a penalty for going left, but that could also slow training. My second thought is to introduce a penalty for rapidly changing direction (i.e. moving right/left immediately after moving left/right) because that indicates you're correcting a suboptimal decision. You could make the penalty for changing direction decrease with the number of frames that the action "None" is taken in between changing direction. This could introduce problems with learning concepts like wall jumping, so further modifications would be needed, but the point is that "teaching" only to move right as much as possible might lengthen the training time for navigating moving platforms.

That would be absolutely sick

that's legit a genius idea

I wonder if this would count as an adversial generated network (gan). One ai builds levels, one plays.

I think if the he can figure out a way to use the shake function and the nunchuck through the forum in the video then that will most likely be a future video.

I will definitely be doing more Mario levels in the future! As mentioned, at current an annoying number of levels do require shaking, but at least for now I'll try and find some good ones that don't require the shake

AI v.s champion's road

​@@aitangoI'd love to see it beat an entire game someday

I’m kinda curious how it’d tackle a tower level. I imagine it’s way harder to teach an AI that already likes just going right that “ _sometimes_ go up = good, _sometimes_ go right = good,” and I think that could be interesting.

there tiktok lives that sometimes show that

@Thisaintarealemail lol that would be insane

​@@musicviolamusician4392TAS

Really glad you think so! I would be very curious too! I think some levels would require so changing of the rewards though, since some levels require you to go up and down rather than just right! I will have to try the AI on some harder levels though so I can give you an idea haha

oh hi mayro

it'd need to train on every level at random probably, since just training it on one dataset causes massive overfitting issues

That's really great to hear, thank you so much! I'll try my best to keep making videos people want to see!

Really glad you like it! I haven't actually figured out the motion issue yet, the Mario Kart video uses the gamecube controller since Mario Kart allows it. I've looked at Dolphin's settings, but sadly I'm using a modded version of Dolphin which allows Python code to interact with the game, but it doesn't have nunchuck/motion input yet :(

​@@aitango I think dolphin's pipe input could help here it lets you use python / whatever language that can open files as an input method it might be harder than the modded dolphin tho

Thanks, glad you like it

Thanks, I hope to do some harder levels in the future

Hey, its cool to see others trying to work on this kind of stuff! Sadly setting up the environment was a real pain and took months... The way I did it was by using a custom interpreter (which controlled the AI) to launch instances of dolphin with a script running from command line. I then used sockets to communicate between the main program and dolphin. The main program would run most of the AI code, and the dolphin side would just take actions, and get the state and reward. Was not easy though. Thanks for the good wishes though, I'm very much hoping it blows up at some point!

Real life reinforcement learning, I like it haha (minus the punching children of course)

This AI probably wouldn't be able to generalize very well since it was only trained on one track, but I imagine if an AI was trained on a few at the same time it could start to generalize. I'm definitely looking to do a video on this in the future!

Thanks! I hope it gets millions of views haha

@@aitango How the hell are you this underrated. EXPLAIN YOURSELF

I'm not sure I can explain that one, but I'm hoping it changes soon!@@jaythecoderx4623

I hope so @@aitango

Thank you very much! I have considered it before, I’m not sure it would get as much attention as videos like this, but I’d really like to help people get into AI so it’s something I’ve thought about a lot!

Welcome aboard, thank you!

Thank you for watching!

it would fail directly

@@Photo650D But how long would it take to learn a second level? Less than the first?

Also, I barely use C++, but I am willing to look into helping you figure out the issue!

Really glad you're enjoying the content! I can't even remember what I called a 0, this video was a while ago haha Thanks, that really means a lot of you're willing to have a look around, I'd love to try and play other games that use the nunchuck!

Really glad you enjoyed it! Yeah I'll keep that in mind

I do write in Python! The Python library I use for neural networks is called PyTorch, which makes most of the really hard stuff a lot easier. If you want to learn more, there are lots of great resources on Deep Reinforcement learning out there. My favourite however is a course by the channel Deep Lizard

@@aitango thx o heard about it. My only steps i took where with tensorflow but this is a good tip

So currently the AI does get a pretty big reward for reaching the end, just to encourage it to do exactly what you say! I include the rewards throughout the level though, as you are right that without them it would take forever to train since the AI wouldn't be given much guidance

So if it was a single AI, it would likely get faster as it could use what it learned on one level to help with another. Much of what the AI learns early on is things like detecting objects, edges and learning what the different actions do. That said, I have no idea how the AI would handle some of the levels on later worlds!

Yeah that would be cool, I'm looking to do that on Mario Kart as well. I doubt after learning a single level it'll generalise too well, but I think it could if it trained it on a few simultaneously

One problem: boo mansions

The current modded version of Dolphin I'm using already allows me to read and write memory addresses! The mod is really good, all it needs is support for other controllers!

Probably both, but it's definitely learning to react to individual situations. If you plopped it into a different level, it should be able to apply its skills and what it learned in the previous level, so it could continue to deal with common things like bottomless pits or enemies.

Unquestionable logic I know

All of my AIs are written in PyTorch!

@@aitango thank you!

You know... If our brain manages to process moving images I'm sure AI can too!

Oh no what have I done haha

@@aitango Hey, don’t worry about it bro. I was just messing with you, the rest of the video absolutely makes up for it. Watched your Mario Kart video just the other day too, you got some good stuff going here.

I just put it as my wallpaper because I thought it was cool, but a friend later told me its actually from world of warcraft, even though I've never played it haha

@@aitango That's so badass lol /gen Also, could I add you on league? (I'm half asking as a bit, because it would be dope to play a match with a youtuber gldkhlkshg)

@@tylerclark3870 I was asking the uploader lol. I saw the shortcut on their desktop. Who're you? /gen /nm

Not sure who the imposter was, but I also happen to have stopped playing league after I hit diamond haha in a very very similar way. Oh that makes sense you saw it on my desktop

@@aitango Yo that's totally valid! I hit plat and I'm starting to lose the drive to play dibsksbsksbsken

Yeah that would probably be interesting! I think I like seeing it go full speedrunner mode though :)

Make 2 variants 1 is high score 2 is spedrun mode

I've seen that on a couple of videos, I never seem to spot it when looking over the subtitles :(

😂

It would take ages, but I'd love to see it slowly progress through all the levels too, learning and optimizing!

I'm currently trying to see if this is possible, because its something I'd love to do. Especially doing online so I could have an AI playing in a full room

@@aitangome when I’m lonely. Lol but seriously could you imagine playing smash bros online with your ai co-op against real players? Would be dope.. but wrong lol

I think this network had something like 10M parameters

Thanks, glad you like them!

The reason I'm saying this is because, ideally, the AI would be able to learn eventually not just how to play one level, but be able to complete the game, including learning to collect star coins and powerups. I think this would require a more complex reward system, and I'm not exactly sure what it would need...maybe a requirement of collecting all 3 star coins before completing the level. But then, how would it understand hidden blocks? A lot of questions, but I'm sure that AI tech will get complex enough to be able to do that.

Would love to try the star world at some point haha. Might ease into it a little first though and try some worlds inbetween

I know its from DS, but I just thought it was a cool thumbnail haha. Glad you liked the video anyway though!