Build a Street Fighter AI Model with Python | Gaming Reinforcement Learning Full Course 

Bro I have same problem can help me more about installing exact steps

@@sohailmomin1486 you have to make sure you're running python 3.7 or 3.8 and then run the codelines in parantheses with normal pip command

Can you please share your code with me?

@@aounzaidi9248 the code is in the brackets

thank you!

Amen to that!

Thanks a mil my guy!!

Thanks so much Karthik!

Ayyyy Meet! Thanks so much 🙌!!

Thanks my guy!

Is it working properly?have you did it?

YOOOO team Lees! Thanks guys!

I hope this comment got noticed. The topic of hyperparameter optimization is very interesting, I often stuck in learning about it

Hahahaha, you caught that did you?! Thanks for checking it out! Oh love that you're building them up from scratch, I had to do the same to really get my head around PPO. DQN is next on the list to knock over some off-policy stuff. Regarding annealing the lr, you can use schedules in the callback to do it. Haven't tried it out but definitely on the list! On NEAT, that's awesome. Seriously it's so addictive to watch!

@@NicholasRenotte Of course I caught that :D And yeah, I can really recommend to anyone wanting to get a deeper understanding of ML in general to just implement some of the algos from scratch. It really does help you wrap your head around it and it's actually not as hard as it looks. The best starting point is just a plain and simple Deep Neural Net to really understand gradient descent. And once you really got gradient descent down, everything is possible, since many many of the popular algos are just variations of it. As i said, I'm working on a deeper dive into reinfocement learning at the moment, and my little goal is to maybe create a project that draws pictures with you. Something like the net draws a "picture" (essentially just a sequence of lines on a canvas), you give it a rating between -10 and 10 as it's reward, which the net uses to adjust itself, rinse and repeat until you get yourself a nice drawing (or something ...). Maybe with possibilities to adjust hyperparams on the fly so that the user can start out with a pretty high learning rate until the net finds an interesting approach. It probably won't work quite as nicely as I would want it to, since you'd probably need to do a million steps until it arrives at some actually cool drawings, but it's a fun exercise nonetheless, or so I hope. Could also use neuro-evolution instead of reinforcement learning, for example. I think to really get the most out of it, it's probably interesting to use some ML algo that's even better with sequential data than RL. Or you could ditch the drawings and generate for example poems instead. Oh well, I'm rambling. But this is again an example for ideas that can produce really addictive results. I, personally, could see myself there for hours just checking out drawing after drawing or poem after poem to see how it changes and (hopefully) improves over time.

Yeah bit of a long one but I figured it was worth it.

AHHHH, man it took a while. Tbh though I had a ton of fun making this vid.

@@NicholasRenotte very cool and seemed fun. Do more games! Maybe some platformers? I haven’t seen any 3D platformers done but I suspect cause of the complexity. 2D like donkey Kong country would be great. Are there better alternatives on the horizon to gym/gym retro?

Thanks @Santiago, tbh I usually run with PPO given it's not as memory hungry as off-policy models like DQN. That being said I've seen some pretty good performance when I can compress the inputs enough when using DQNs.

Also...it's mostly informed based on what the observation space and action space look like (not all models support all spaces); stable-baselines3.readthedocs.io/en/master/guide/algos.html

@@NicholasRenotte thanks!

I'm not sure what version he was specifically using but you need to use 3.7, 3.6, 3.5 for retro

I used a HPO framework called Optuna to select the best hyperparameters @MultiKAP

​@@NicholasRenotte Yes i understand but for example in 'n_steps': trial.suggest_int('n_steps', 2048, 8192) I saw stable baselines suggest as a default n_steps=2048 but how did you choose the range 2048,8192. Thanks@!!

As you yourself mentioned, to get info out of the game you can always just capture the full frame and feed it to your ML model. That's probably the best idea anyways, as the model then has essentially exactly all the info that a human player would have and nothing more. Though you need to be a bit careful with this approach and think about the right sort of pre-processing, what's good in your specific case and what isn't. For example, as Nick mentions in this video, he turned the image he fed to the PPO into greyscale, which caused problems when the AI had to fight Ryu vs Ryu. Games often use color to distinguish otherwise very similar looking objects, or to give hints about the environment, etc. So while the greyscaling cuts down on the complexity of the info you feed into your ML model, which helps it to learn, there's also the possibility for important info to get lost. How to feed data (the actual controls/actions) back into the game is very very specific to the actual game. Sometimes you'd need to use external software to emulate key-presses, sometimes you could write a mod for the game to interface with, or in some cases you'd maybe need to unpack the whole game and wrap it into an environment like the OpenAi gym people do for us. There are multiple considerations that are important here: - Navigating menus and stuff is something to really think about. You don't necessarily want to waste time and complexity training your AI to use menus, but you don't want to manually navigate after each training run either, so you'd need to automate that somehow. - Don't ever use something like this to cheat in online games with or against real unknowing human players, that's really neither fun nor fair for anyone. There are plenty of fun things to use your AI gaming agents for other than being a douchebag (Sorry for this rant, I do not want to insinuate that this is what you intend to do or anything, but I feel like it is important to mention that it's crucial to think about morals and how your bots could be used before building/releasing them. There are enough people ruining other people's fun in online games as it stands already, no need to empower those guys with AI more than they probably already do.) A really cool and very positive example is those "events" where human players try to beat the AI, makes for great showmatch content and there's a lot to learn for human players as well as for the AI guys. The chess engine AlphaZero or the OpenAi Dota 2 bots come to mind here. Especially the latter is a fun watch if you're into MOBAs, I guess.

Your numbers 2 and 3 are interesting points, would be really nice to take a closer look at both of them! In regards to number one, I actually see it as a positive that Nick leaves those kinds of mistakes in, because I imagine it helps especially the newer programmers that are watching his videos get a grip on the common errors and oversights that do happen and where to look first if stuff doesn't work.

Thanks VitCon! I actually leave in the mistakes now to show what it's really like, I was so intimidated when I first started coding because of all the RU-vid pros that built full stack apps without a single error. I love VSCode though, but mainly use it for converting stuff to scripts or full stack. On the other two, these are VERRRYYYY interesting. How did you get the stage loading to work? I couldn't get the integration UI to load the ROM on my pc. Was pulling my hair out. On using health for the reward, I tried it BUTTT, you have to be careful because the health bar resets after every match. So need to use the match win/lose counter to detect when a new match has commenced (not a new episode). I didn't do it initially and threw of training massively, but yes, 100% agree if done right this would likely help get us to defend.

@@NicholasRenotte you can save the game state and reload it when calling retro.make. That’s how I train each fight individually.

@ ohhhh, so you’re not even using the integration UI. Damn didn’t even think of that

@@NicholasRenotte Can this tutorial be followed if you use VSC? JupyterLab is really confusing and I can't get it to run commands like you can and I keep having to abandon the projects I start with your videos :(

Yo Sean! Solid feedback, I had to think about this for a little while but yeah I think you're onto something. If n=# of games, an increased value of n would increase the opportunity to score more (regardless of winning in totality). I think this could be fixed by adding an disproportionately large constant for winning a round (relative to scoring points in battle), could possibly tune this so that the agent receives a larger reward for winning earlier rounds as well. Thanks for the feedback, appreciate it!!

@@NicholasRenotte I have some ideas in the works that I'm going to test and I'll follow back up when I get the results.

@@xxfloppypillowxx Swet, let me know how you go!

@@NicholasRenotte so after doing some testing and running into a bunch of issues with doing it I found out that it actually worked quite well using the health changes, I would need to tweak the numbers for better results but as this is more of a learning project for me I'm notgoing to spend the time getting the best results I can. I also added a penalty in to punish needlessly jumping but I definitely set that punishment too high. It was an interesting test for someone who DOES actually play fighting games (though I don't play street fighter specifically) to try to help the AI learn to play "properly" because in games like street fighter jumping is *rarely* used since you cannot block while in the air making it extremely dangerous to do. One thing that it did not seem to learn at all was how to block attacks and I'm not exactly sure how I'd incentivize it to learn to block with the data that is available, I could potentially reward the AI for each frame it's alive but it would take quite a while to figure out the appropriate rewards

Yeah, it's a massive pain. I wanted to train per level but the integration UI just didn't want to load up the rom.

@@NicholasRenotte that’s so odd. You usually have to make a save state and then load the state along with the rom. I’ve got games to work in retro gym but I used the neat algorithm with it.

Hmm, I got that bit I just couldn't get the actual ROM to load in the interface for PC. I wonder if it was a buggy version, I didn't end up spending too much time on it, was trying to power through. But solid suggestion!

I don't know how much prior knowledge you have, so I'll give a rough outline for where to start for you and others who might care: - First, start out with understanding the completely standard neural networks, with special focus on gradient descent. There are lots of tutorials you can find online about that. Gradient Descent is basically the foundation for many many other ML algorithms, so that's why it's really important to learn first, in my opinion. - Then, learn about the standard version of Q-Learning (not DQNs, just the normal Q-Learning for starters). The basic outline for this is that you have a so-called "q-table", which is basically a look-up table for the values of all possible actions in all possible states of your system. It learns essentially by taking actions (either via Exploration aka random choice, or Exploitation aka sampling from the q-table; Important to note and understand here is that the exploration rate usually starts out at 100% and then slowly decreases as the model learns) and then adjusting the q-table with gradient descent based on the reward it gets. Another good keyword to look up here is "Bellman equation", since this is basically the equation that is being used to calculate the updated q-values. - Once you got the normal Q-Learning down, you create a DQN by essentially replacing the Q-Table with a deep neural network according to your needs. Important concepts to understand here are why you freeze and copy the DNN every so often instead of sampling q-max from the same network and why in many cases you need a sample memory. This, at least, is the path I took, and to reiterate: Essentially it all comes down to understanding Gradient Descent in the first step. And then you can go to basic Q-Learning and then to DQNs afterwards. The step-by-step approach really helps break the info down and build the necessary foundations for every successive step.

@@NoMercy8008 Wow great explanation sir. I hope I will get the passion you have when I see some fragments of success :)

I'm using 3.7.9

I love this suggestion. Was actually toying with the idea while making this video. Just keeping the intro, client talks, memesplainers and results. Then maybe doing a voiceover for the cut down coding bits. Ideally aim to hit that time frame you mentioned.

@@NicholasRenotte Good Morning from 🇩🇪 Germany, Nick! Thank you for this fast reply! Don't get me wrong! There's absolutely a need and place for more thorough depth to teach this stuff. 2-3 hours is probably REAL fast given all the things that go into this type of problem solving in general and the specific parts of writing the code properly, whilst conveying it entertainingly and didactically efficient. Not easy and I think you do a terrific job there! This can absolutely come down to: "Listen people, my stuff is 2-3 hours long and that's just that. Make it a priority, take the time or don't". Fully 100% legit and fair enough! There's a point beyond which things cannot be simplified or shortened much more. Simply put, I'm just looking at Codebullet! Obviously it must not be this obscure. That's not an insult. The guy is terrific, though with him it seems to be more about entertaining a whole lot more people, who aren't necessarily interested in learning to code or do machine learning, than it is to teach this as a matter of fact. It's entertainment first with him. I can see myself definitely watching 15-30 minutes for a project like this REGULARLY and after that go for the full length, even if that's much, much longer. Kind of like a trailer or teaser first. Hell! The full thing can even be a paid course, sign me up if it's worth it,....it's just hard to do if it's 2-3 hours upfront no matter what. Just yesterday for instance, I enjoyed Sebastian Lague's latest 30-minute video,...and I'm not even doing shader coding (yet). Now afterwards I'm wondering if I'd like to look into a tutorial for exactly that. All the best and thank you, thank you for your content!! J.

@@johanneszwilling I love this feedback and I completely get it. I’m the exact same, I’ll watch something even if I don’t necessarily have the immediate intent to code it. I put a ton of effort into the client convos and explainers and actually thought they’re probably good enough to run as a stand-alone video without too much of the complex coding. Will do a rough cut this weekend and see how it goes!

No Kens were hurt during the making of this video* *I cannot be held liable for the accuracy/inaccuracy of this statement*

Restarting the Kernel fixed it for me

It's python 3.8 for gym-retro which is the latest version supported, but when running but then the code block at 20:40. Running that gives me an error of env.reset which can be fixed by removing env.close(), and then a new error of env.render() module not found. comes up

ok seems like downgrading gym to 0.21.0 makes it work. do "pip uninstall gym" then "pip install gym==0.21.0"

Ayyy, thanks brother, likewise!