Create your own A.I. in Unity | ML-Agents Tutorial 2020 

Thanks!

Anand Kuwadekar thanks man :)

thanks man :)

Thanks :) I‘ll also talk briefly about the differences and use cases for PPO and SAC in the next video. Yes, AI will almost always figure out ways that weren’t intended. Good example with the close environments.

Happy to hear, thanks :) It’s the name that you specified in the Behavior Parameters component.

@@BotAcademyYT It worked. Thank you! Now I can also start the imitation learning. Really nice

Hello ,me i don't find mlgent config in the folder of the project (i have downloaded it from github) 😞

Thanks! Yes I mean more training steps. The model in the model slot will be ignored when training. But there is an argument that you can pass to specify a folder generated by a previous run. If you then also pass --resume, I'll resume a previous training run.

Glad it helped you :) For visual observations, check out this video: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-5y3MD2QqV8E.html Thats a good idea. I wrote it down and will make a video about it - might take some time though. You might wanna check out this docs file: github.com/Unity-Technologies/ml-agents/blob/master/docs/Python-API.md

@@BotAcademyYT Appreciated for your help!

Thank you! Wasn't mentioned in the first video. Sure, you can think of a branch as a single decision the agent has to take every step. So if we have two branches, the agent has to make two decisions. For the first branch the agent can choose between zero and one (in the code we map the decision 0 to 'don't accelerate' and 1 to 'accelerate'). If we would want the agent to accelerate backwards we have to set its size to 3 and then map the value 2 to the force -1 in the code (so that he accelerates backwards when choosing the 2). The second branch has the size 3 cause the agent can: 0: don't move left or right 1: move in one direction 2: move in the other direction So for example if the agent chooses the value 1 for the first branch and 0 for the second he would accelerate straight forward without changing its direction.

@@BotAcademyYT Thanks for your reply. Could i think of branches as some sort of output neuron in the network? How do i specify the size of the neural network ? like number of hidden layers and number of neurons in them?

yes exactly! The number of branches and their sizes determine the number of output neurons. You can specify both the number of hidden layers and number of neurons via the config. If you head to the config of the BallAgent in my video, you can see those two config parameters.

@@BotAcademyYT Thanks a lot. Looking forward to get time to play with it :) p.s. WHen is episode 3? ;-)

You’re welcome :) Might take up to 4 weeks cause of ongoing semester projects.

nvm bro. fixed it manually. great video!

You can change some of the parameters like the number of layers and neurons per layer through the config - check out my third ML-Agents video where I talk about the parameters. I hope that answers your question, if it doesn't and you really want to change the network architecture on a deeper level, let me know.

Thanks :) Let me see if I understood the question correctly: You want to know how the ml-agents python program that we start with 'ml-agents learn ...' is able to find the unity project to connect with it? If that is the question, I think that it gets access to the resources through the connection on port 5004 when we start the environment. I think that opens up a socket connection between unity and the python program under the hood. Please let me know if that wasn't the question.

@@BotAcademyYT Yes, you understand me correctly and answered , Thank, And my problem is the the remark "Could't connect to trainer on port 5004 ... " It seem to me that it is my fault of setup ( video 1) I hope to solve it (is there a clue) Thanks anyway . I looking eagerly to the next video about the explanation of parameters in train_config.yaml . Great Job.

You’re not the first one with this issue: 1. Make sure that the behavior type is set to default as seen in the video at 16:17 2. Make sure to run the environment directly after running the ml-agents learn command - if you wait to long, it'll time out Does that help? Otherwise you can download my repository with the exact unity environment as shown in the video via the link in the description. Might also help.

Yes, rigidbody applies gravitation to the agent

@@BotAcademyYT ok thanks. I'm very new to ML-AI even tho my first language was python

Thanks for your comment / thoughts! 1. As always. It depends. With a normal Neural Network, the agent doesn't remember its last states. It just acts based on its current observations. If we would use a Recurrent Neural Network (for example an LSTM) we wouldn't need the velocity cause the agent can remember its previous position(s). There is a configuration for this called "use_recurrent", I haven't tried it on this example though. 2. Good point. The reason I didn't chose a negative reward when falling is that I think it is to risky at the start of training for the agent to actually jump. The agent will most likely miss the cube multiple times resulting in a negative reward. So the agent might be ok with not moving at all to avoid the negative reward. What you could do though is giving the agent a small negative reward every time step - but you should only add this if you want the agent to reach the cube as fast as possible. If that is not the case - keep it simple (always a good choice for the reward setting)

​@@BotAcademyYT And thank you for your answers :-) 1. Ok that makes things more clear. I just looked up the docs ( github.com/Unity-Technologies/ml-agents/blob/master/docs/Training-Configuration-File.md#memory-enhanced-agents-using-recurrent-neural-networks ). But am I right that this is only related to actions? Because I think my confusion came from the decision requester. I expect that the agent will collect observations for e.g. 5 steps and make it's decision based on that. I haven't distinct between observations and actions so I just thought: actions are also memorized. But at least now I think I understand. Using this use-recurrent seems to be expensive. Just passing its velocity seems to be the better approach. 2. Well - at least here it seems that I've been on the right track in the end (I guess the ai trained me more than vice versa). Again: thank you very much for your answers. This topic is really interesting and really really complex. I'm thankful for every information I can get :-)

You're welcome :) 1. I think it is also related to observations. Cause from the theoretical / mathematical standpoint, the Agent never receives a sequence. Just the current observations (9 in this case). So I assume that when I set the decision requester to 5, the agent will receive the observations from the frame before he has to take an action again and not from the last 5 frames. Yes its indeed quite complex and hard to find good resources about it.

​@@BotAcademyYT I think you are right. I checked the docs and also other YT-Videos which I've previously have seen. I actually had in mind that the guy said: "put it to 5 and it will collect till step 5". But he didn't - I had it completely wrong in mind. I think what I wanted is "Vector Observation / Stacked Vectors". Actually I wanted to not continue with my latest AI (cause since days I don't do any game dev anymore - only AI). But now the project is open again and I have to try things out ^^.

Stacked observation sounds like what you meant. I haven't tried them out and don't know how they work in connection with the decision requester to be honest - added it to my list of things to try out. Enjoy working on your project!

I would use Raycasts as observations. The ML-Agents Hummingbirds course in the Unity Learn Platform which is still free at the moment covers Raycasts

Hey. Thanks! 1. Yes, will create a video about that at some point this year. 2. Uhm, you need to change the vector size from 2 to 3 (in Unity Inspector) and then adjust the code like I did for left and right. That should be it. 3. I just taught myself when creating those videos (no prior experience to be honest). I think I can understand it so quick cause of my computer science background 4. You can go as complex as you like :) 5. If you want to take this serious, I'd start with the basics (learn C#, learn Unity Basics and then start looking at ML-Agents) When I first started with coding 5 years ago, I tried Android app development and was quickly procrastinating cause it was all soo complex (like ML-Agents for you I guess). Now I know that Android app development 5 years ago required good Java knowledge. So I should've started with Java instead of Android. There are great videos about C# and about Unity on RU-vid and other websites. Start there, keep going step by step and you'll do great.

Hey, thanks! Not sure why this happens. I would need to see the observation space and code snippet. If you have Discord, you can post it there so that the other users and I can take a look.

That’s weird. Make sure that you wrote it correctly and that you have installed ml-agents in Unity. If it’s still not working, please join the discord server and post the problem there with a few screenshots.

Sure. We get two values from the Agent (vectorAction[0] and vectorAction[1]). For the first branch (vectorAction[0]), the agent can choose between zero and one (in the code we map the decision 0 to 'don't accelerate' (controlSignal.x = 0) and 1 to 'accelerate' (controlSignal.x = 1). If we would want the agent to accelerate backwards we have to set its size to 3 and then map the value 2 to the force -1 in the code (so that he accelerates backwards when choosing the 2). The second branch has the size 3 cause the agent can: 0: don't move left or right 1: move in one direction 2: move in the other direction We need to map those values (0, 1, 2) to valid forces (-1, 0, 1). That's whats happening in the rb.AddForce line. It's important to understand that the Agent just chooses random numbers (in the allowed range) at start of training and we need to move the agent based on those numbers (thats what this part of the code is doing). That way the agent can learn what happens to him when choosing a specific number. Hope I could make it clear, if not feel free to ask again.

@@BotAcademyYT Thanks for this great tutorial. This tutorial is very useful for ml-agent beginner like me. Can I ask you more detail about these part? I can't understand why the first branch(vectorAction[0]) can choose between zero and one. In the code, I have no idea about the decision 0 to 'don't accelerate'.

I've only seen this so far when the Unity extension was not added to Visual Studio (see video at around 1:35). If it's still not working, I'd try reinstalling.

Have you added the packages? What do you mean by not recognized? -> you get a console error when runing the scripts: maybe you forgot the using ML-Agents on the Script -> you dont get autocompletition: maybe you dont have the unity package for Visual Studio Or maybe there is another error in your Script, are you using a self-made one?

U can't. The logic (and all other logic for your Unity environment) needs to be C#. With Python you can only change the reinforcement learning algorithms implemented in the mlagents python libraries (or create & add new algorithms).

I would recommend downloading the version that I created in the video (link is the description) and try if it works with it. If so, try comparing it with your version to find the problem.

@@BotAcademyYT thanks i found what was the problem... In the OnEpisodeBegin() for mistake i referenced my target (transform) local position instead of the (transform) local position of the ball itself causing the ball to have an incorrect starting local position OnEpisodeBegin method keeping updating the ball out of position (outside the floor) so it keeps falling down while updating the Target Position rapidly at same time... Thanks for the rapid Reply.

It shouldn’t matter if you’re familiar with python and know what you’re doing. Steps are basically the same

seems like you're using a different version than I. So double check that you used Release1 with the correct mlagents unity version (the one shown in the video). If you're using a more modern version, check the comment section. People have shared an updated config file in there.

Thats a good point and actually what I did first. But since the ball is rolling the camera rolls too which is not really what we want 😅

very good question. Tried it myself and had the same issue. I'll ask the MLAgents Team about this. Will let you know their answer.

​@@BotAcademyYT Are you going to open a Github issue on their repo.? because I think many people would have the same question. And thanks for your videos, it helped me to set up the ml-agents smoothly.

Thank you, glad that it helped you :) Yes, I tried to open an issue but was redirected to the MLAgents forum when I selected "Question". I think they just want bug reports & feature requests as github issues and I am not 100% if it is really a bug. So I asked in the forum.

@@BotAcademyYT Let's wait till the reply comes from their end.

no answer so far. Just opened an issue on Github to get feedback from the dev guys. github.com/Unity-Technologies/ml-agents/issues/4125

you mean without using their PPO / SAC algorithms? If so, they use Tensorflow. So you can write your own algorithms and connect them via the mlagents python connector

PPs: In my Unity, my target can move one step but after that 'Bake paused in play mode' displays at the bottom right corner and everything shut down. Then I click several of the times the play button, that error displayed at the console.

I‘ve never had this issue tbh. Can you double check that everything is set up correctly?

@@BotAcademyYT Hi, I checked again and build another BallJump project, but still, it does not work. I upload my project on GitHub github.com/XRR422/BallAgent-Jump.git; could you help me solve it? if possible. Save me, please. Thank you!!

@@BotAcademyYT Hi, I rebuild everything on my Windows Desktop again. It fails again and displays 'couldn't connect to trainer on port 5004 using API 1.0.0'. Any solution for this error? Thanks mate!!

@@xingranruan708 API version 1.0.0 is different to the one I used in the video if I remember correctly. Please make sure that the mlagents python version and the mlagents unity version are compatible.

You'd need a modify quite a bit. This video might help you: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-5OVf3FBHzHI.html

@@BotAcademyYT Thank you very much. I am making a video game with ML / agents for my thesis. I have one last question. What type of neural network do Ml Agents use? Do you use a convolutional neural network, a recurring neural network?

the agent logic that I showed is just possible in C#. If you want to change the underlying reinforcement learning algorithms, you can do it in python through the ml-agents python connector

@@BotAcademyYT do you have any resources for that and thx

@@rrkatamakata7874 this should be a good starting point: github.com/Unity-Technologies/ml-agents/blob/master/docs/Python-API.md

@@BotAcademyYT thank you so much you are wonderful person

thanks for letting me know. I'll make a video about it. Might not be the next one though. Currently working on a video where I go through the configuration parameters.

there are only two things I can think of: 1. Make sure that the behavior type is set to default as seen in the video at 16:17 2. Make sure to run the environment directly after running the ml-agents learn command - if you wait to long, it'll time out If you are 100% sure that 1. and 2. are not solving the issue you could try using the exact versions that I used, that is: Python 3.7 Release 1 com.unity.ml-agents@1.0.0

@@BotAcademyYT Thank you for the answer, 1 and 2 point I already checked. I will try to put release 1. I really hope that will help, because "ml-agents" is really a very cool thing.

It definitely is :) If changing to release 1 isn't working, you could try downloading my repository (link in the description) with the exact setup created in the video, import it and try again. That way we can make sure that you are using the exact same environment & agent.

@@BotAcademyYT This is some kind of magic, it really worked. But why? Well, thank you. Next, I will try to figure out what my mistake is.(I put your "BallJump-master")

good that it is working with my code. So your setup is correct. I would check the logic code first, if it is the same or if there is a mistake. Let me know if you figured out what it was :)