How to stop lag affecting the results: Use FixedUpdate and fixedDeltaTime instead of Update and deltaTime. Update gets called every frame, so the actual time between varies. FixedUpdate will get called with a constant average frequency (note actual period in real time may vary, but thats why you use fixedDeltaTime). The way it exactly works is that the time since last frame will be added to some accumulator, eg the accumulator is currently 0 and then you get a frame after 0.023 seconds, it will now be 0.023. Then the first thing (before physics) is that while the accumulator is above the fixedDeltaTime (say 0.02), it will do a fixedUpdate and subtract the 0.02 from the accumulator. Then there will be 0.003 left in this example. This is useful as the "leftover" time accumulates, until you get multiple fixedUpdates per frame (of course, you may be getting many anyway if framerate is bad). Also useful is that each fixedUpdate is called before each physics step, so anything affecting the physics (or relying on it) should be done here as well.
On top of this, I'd like to say, the most likely REASON for this unpredictable and seemingly nonsensical lag, is garbage collection. Memory optimization, especially lifetimes and explicit collection timing, are probably the most critical improvements that the game could have used.
filmgbg you kind of can, using techniques like object pooling, where instead of creating/deleting objects, you take from the pool (and create if needed) and return them when not needed. Since the pool has references to unused objects the GC doesn't kick in untill the pool itself isn't needed
Good job! here is my analysis: #1: Put the the code in Unity's fixedUpdate function rather than the update function to avoid the lag influencing the AI. #2: Either give him a longer sight range such that he will be able to spot the dead-ends earlier, or equip him with A* path-planning, such that he will only have to follow way-points. #3: Avoid overfitting by testing him on all the maps at the same time, or by giving him a new randomized map in each iteration. #4: Add one more hidden layer to the neural network: 1 layer with 100 neurons perform equal to 2 layers with 2-3 neurons each!
All good ideas. For #4, it may be worthwhile (given sufficient training time) to use some kind of optimization algorithm to determine the best NN architecture for the application. And on the subject of time, a quicker method of training, like a version of the game with no or simple graphics, would help significantly.
One thing that would help Forrest, is increasing the turning speed with lower distance values, so he has time to turn around before hitting a wall. and also feed him with a backwards very long linetrace as well, so he knows what is behind him, and how he is progressing.
Your observation @3:26 is essentially a damning critique of our western education system: So deep in the basics that the learner never achieves the general understanding that would allow them to navigate challenges they never trained to face.
I am taking machine learning class this semester, your project is very interesting! One possible problem I can think of was the input. I don't think there was enough input to train your agent well. You can play this game well because the environment was fully observable to you. However, your agent didn't have full access to the entire map. Just imagine how hard it would be if you are playing this game in first-person view. Even the top player may not be able to do well if there is some dead end trap.
Just found this channel, so much information and yet you leave the viewer wanting more to sink their teeth into. Great job! Also, looking forward to trying out some of the machine learning games!
I spent DAYS trying to figure out how that demo worked when I was just a kid, not even a teen. It's been with me ever since and when machine learning blew up I kinda felt I understood it straight away, though...only in a very very general sort of way. Thanks for this fun demo.
I loved this series! I think you did a great job of conveying the fun of experimenting with software, and I suspect quite a few of the people who watch this series will be encouraged to try their hand at machine learning as a result. Great job!
You've taken a very different approach to machine learning that has gotten me fired up and ready to dive in... Time to feed that curiosity.. :D Just subscribed to learn more from you.
First time I've seen you talk while moving your lips. Impressed. But real talk, your channel is amazing, and has inspired me to really start looking into machine learning. Keep up the great work!
When Forrest failed on level 4, it looked like he ran directly into the wall at a 90 degree angle. In every other level, he was moving around corners, so walls would always approach on one side and Forrest would know to turn the OTHER direction. If the wall approaches from a head-on angle, Forrest would not know which way to turn since the wall would activate his front-left and front-right equally. I believe including a similar scenario in your training would have corrected this, as you alluded to in the video, but I'm still pretty new to this topic. Love this content, you inject a lot of fun and personality into this topic! thanks for your work
It took me 4 parts of your video only to understand that forest is no more than what you WILL ALWAYS teach him to be and do. I hope we as human are not mere AIs to the Gods. Nice work btw👍
You need to add a gradient for each of forrests "feelers" so that it becomes more negative to be too close to the wall in any one direction. Maybe use a logarithm. It's also better to use a predictive time series so you would actually have forrest running a frame or number of frames behind where he would run, and then anticipate where to go based on values and use a second, deeper network, in order to make decision trees based on which training sets to use in the next steps. And there are multiple ways to do this so you might want to look at making a neural network that tries them all randomly until the best combinations are found based on the same recursion. Then just add a lap marker at the starting point and make forrest drop markers for where he has already been and train him only on not hitting the walls and covering the least amount of distance to complete the tracks, and causing the first lap markers to be ignored at a certain point before the second lap or erase them up to a distance behind him, before they are in his field of vision, which is probably better except for in a few cases, which is a harder problem to case for randomly generating courses, which is another algorithm that is inversely co-variant with the difficulty in solving them, in some aspects. Being able to calculate that in the same means is of value as well, and you can see why now things are boring, and why it is of no value to explore things that don't make my possible as an independent conservative. Of course doing this in ways that affects people is a whole other domain and is in the realm of managing probability and risk. Since you wanted to get political and this is a matter of science involving politics and what we're not the ones doing..
Wow sick series, really enjoyed it, I feel like I understand machine learning much better after watching this than other videos which can get caught up in jargon. You keep it funny and interesting while getting into complex subject matter. I also really appreciate how you analyse and learn from yourself, it's really cool to see your process. Will definitely be watching more of your videos, can't wait for your channel to properly blow up!
This is awesome! For more complicated levels, it'd be awesome if you added a way to run into unexplored areas. Like a reward for moving away from everywhere you've already been. Which would require a real time memory of that as the agents explore. That way you could keep agents from getting turned around and going around backwards.
I watched all the Forrest machine learning series and I have to say they are really amazing. Also, I like how you find new obstacle in every video and give your best to find the problem. I'd like to give my opinion about why I think Forrest doesn't play as good as a human. My guess is that since we can observe the whole track and find which path lead to the finish line while Forrest has sensors around him so he only can see what is in front of him, that's why it can't know which path is the right one when he runs a new track for the first time.
The issue is that Forest was given 5 inputs, say he is running exactly at a wall, he has an equal choice to either turn left or right, causing them to fight each other. try giving him SLIGHTLY randomized benefits. Also front, fleft, and fright should all be double lines, just slightly off from each other, that way if it's a thin pillar, he can detect that it is ever so slightly to one side.
I feel like your features could have been improved, from just distance-to-wall measurements to also using the fraction of the course that Forrest ran (including negative values to punish running backward), as well as a time penalty. That way you could weed out bad habits like running in circles, running backward, and biases against exploration. Of course, all that was just talk, and I'd need to try coding it myself, before I can understand the feasibility of such an endeavor. This was an awesome video series. Hope you work on further projects like this.
Second process that monitors just his forward. When the forward gets very low or close them then calculate average 2 on left then right. Then go the way of higher.
I both found this both extremely painful to watch and amazing to watch your thought process and presentation and closely follows a few of my failures in NN. From the offset it was clear that without any memory, a feed forward state would need more info on it's environments. Ironically I think if you had the NN setup with just 2 more input angles this may have worked but the problem was basic geometry for fitness on your tight hex courses you were essentially getting aliasing so 2 more inputs would have removed aliasing. Very good series and well explained. I have found that if you can think of what you would need to solve a problem and going further in deep learning the number of functions you may need to use to solve a problem + wether it needs to remember some state this can give you good clues on how many inputs you need, what layers you need and what activation functions you need on those layers.
I think one of the problems your forrest faced is that he couldn't see around the corners like you could. Maybe weighting the average so that the straight ahead reward value was 2x that of the 45 degree value it might have helped. At some points he had to just guess because he couldn't see around the corner as well. Bear in mind this is coming from somebody that has no actual experience in machine learning, it's just some observations. I'd love to get into it, but I really have no idea where to start. Anyway, incredibly interesting and thanks for the video!
After you said this would end up on github (episode 2?) I planned to build upon it in some way. In the past there have been many projects where I've gotten far enough into it to realize that it was not a task to take on at that moment as it would basically require building a game engine of sorts to be able to test any of the code. This series does remind me of my MTG neural net that never learned...I should take a look at it again (not true MTG, really simplified sorcery speed burn vs vanilla creature).
Maybe you should make a way for him to see around corners like put a T shape sight pattern. It's like when you use your mind to figure out a maze or find your way through a new part of a city or town you have never been to. It will be like giving forest an imagination.
6:55 If your laptop has a dedicated GPU it is possible that it has hardware encode/decode. If you change your encoder option in OBS from x264 to NVENC (Nvidia's encoder) it will reduce the strain on your CPU. AMD has the same thing and it's also possible with intel igpus apparently, using intel's "quick sync", but I have never tried it. obsproject.com/forum/resources/how-to-use-quicksync.82/
I just wanted to say how much I enjoyed following this AI project you put together! I'm thinking about learning python so I can start a similar process for mastering games by emulating AI that has solved it - like 80's retrocade games. This was really educational and I appreciate all the resources you included - especially being a visual learner! I subscribed and hope you wanna do more ML / AI game oriented videos in the future! Also, is that Eco Virtual you ended the video with?
loved your analysis of the subject of ai/machine learning. I want to learn about this myself and your tactics are extremely approachable! TY for the community at large. this is a subject that is hard to get into with little/no knowledge but the way it is presented.... hats off! it's great! and I hope to learn more soon , as tis a subject that I'm deeply interested in! That is all! ty for your vids!
I love your videos! The way you explain things makes so much sense, especially after seeing countless other youtube videos that just didn't give me enough of a reason to care about the minutia of ML. I would also love to help out on your video game using whatever skills I have! :)
Probably spacing the rays of distance evenly every 45 degrees isn't enough/the best option. Try spacing them unevenly but symmetrically, e.g. 0, 25, 90, 155, 180 degrees. Also increasing the number of rays might help a lot. Good video series nevertheless. Kudos
I definitely think using distance from the walls as fitness was a main problem. I would use length of survival and number of laps, which is what the actual objective is
Oh yeah, regarding Thing #5, when you added little "rooms" with hexes inside them, you added something akin to cycles in a graph. That would require you to create some way for Forrest to memorize which "hex-spaces" he had already visited. I'm now wondering whether or not, and how, a neural network could figure out a graph traversal algorithm. o_o
What if we are all just variations on Forrest Gumps-asked to figure out puzzles of the universe. With each generation hopefully slightly closer, although it seems like plateauing happens a lot
Amazing video series! Looking at the reason why Forrest failed, I would argue that the 5 lines as inputs might not be enough information to determine a solution to every course. This would basically mean that the global minimum for the system is not low enough to solve every course. Thinking about this and looking at the graph at 3:30, I would assume that the closer the lowest point to is to the minimum required to solve the problem, the more difficult it is to determine, weither the problem has a solution. Meanwhile, a narrow valley would mean that the probability of finding it is lower and therefore requires more training time. A practical example of a narrow valley could be Forrest wiggling left and right to get more information from the environment, similar to the way some birds move their head while walking to increase spacial awareness.
What about creating an algoritm that will create map`s based on forest skills? so for example if forest is starting it will create straight without many curves, and going harder and harder?
Aw :( I'd really like to see this continue with all you've learned. Is there a possibility to add a "memory" to Forrest? Like, a memory input that gets changed when a new output gets triggered?
I know I'm way to late for this but still I'm gonna say some things that i think might help. First, with just 5 directions taken into account for the inputs you're bound to have problems with complex shapes, hexagons in this case. I would add some more, maybe 4 or 6. Maybe it's overkill, but i think at least 2 should go to his back in order to help him reinforce faster the idea to not turn back, which takes me to the second point. You should do the map creation tool with something to signal the main direction you must go, Something like a line which goes from beginning to end. Would be easier to explain with some drawings but i hope you get what i say. I think it might be a good way to fix the fitness function because having the goal and the start be the same makes it a bit harder to fix. The other way is to add a punishment based on a past position. For example, if the next move gets you closer to where you were half a second before you should give it a punishment as it might mean he's turning around or going in circles. Also punish over time as to get a network that tries to get faster to the goal and doesn't stay running in circles. Also you should maybe give a try to different ways to make the neural network, or different ways to make it work. Backwards propagation and the other options. At least try, don't know which suits this case best. Apart from all that, i think you said mostly everything you should be taking into account for the future, and it's been very interesting seeing this, I think I've learned a lot, so thank you for this awesome idea. Also, love the way you present all of this, you make something really dense and complicated very entertaining to watch ^^
in real world race, even tough racers run at their optimum speed, they slowdown on a turn so they won't jump into spectators arena, if they are racers who don't slow down, they compensate it with something called as drifting, or in motorbike race they lower their body position in a diagonal way but still they have to brake a little
i would like to ask a couple of questions: 1) would a scale value offset to the lap complete multiplier to account for lap time, maybe it will pickup routes better? 2) is it possible to have a state machine choose the algorithms for training to switch depending on level of success vs low growth/variation over time? cool project thanks for the info!
Am I so arrogant to say that you didn't use long enough lines? Forest would go down a path he never knew was a dead end until it was too late. Stationary penalty, and backwards penalty. For stationary, draw a temporary trail behind him in the areas where circling can occur, and if he connects with the start of the invisible line, take away a portion of his reward, he loses when he hits 0. For Backwards, make directional boxes overlapping the area of a track where he could potentially go backwards, if he advances in the wrong direction, he loses. Speed, for an obstacle the AI doesn't know how to tackle, slowing down will make it easier to process. With the hexes, this makes it easy. Give a speed penalty with how close Forest is to the wall, by overlapping another invisible hex over the original giving it an area of slowness, this will also potentially solve the circling issue. The faster the AI is, the more reward he will get. Also give it a minimum it can't go under, making it try to not make as many mistakes as possible. I know nothing of AI,Neural Networking, or coding in general. So apologies for talking out my rear. If my thread can be dismantled, I will take a point down, and maybe the entire thing, if I am indeed wrong.
Don't forget the last possible reason. Not enough inputs. 5 lengths are really not many and could make it impossible to reach the goal. I don't think it's the only reason, but actually it could be.
Lol I’m actually watching this to try to create an AI for strategy games that can react more dynamically and make calculated decisions to achieve an end goal, while having some flexibility in how to achieve it. These videos showed me a lot of the problems I’m going to run into but I am working to implement this technology into gaming too.
simple english incoming : you got 5 inputs which are the distance from forrest to the opposit wall. my thought is : forrest should try to alwys be at the centre, like being the same distance form the left wall as to the right wall ? It's 2 am soo don't expect perfect english :s
Hello, I think that your videos are pretty amazing and difficult not to watch once you start. I wanted to see if you would not mind sharing what tools you use for developing these games as well as videos. Personally I am a software developer with decades of experience in building variety of apps, but never anything with significant amount of graphics or AI. I would really like to do some self-learning projects as well as interest my kiddos in programming. We have played with MIT Scratch in the past; however, I really enjoyed your style of learning as well as presenting. Great Job!
Dude this is really cool. I'm still just beginning to learn python but I wanna move into working with ai later on and I'll definitely be referencing your projects ^^ I like your style of video btw sick of most youtubers just screaming at my screen haha
Instead of training gump on turning left and right, wouldn't it have been easier to have the ai learn to turn clockwise or anticlockwise, thus reducing the number of variables.
Wow ! I wish i could do something of this the level 😅. I will try to do something like this but with a PID, its more in my range... Keep up the good work !
Why not auto-tune the number of variables in the thought process to be equal to the number of variables it discovers while exploring each level, thus creating a network which can grow smart connections on the fly, increasing in complexity to keep pace with the more difficult levels. That way you can train it on easy circle level first for a bit, let it auto-tune the size of its network, use that as a baseline, then show it harder levels and it let it grow naturally.
As someone who made this exact thing before for my college conclusion project... You are going through the same problems I had to. Look, some of the problems I can see just by watching the videos, you were right in some problems and didnt mention others. The problems I see are: 1. Network Design, you were 100% correct on that one. Maybe some more nodes on the hidden layers could've helped. Maybe more inputs, instead of 1 input for each 45 degrees, 1 each 30 degrees wouldve made for better sensory input. 2. One thing that you didn't mention, you could increase the sensory length for the inputs. Some of the failures shown in the videos makes me think that Forrest didn't know what to do because he couldn't see far enough to find the walls and became loss in what could be described as open nothingness. There's not much of a way to know what to do in these situations because any input would be good. I'd say iterate a bit with the sensory length or allow the brain to decide to slow down forward movement so as to give it more time to turn with suddently facing a dead end he couldn't have predicted for lack of sensory data, remember we humans see VERY far away for a reason, we don't just see nothing about 2m in front of us. 3. More training. Mutation rates and random TP additions can help getting off the local minimum so if you feel like you're stuck just give it more time, maybe manually add a new TP to try leaping through the barrier. 4. Courses. Hex courses wasn't really the problem, you kind of mentioned it in the videos but I'll reiterate: You should really make only one training course, and that training course has to contain EVERY POSSIBLE OBSTACLE Forrest might face. The perfect course would have to have every possible curve, dead end, open zone, etc. This way it can train to respond to any thing. That is because artificial inteligence is only that, Artificial. What it's actually doing is recognizing patterns in its sensory data and responding accordingly. It cannot respond correctly to a pattern it's never been trained for, so the training course must have every possible pattern for it to recognize and respond well. [ 5. Penalties. You are absolutely correct, he needs a penalty if he goes the wrong way. A lot of stuff got fixed after I started subtracting points (with an *-2 multiplier on the distance ran the wrong way) from the fitness score on my project. That's what I can remember rn, but you did really good the way you did, so don't get frustrated about it!
Hey, question. What would happen if you put logical bounds on the output of the neural network. Like if you're about to be too close to a wall to avoid a collision, override the output and turn to avoid the collision, then give control back to the ANN when you're out of harms way? A hybrid approach. Use the neural network for navigation, but have an emergency hard-logic fallback for imminent collisions? Let me know what you think. Overall, great series. I subbed. Well worth watching, will recommend to friends.
First of all i have to say i have zero knowledge in ML. But watching the last runs it seemed like Forrest did not have a lengthy enough vision to anticipate the oncoming obstacles, it seems like he did not have anough time to calculate his route. Maybe that was because of the lag you mentioned. I feel like giving him a bigger amount of vision makes it more able to anticipate the route and also to implement code to remember the place he came from so he does not go back to where he came from.
Putting everything in FixedUpdate would avoid the issues related to lag and capture.... Of course, if you're seriously lagging, the game will run slower, instead of time skipping ahead and crashing Forest.
It seems like they run straight whenever distances to all sides are the same (just as when x or y-angle are the same which is right) - wouldn't it be possible to train it in "trapped" situations to learn not touching walls or maybe measure the distances differently (something like EM-Algorithm to measure the incoming data in order to detect structures and encourage a different output if they detect similar values from all inputs?) Is a completely groundless suggestion though, i have never actually tried it
That is a really good project Thank you Jabril. But I am unable to open the C# project in Visual Studio Ultimate. Please tell me which version of Visual Studio is needed. better send me link for installation. Thank you!!
You should revive Forest and train him on a proper course. Design one using good ontology. A track using hexagons is not the best approach as Forest searches the surroundings and is making decision using only 5 directional vectors. And when using those vectors, and hexagons, you have all the time at least one chance to encounter a perpendicular at 45 degree wall angle vs a left forward command (or right forward). If not two, at the same time. This means Forest will choose the weights in a pure random fashion, like in a lottery (because his interpretation on that particular direction is like hitting a wall at 90 degrees). Multiply this several time during the course of the game and you will see that he actually performed much better than expected. Seeing the algorithm choosing the strategy to run in circles made me laugh. :) It really outsmarted you, despite the fact you thrown at it the worst AI learning design possible (and here I am talking about hexagons, not the difficulty of the course). Don't be lazy, design one where the program could actually learn a real predictable strategy (this is not a lottery), and you will see different results.
Have you considered that perhaps your 5 inputs don't provide enough information to complete the task. Would you be able to create a rules-based runner that only worked with those 5 inputs? Your bird's eye view provides you with data that "Forest" did not have.
Hi Jabrils, I downloaded your official release of your game and was wanting to use your project for my deep learning class in college. Would you be able to post a video or documentation on how to use your interface. It looks like you need to upload "Forrests", a python neural network I suppose, into the Forrests directory but I'm not sure what format it should be in. It'd be great if you could provide a tutorial or documentation on how to interface with your game. Thanks!
Issue: when Forrest sees an obstacle that divides his path, he may come to situation of life and death. Without more knowledge he is doomed to choose based on his experiences, and if he can't, he'll do some random shit. Of course, it's random in our perspective, for Forrest it's perfectly fine choice based on his past experiences. In order for him to improve would be to: a) learn the course. Not what we want, but it'll solve the puzzle b) learn the way that the courses are built, so he can avoid such situations. It's impossible with such a small neural network, but technically he could overcome the challenge by learning the way you create the track to predict which crevice is going to lead him to certain death. Certainly more fun, but still not solving the main issue of the Forrest killer: his perception. Your project was not a failure at all. You've tried to guide a blind man through a maze telling him how far are obstacles in selected directions. Considering this, you bot was doing more than fine. He even had great successes.
Seems to me it failed because Forest had no memory. Thus he couldn't determine specific features of a track - only what he can see right now, which is only part of a feature, leading to the running in circles etc. To illustrate let's say you had two forks: one lead to a dead end, and the other lead to the finish line. Even in the best case situation in which Forest turned around at the dead end, once he returned to the fork he would have no way to determine which road to go down again since he would not be able to remember that he just tried one of them, and so he could well get stuck in an infinite loop. A possible solution to this that would work in a neural network with no memory would be for Forest to follow the left/right wall all the time, but that's no fun. So if you did some sort of Neural Network which used past states (rather than just present) in its calculation of future states then you may get better results.
I know this is old, but I'd strongly suspect that there is no thought process of an allowed design that can handle every possible valid course. For the hex map design, particularly when allowing for both large open areas and freestanding columns, only the five wall-distance inputs just might not be enough of an input. It feels like it wouldn't be too hard to deliberately design courses that would break any otherwise successful thought processes. Beyond that, the measure of success was poorly chosen. Wall distance aids in measuring survivability, but survivability doesn't measure course progress when Forrest can freely run backwards and run infinite loops. As for the lap multiplier, it takes too long for it to trigger, particularly for larger maps. You really need some way to reward getting closer to the goal, either by implementing a GPS-route-style distance to the goal or by having the map generator automatically generate multiple checkpoints that can be used to mark progress.
