This is insane. The fact that this is just a script and not a real player still blows my mind. Can you try to make it bugslide and nosebug? That would be even more impressive
imo, part of the problem is the immediate punishment/reward system, so it does what it gives it the immediate reward. Maybe a system that only gave the AI the score after it completes the run could help.
If the physics are so unstable that despite zero input on a flat surface, the initial conditions continues to change with just the advancement of the clock, that's a random number generator, even if not an intentional one. The physics are deterministic only in the loosest sense of the phrase.
I think you are near to prove that the physics is not deterministic, at lest not when time is considered. The physics engine could be influenced by the fps or floating round errors on time values. That would make the game non deterministic to the gaming context. I am sure you can come up with experiments that prove how the variance increases based on seemingly unrelated variables, like you did with time.
I can't say for sure about the randominess of the pipes or the game itself, since I don't play Trackmania xD... But my guess about why the AI is inconsistant, its because of the "self improvement" nature of it. The question of: "Why can't it just be consistant?", fits at the same reason for: "Why it needs to go faster?". Because it is what it was told to do: "Your objective is efficiently achieve the goal". The AI is probably trying to find a more efficient way to overcome the situation, and the more it tries, the more deeper the learning it requires. For example: To be able to go straight, it needs 100XP, but to straight and faster it will need 10000XP, failing might result in "losing points", but not losing XP, since the AI needs this XP to be able to understand why it failed, it will keep falling.
Your problem is that your AI is trying to predict aleatoric uncertainty (uncetrintin inherent to the (catotic) sytem) wich it will never be able to predict. Instead you should try to focus on a reward that pushes the network in situations with high epistemic unceratinty (things that the network can in principle predict but has not learnd so far). In short as you mentioned in the video you'r missing an intrinsic curiosity based reward that lets the network discover more stable strategies.
Scientist on applied physics / mechanics here. Not expert on chaotic behaviour but quite knowledgeable on ordered and disordered states. As there are many different types of chaotic systems, I suggest to plot the failures over distance/time and do a variance study. You can find a lot of information from the slope of the variance of failures vs distance (maybe vs turn direction and/or number of woobly cycles before the turn). Then you might be able to determine the degree of chaos with which you are confronted here, cf. Hurst exponent and fluctuation index in power laws.. Checking perturbation theory (which you kind of did intuitively by inputting small perturbations) might also be a good idea. If the system is periodic, statistical random fluctuations tend to cancel out otherwise diverging results and provide a reliable prediction (cf. Laplace and Newton problem with Saturn trajectory deviation or Three Body Problem). From a mechanical aspect, I would investigate the cyclic nature of woobly pattern, as my guess is that it is there that instabilities arise due to a quasi-periodic movement which can be highly susceptible to very small changes in the floating numbers the game holds and therefore your AI trains. Sorry for the mess if this is unclear ! Hi from France. Keep up the good work !!
If you give the AI two carrots (shortest distance travelled to get to the end / shorts time to get to the end ) I believe that it could yield the results you are looking for.
Did you reach the solution? It seems the problem is NOT the caos. In real scenario, the problem seem s to be a PID process. Imagine water flowing trough a pipe and you add a pressure meter. By the time the preassure meter signal reaches a computer, some time have elapsed, then the commputer changes the pump force to correct the pressure, another time have leapsed. To prevent this, the pressure Proportional, Integral and Derivatives are calculated in order to predict the future pump force necessarry to be applied at the time the signal reaches the pump. So, to solve the problem, even with AI, you need to calculate this parameters, and I think that is why the car is wobling. The problem occurs in drones also, which are in a permanent state of dynamic equilibrium. Search for PID process.
Maybe the problem with the "impossible level" is that you are givin carrots as far as it goes. Maybe if you give carrots as it don't stop touching the pipe it would be better. It is intersting cuz it is almost like we think when we are playing: touch pipe = safe and consistent play
