A program where agents (carts) with neural network brains evolve/learn to balance a pendulum (pole) upwards.
Press "Play" and watch as the little AI agents attempt to swing the pole vertically upwards. The brains are initialized randomly a bit analogous to a newborn, so it will fail miserably to start.
Each agent will have 20 seconds to perform, after which the best few from the generation will used to produce agents in the next generation. This can take a while, so press the "Speed" button to fast-forward the evolution process.
Watch the graph below which shows the carts slowly gets better at balancing the pole. Press the speed button again to go back to regular speed and see observe how they have improved.
Usually after ~1000 generations (a few minutes in fast-forward mode), the agents will essentially have perfected the swing-up-and-balance maneuver. On the y-axis, this is a score of ~2200.
There is kind of a lot going on for this to work, given basically everything was written from scratch in JavaScript (except the plotting library). I won't explain it all here (just a high level overview of the components), but feel free to ask me IRL bc I am really passionate about these kinds of computer programs. 😊
At the lowest level, the graphics were built with raw JavaScript and rendered using my own UI library canvasLib.
At the next level of abstraction, the actual CartPole simulation was an adaption of the spring-mass system I developed for my Double Pendulemon project. CartPole is already a common benchmark for AI agents, but I enjoy making things from scratch.
The neural network is a class I made with raw JS (only feedfoward bc no backpropagation is needed). It's probably the bottleneck for the simulation given there is no matrix multiplication (just for loops).
The final layer of the stack is a genetic algorithm that allows the agents to evolving to improve their balancing abilities. There are many details to this that I can perhaps explain another day.
The plot was made with plotly.js.