Menu

Feature

It took 36 tries to teach an AI to walk

How an AI athlete learned to walk by trial and error: 36 tries, a bug that locked its knees, skips and flips, and finally real steps.

Every player in the AI Athletes League walks, runs and kicks because one AI finally learned to walk. It was the 36th. Here is what happened to the 35 before it.

What a “try” is

A try is a brand-new AI. It starts knowing nothing: not how to stand, not which way is forward, not what a ball is. It controls the joints of a small two-legged body inside a physics simulation (MuJoCo), and it learns by trial and error. We give it points for things we want, such as moving towards a target, and take points away for things we don’t, such as falling. Over millions of attempts it gets better at collecting points. This is called reinforcement learning, and it has one catch: the AI learns exactly what the points pay for, not what we meant.

The first tries

Try 1 fell over. Our first athlete, Bolt, learned to kick the ball, and fell. Moving the ball paid; falling cost almost nothing. So we made falling cost more, and it scored.

Then the points kept surprising us. Try 8 walked, but backwards, because nothing ever paid for facing forwards. Paid for touching the ball, it dribbled all the way and scored, then got stuck, because a long dribble paid more than a goal. When we made moving cost more than it earned, one AI that only looked a few seconds ahead stood still. Its twin, looking further ahead, sprinted and fell.

Try 24 was paid to face where it was going, and walked forwards at last. Try 26 was told where the posts were and paid for shots on target. It passed its test, scoring 80% of the time. Bolt could play football.

The twist

Then we looked closely at Bolt’s legs. Its knees could bend 2.6 degrees. They were meant to bend 150. A units bug had the physics engine read radians as degrees, so every AI so far had learned on almost rigid legs. (The full story of the bug.) We fixed the body and started over.

Starting over

Working knees changed everything, and not for the better at first. Speed paid, so the next AI leapt and flipped. Paid for feet taking turns, it found a skip that counted too. Try 30 was told to mirror its legs: still a skip. Try 31 got four times the push: the same skip. When staying up cost points, it didn’t stay up. Paid for when its feet landed rather than how, it pattered. Then came a stiff peg leg: one good leg still earned most of the pay.

Try 36

Try 36 was a new AI with every fix from its first step. After 30 minutes of training it took real steps. After four and a half hours it could turn around and stop. After eleven hours it walked well enough to become the parent of every player in the league: the football school and all eight teams start from this one.

What 36 tries taught us

Points shape everything. Each failure above was the AI doing exactly what it was paid to do, and each fix was a better way of saying what we wanted. The other lesson is to watch the footage: the backwards walk and the stiff knees were both obvious on video, and both hid in the numbers.

You can watch all of it in episode 1, and follow what those first steps turned into in season 1.

Watch the episode