I trained an AI agent to play a game for 76 hours, and it learned from its mistakes.

Meta description: A from-scratch reinforcement agent trained 76 hours on a laptop CPU shows old-tier replay mattered most, while failure replay hurt and plasticity loss was real.

Tags: reinforcement learning, Double DQN, catastrophic forgetting, ablation, plasticity

Featured image: Abstract friendly illustration of a tiny CPU chip, a runner game, and a fading neural network; no people, no logos, no text.

A developer built a from-scratch reinforcement-learning agent, trained it for 76 hours on a laptop CPU, and then tested it with 15 ablation runs. The result was not a simple success story. The agent became strong enough to beat a hand-written player on harder levels, but the most useful mechanism turned out to be the plainest one in the system: replaying already-passed tiers. Two mechanisms designed to learn from failure, and the curriculum that moved the agent forward, each made held-out performance worse when removed.

The experiment was also a test of what happens when a neural network never stops training. Over 226 million steps, the network slowly lost plasticity, matching predictions from recent research. A fix called continual backprop kept the network healthy, but it only sometimes made the agent play better. The code, raw metrics, and numbers are published in endless-learner.

A tiny game and a tiny learner

The environment was a side-scrolling runner game built on purpose to stay small. It included low and tall blocks, gaps, overhangs the agent could duck under, and blocks that slide toward it. From tier 5 onward, obstacles appeared in tight combinations. The agent had four actions: nothing, short jump, long jump, and duck.

Each level was a pure function of a tier and a seed. Tiers had no top: speed, obstacle density, and the obstacle mix kept rising until they hit limits set by the physics. Before any level was handed out, a depth-first search over jump timings checked that it could be finished. The search used the same physics code as the game, so a death was never blamed on an impossible level.

The learner was Double DQN, introduced by van Hasselt et al. in 2016, with 3-step returns and a target network. It used a 155-128-128-4 multilayer perceptron. The forward pass, backward pass, and Adam optimizer were written by hand and checked against central differences, with a maximum relative error around 1e-8. On one core of an i7-1265U laptop CPU, the system ran at a median of 870 to 1,180 environment steps per second. That is slow compared with a GPU, but it made every gradient visible.

The developer wanted an agent that learned from failure and never stopped training. On top of plain replay, two mechanisms were added. Prioritised experience replay, from Schaul et al. in 2016, sampled transitions in proportion to their temporal-difference error. Transitions just before a death were surprising, so they were replayed more often. A death bank saved the game state from 20 decisions before each death, and one quarter of all episodes restarted from one of those states. This let the agent practice the exact situation that killed it, reflecting the idea behind Go-Explore’s “first return, then explore,” from Ecoffet et al. in 2021, and reverse curriculum generation, from Florensa et al. in 2017.

To keep training going, a curriculum promoted the agent when it passed 80% of fresh levels at its tier. Levels with high temporal-difference error were replayed more, an idea from prioritised level replay by Jiang et al. in 2021. To reduce forgetting, 15% of episodes replayed a tier the agent had already passed, and a reservoir sample of the whole history made up one quarter of every training batch.

Performance was measured every 200,000 steps. The agent played 20 held-out levels per tier with no exploration. Those seeds were never used for training. Because a single evaluation round was noisy, each reported number was averaged over a window of rounds. One figure showed the same unseen tier-6 level: the agent after 30 minutes of training in purple, and after 29.5 hours in blue.

Three early bugs set the ceiling

Before the surprising results, three bugs in the author’s own assumptions had to be removed. Each one cost a day, and each one offered a general lesson.

The first bug was that the agent could not see gap edges. The initial observation was nine rays fanning out from the agent. A fixed fan touched the ground at only four distances: 0.8, 1.6, 3.1, and 8.2 units ahead. The agent knew a gap existed, but not where its edge was. In four runs of 1.2 million steps, none got past tier 1, the first tier with gaps. Gaps killed the agent in 26% to 44% of its exams. Adding a terrain scan, where the course was sampled every half unit for 12 units ahead, fixed the problem. All four runs passed tier 1 within 230,000 to 340,000 steps. What the agent can observe sets the ceiling, and no amount of learning gets around that.

The second bug was that exploration hid how good the policy already was. With 3% random actions per decision, a timing-critical game killed the agent often. Exploring episodes passed about 40% of tier-0 levels, while the same network, played greedily, passed 100% of the held-out ones. The curriculum had been promoting the agent on the wrong signal. After the fix, 30% of fresh levels were played as greedy “exams,” and only exams counted for promotion.

The third bug surprised the author most: the policy changed under its own feet. With learning frozen, a snapshot of the network passed 100% of 100 random tier-0 levels. The same network, playing while it trained, passed about 55%. Each update flipped the greedy action in some states, a known effect called policy churn. An episode lasted about a hundred updates, and in this game a jump one tick late was a death. The fix was to act using an exponential moving average of the weights, called Polyak averaging with 0.5% per update, instead of the online weights. Over two seeds, that version reached frontier tier 4 on both, against tiers 2 and 4 for the online weights. Two seeds is a direction, not a measurement.

The agent climbed, then plateaued

With those bugs fixed, the main run climbed fast. The first seven promotions happened in the first 3% of the run. The agent reached tiers 1 to 7 at 0.09 million, 0.25 million, 0.42 million, 0.60 million, 0.75 million, 3.3 million, and 6.3 million steps. That was 2.4 active hours for all seven promotions. Then it spent the remaining 220 million steps, about 74 more hours, between tiers 5 and 7, and never held tier 8. One other run was promoted to tier 8 at 19.8 million steps and demoted again at 34.2 million steps.

For scale, the developer wrote a rule-based player that read the level directly and timed each jump to centre its arc over the next obstacle. That player was perfect up to tier 4, then passed 60% of tier-5 levels and 30% at tiers 6 and 7. The learned agent got 52% to 65% at tier 6 and 49% to 63% at tier 7, depending on the run. At tiers 6 and 7, it beat the player with perfect information, which the author did not expect.

The clever mechanisms hurt

To measure the value of each component, the developer removed one at a time. The ablations used five variants, three seeds each, and 5 million steps per run. The score was held-out success on tiers 0 to 5, averaged over the evaluation rounds from 4 million to 5 million steps.

Two findings stood out. Replaying old tiers was what held everything together. Without it, every seed reached tier 7 at 1.9 million to 3.3 million steps, sooner on average than any other variant. But it forgot the easy tiers almost completely. Averaged over the seeds, it passed only 5% to 23% of tier-0 to tier-3 levels, where the full config passed 61% to 96%. That is catastrophic forgetting in its plainest form, as described by Kirkpatrick et al. in 2017. The author thinks a curriculum can make this worse by design, because it is a schedule for leaving old data behind.

The second finding was that the two failure-learning mechanisms made the agent worse. The curriculum did too. Removing any one of those three beat the full config on every seed, and the seed ranges did not overlap.

What made this result interesting was that both failure mechanisms did their local jobs well. In the main run, the death bank restarted 323,444 episodes and fixed 23,117 failures, in a median of 2 attempts, without giving up on a single one. Transitions just before a death carried 5.5 times the temporal-difference error of routine ones, 0.56 against 0.10, so prioritised replay was focusing exactly where it was meant to. Every local metric said learning from failure was working. The global metric said the agent was better off without it.

The author’s guess is that each mechanism narrowed what the agent trained on, and that narrowing cost more than the focus gained. Prioritised replay oversamples surprising transitions. The death bank spends one quarter of all episodes on fragments just before a death. The curriculum keeps the agent on one tier at a time. Each skew is small, but they stack. That explanation has not been tested directly, and these results were measured with a 5 million-step budget. With much longer training, the balance could change.

The broader lesson is about metrics, not replay. A mechanism can have excellent numbers of its own, such as failures fixed or error concentrated where it should be, and still fail to move the number that matters. You only find that out by taking the mechanism away.

226 million steps: the network lost plasticity

The second question was what happens to a network that never stops training. Dohare et al., in Nature in 2024, showed that networks trained on a changing task gradually lose the ability to learn. Units die, the representation collapses, and new things get learned worse than a fresh network would. A curriculum without an end is exactly that kind of setting.

The effect showed up clearly. The author tracked three things: the share of hidden units that never fire on a batch of states, the effective rank of the last hidden layer called srank from Kumar et al. in 2021, and the weight norm. At 226 million steps, 21% of the plain network’s units were dead. The srank fell from about 117 to 58. The weight norm grew from 24 to 194.

The proposed fix was continual backprop: keep replacing a trickle of the least useful hidden units with fresh ones. A unit’s usefulness was a running average of its activation times its outgoing weights. New units started with zero outgoing weights, so a reset changed nothing until the unit had learned something. The developer ran it on the same seed and configuration. It held dead units at 0.4% and srank at 116 for the full 226 million steps, and a second seed showed the same result. As a fix for network health, it worked completely.

Whether that made the agent play better was less clear. The key comparison was on tier 7, the hardest tier any run held.

For seed 0, the plain run had two long dips. It fell from 59% to 17% between 40 million and 100 million steps, and from 59% to 36% after 200 million steps. The continual-backprop run stayed between 50% and 65% the whole time after its first 10 million steps. From 100 million to 200 million steps, the two runs were level.

For seed 1, the plain run never dipped. It averaged 63% on tier 7 over 100 million steps, slightly above continual backprop’s 58%, while its dead units still climbed to 14%. On easier tiers, tiers 0 to 6, all four runs averaged 74% to 76%.

After seed 0 alone, the author was ready to write that continual backprop saved the agent. Seed 1 is why that sentence is not in the article. The narrower evidence is that plasticity loss is real and continual backprop removes it, but a network with a fifth of its units dead can still play well for a long time. The decay only sometimes turns into worse play. The author’s reading is that continual backprop protects against the dips, but two seeds cannot say how often those dips happen.

Neither version ever held tier 8. Whatever stops the agent at tier 7 is not plasticity loss alone. The author guesses network capacity or the observation, but has not tested either.

What this means for builders

The first practical takeaway is to measure the plain version before adding clever machinery. The most effective mechanism in the system was a reservoir sample and a 15% chance of replaying an old tier. The more sophisticated failure-learning ideas needed an ablation to reveal that they were costing performance.

The second takeaway is that local metrics can look healthy while the global metric stays flat. The death bank fixed 23,117 failures. It still hurt held-out success. The prioritised replay concentrated error where it was supposed to go. It still hurt.

The third takeaway is to run the second seed. One seed gave a clean story about continual backprop saving the agent. The second seed changed that story. For anyone building small reinforcement-learning systems, especially on CPU-only hardware, this experiment is a reminder that visibility, simple baselines, and honest ablations often matter more than adding another clever component.