Our game analyser had been running in production for a while, grading every move of every game people played. It looked fine. Evaluations were sensible, the graph moved when the game turned, the labels were mostly where you would expect them.
Then we pulled ten complete games out of the database — 542 half-moves in total, a mix of weak-player games imported from Chess.com and famous historic games — and counted how often each label actually fired.
"Blunder" had fired zero times. Not rarely. Never.
Two reasonable rules that cancel each other out
To be called a blunder, a move had to fail two tests at once: it had to lose a large amount of material in centipawns, and it had to lose a large amount of win probability.
Both of those are sensible measurements on their own. Centipawn loss tells you how much the engine's evaluation dropped. Win probability converts an evaluation into "how often does this position actually get won", which correctly captures the fact that being three pawns up is nearly decisive at +3 and almost meaningless at +12.
Requiring both looks like sound defensive design. It is, in fact, close to impossible to satisfy — because the two quantities are anti-correlated in exactly the positions where blunders happen.
- Near equality, a small centipawn loss swings win probability sharply. Drops big enough to clear the material threshold are rare here.
- Once either side is clearly ahead, win probability saturates. At 97% you can throw away enormous amounts of material and barely move the number, because you were winning before and you are still winning now.
So the rule demanded a large move on a scale that flattens precisely when the other scale gets large. There is almost no position in chess that satisfies both.
The move that broke it open
The clearest single case was move 71 of one game. The player played 35.Kf1, walking directly into a forced mate.
The evaluation drop was 8,355 centipawns — the equivalent of handing over about eighty-three pawns. The win-probability loss was 1.5 points, because the position was already lost.
It was never even a candidate for "blunder". The analyser labelled it "good".
Once you see it, the fix is obvious: the gate has to be OR, not AND. A catastrophic material loss is a blunder whether or not the game was already decided, and a catastrophic swing in winning chances is a blunder whether or not much material moved.
We also graded the severity bands by context, because "how bad is a 60-centipawn slip" genuinely depends on the position. In a dead-lost position it is noise. Near equality it is a real mistake.
The other direction: almost every great move was "forced"
The same audit found a mirror-image failure.
The analyser marked moves as forced when the engine's best move was far better than its second-best — a gap of more than 300 centipawns between the top two candidate lines. The reasoning is that if one move is enormously better than everything else, the player did not really have a choice.
That sounds right and is completely backwards, because it measures how much better the best move is rather than whether alternatives existed. And there is one situation where the best move is always enormously better than the alternatives: a winning combination.
So the system systematically labelled the finest moves in every game as though the player had been dragged into them. Morphy's queen sacrifice in the Opera Game. Anderssen's finish in the Evergreen Game. Fischer's entire windmill combination against Donald Byrne.
Measured across the ten games: of 53 moves labelled "forced", 42 — seventy-nine percent — were produced by that heuristic and were false. We deleted it. Only a genuinely single legal move counts now, and the count dropped to 11: exactly the number of positions in those games where the player really did have one option.
A third label, "great", turned out to be unreachable dead code. It had never fired once in 542 moves.
The accuracy score had the same disease
Win-probability saturation caused a second, subtler problem. Because a losing player's win probability barely moves once they are lost, every subsequent move scored as near-perfect.
One game in the sample was played by someone rated around 233 who hung their queen on move seven. The analyser gave them 89.9% accuracy.
The fix is to weight each move by how much the win-probability curve was actually moving locally — so moves in sharp, volatile positions count heavily and moves in dead positions barely count at all. The same game now reads 75.4%, which is a fair description of what happened.
How to find bugs like these
This is the part worth stealing, because none of the above was found by reading code. We read that code several times. It looks correct, and in a narrow sense it is: every line does what it says.
What worked was making the classification replayable offline.
Our analysis records store every input the classifier receives for every move — both evaluations, the centipawn drop, the win-probability loss, the engine's principal variation. That means the entire labelling step can be re-run from stored data with no engine running at all, in a fraction of a second, against modified thresholds.
Which turns threshold tuning from an argument into an experiment. Change a constant, re-run 542 moves, diff the label distribution before and after. When we predicted the effect of a change in advance and then measured it, the predictions landed exactly — "forced" 53 to 11, accuracy 75.4%, to the decimal.
The counting is the whole technique. A label that fires zero times in 542 opportunities is broken, whatever the code looks like. A label that fires on 79% false positives is broken. You cannot see either of those from inside the function — you can only see them by running it over a corpus and counting.
Why historic games are the right test set
One more thing that made this tractable: we tested against famous games.
The advantage is that the correct answer is already public knowledge and has been for a century. Nobody needs to adjudicate whether Rubinstein's rook sacrifices are brilliant, or whether Morphy's queen sacrifice was forced. If your analyser disagrees with a hundred years of consensus, your analyser is wrong, and you do not have to argue about it.
Weak-player games are equally valuable in the opposite direction, because they contain the actual blunders. A test corpus of only grandmaster games would have hidden the blunder bug completely — there simply are not enough catastrophic errors in master chess to notice that the label never fires.
Ten games. Half brilliant, half terrible. That was enough to find five distinct bugs that had survived every code review.