CodeLudo 🔥 Daily Guides

100% Accurate and Useless: Pick a Model Before You See Its Score

✓ numbers produced by the code on this page · Published 2026-08-20

Overfitting is usually taught with a picture: a wiggly line threaded through some dots, captioned this is bad. Everyone agrees, and then does it anyway.

The picture fails because it shows the mistake from outside, already labelled. Nobody overfits on purpose. They overfit because the number in front of them is going up, and the number going up is the only feedback they have while working.

So this page shows you what you actually see while working, and hides the rest.

Tune it, commit, then find out

Choose the model the number going up is the trap

Overfitting is usually shown as a wiggly curve threaded through some dots, already labelled as a mistake. Nobody overfits on purpose. They overfit because the number in front of them is going up.

Here are 72 training points and 72 held back, in two classes, with 24 labels flipped by noise. A nearest-neighbour classifier with k = 1 scores 72/72 on the data it was given — a perfect 100% — and 36/72 on data it has never seen, which for two classes is a coin flip. It has memorised the noise.

k = 7 scores only 57/72 on the training data and 50/72 on the held-out data. It is worse at the thing you can measure while working and better at the only thing that matters. Push k higher still and the boundary blurs away: at k=27 it is down to 43/72.

Drag k, watch the training score, commit when you are happy. You get one shot at the held-out data — because a test set you tune against has stopped measuring anything.

The instinct is to push the training score up. At k = 1 it reads 100%, the most satisfying number available, and it is the worst choice on the board.

Why the perfect score is the bad one

python 
GRID = 12


def make_points():
    """A deterministic 2-class set on an integer grid. The true rule is a straight boundary, and
    then roughly a sixth of the labels are flipped - because real measurements are never clean, and
    noise is the thing overfitting learns by mistake.

    Integer coordinates mean every distance is an exact integer, so nothing here depends on
    floating point and the numbers are identical on any machine."""
    train, test = [], []
    for x in range(GRID):
        for y in range(GRID):
            label = 1 if x + y >= GRID - 1 else 0
            if (x * 7 + y * 13) % 6 == 0:          # deterministic 'noise'
                label = 1 - label
            (train if (x + y) % 2 == 0 else test).append((x, y, label))
    return train, test


def classify(point, train, k):
    """Majority vote of the k nearest, by SQUARED distance so the arithmetic stays integral.
    Distance ties are broken by position in the list, which is fixed - so this is reproducible."""
    x, y = point
    ranked = sorted(train, key=lambda p: ((p[0] - x) ** 2 + (p[1] - y) ** 2))
    votes = sum(p[2] for p in ranked[:k])
    return 1 if votes * 2 > k else 0


def score(points, train, k):
    return sum(1 for (x, y, label) in points if classify((x, y), train, k) == label)


TRAIN, TEST = make_points()
noisy = sum(1 for (x, y, l) in TRAIN + TEST if l != (1 if x + y >= GRID - 1 else 0))
print(f"{len(TRAIN)} training points, {len(TEST)} held out, {noisy} labels flipped by noise")
print()
print(f"{'k':>3} {'train':>7} {'test':>6} {'train %':>8} {'test %':>7}  what is happening")
rows = []
for k in (1, 3, 5, 7, 9, 13, 19, 27):
    tr = score(TRAIN, TRAIN, k)
    te = score(TEST, TRAIN, k)
    rows.append((k, tr, te))
best = max(rows, key=lambda r: r[2])
for k, tr, te in rows:
    note = ""
    if k == 1:
        note = "memorises every training point, noise included"
    elif (k, tr, te) == best:
        note = "best on data it has never seen"
    elif k == rows[-1][0]:
        note = "so smooth it has blurred the real boundary"
    print(f"{k:>3} {tr:>7} {te:>6} {tr / len(TRAIN):>7.0%} {te / len(TEST):>6.0%}  {note}")
print()
print(f"k=1 scores {rows[0][1]}/{len(TRAIN)} on the data it was given and {rows[0][2]}/{len(TEST)} on data it was not.")
print(f"k={best[0]} scores {best[1]}/{len(TRAIN)} on the data it was given and {best[2]}/{len(TEST)} on data it was not.")
Output 
72 training points, 72 held out, 24 labels flipped by noise

  k   train   test  train %  test %  what is happening
  1      72     36    100%    50%  memorises every training point, noise included
  3      71     38     99%    53%  
  5      63     48     88%    67%  
  7      57     50     79%    69%  best on data it has never seen
  9      56     50     78%    69%  
 13      55     48     76%    67%  
 19      49     44     68%    61%  
 27      46     43     64%    60%  so smooth it has blurred the real boundary

k=1 scores 72/72 on the data it was given and 36/72 on data it was not.
k=7 scores 57/72 on the data it was given and 50/72 on data it was not.

Read the two columns against each other. They do not move together, and past a point they move in opposite directions — which is the entire phenomenon.

At k = 1 the model is right about every single training point, because the nearest point to any training point is itself. It has not learned the boundary; it has stored the answers, including the 24 that were flipped by noise. On new data it scores 50% — for two classes, indistinguishable from guessing.

At k = 7 it is wrong about 15 of the training points and right about more of the unseen ones than anything else on the board. Those 15 are largely the noisy ones, and being wrong about noise is the correct behaviour.

At k = 27 it averages over so many neighbours that the real boundary is smoothed away too. That is the other failure — underfitting — and it is why this is a choice rather than a direction.

The part that is a discipline, not a technique

The widget hides the test score until you commit, and that is not a UI flourish.

If you look at the test score, adjust k, and look again, you have tuned against the test set. It is now part of your training data in every meaningful sense, and the number it reports has stopped being an estimate of anything. You will get a good score and no information.

This is the mistake that survives into professional work, because it does not feel like cheating — it feels like being thorough. The defence is structural: a third split, usually called a validation set, which you may tune against freely, with the test set opened once at the end. When you have too little data for three splits, cross-validation rotates the role of the validation set through the data instead.

What this tells you about the models in the news

The same asymmetry scales. A language model that reproduces its training data perfectly has memorised rather than generalised, which is why benchmark scores are taken seriously only when the benchmark is genuinely held out — and why benchmark contamination, where test questions leak into training data, is such a persistent argument. It is this page’s mistake, at a scale where nobody can fully audit the training set.

It is also why “the model got 100%” should provoke a question rather than a celebration: on what, and had it seen it before?

What this page simplifies