Session 2 — Generalization: choosing the flexibility without peeking at the test dots

Boss: round 1 of 3 · wins 0
Run cross-validation below, pick a degree, then press Lock my pick. A round is won if the pick sits on the floor of the test curve (within 1 of the best, or within 15% of its error). 2 of 3 wins.

A · The three-way split

Homework (training) · mock exam (validation) · final exam (test, sat once, at the very end). The curve is fitted on the training dots only.

training dots (homework)validation dots (mock exam)test dots (final exam, hidden)fitted curvehidden truth
Error on training dots
–
Error on validation dots
–
Error on test dots
???

Error vs degree. Training error only falls. Validation error makes a U. Pick the bottom of the U (the validation line), then reveal the test line to see how close you were.

B · Cross-validation: everyone takes a turn as the judge

All non-test dots are cut into k slices (folds). Each slice gets one turn as the mock exam while the model is fitted on the other slices. Average the k scores.

Press Run cross-validation.
Folddots in slicevalidation error

CV-average error vs degree (red). The single-split validation line from panel A is drawn faintly for comparison.

C · Bias and variance made visible

20 fresh datasets from the same truth (same noise, same number of training dots), each fitted at the current degree. Bias = how far the average curve is from the truth (consistently off target). Variance = how much the curves scatter around their average (jumpy archer).

20 fitted curvestheir averagehidden truth
Bias² (average curve vs truth)
–
Variance (spread of the bundle)
–
Bias² + Variance
–

The same two numbers for every degree 1–15 (20 datasets each). The U from Session 1 is bias² + variance, plus the noise we can never remove.