Overfitting and how to avoid it

What you will learn

• What overfitting is and why it is the most common reason backtests fail live.
• How the number of tests you run creates "winners" by pure chance.
• How to read a parameter grid and prefer a plateau to a peak.
• Practical rules for simpler, more robust systems.

Explanation

Curve-fitting was introduced briefly in the course "Technical Analysis in Depth". Here you learn to measure it and to design tests that resist it.

What overfitting is

Every price history contains two parts: a possible repeatable pattern (the signal) and random movement (noise). Overfitting means the rules have adapted to the noise. The backtest then looks excellent, because noise in the past is fitted perfectly, but the noise will not repeat, so live results are worse. The more choices you make while looking at results, the more noise you fit.

Degrees of freedom

Degrees of freedom are all the ways a rule set can bend to the data: each parameter, each filter, each "only on Tuesdays" exception and each market you drop because it did not work. A frequently used rule of thumb is to keep the number of trades large relative to the number of parameters, for example several dozen trades per free parameter; treat this as a minimum, not a guarantee.

The multiple-testing problem

Suppose you test 20 rule variants that truly have no edge, and you call a result "significant" if it would happen by chance only 5% of the time. If the tests are independent, the probability that at least one looks significant is:

1 − 0.95^20 ≈ 0.64, so about 64%.

With 100 variants it is about 99%. The best of many tests is therefore expected to look good even when nothing works. A simple correction, the Bonferroni adjustment, divides the threshold by the number of tests: 5% ÷ 20 = 0.25%. Real tests of similar variants are not independent, so the true numbers differ, but the direction is always the same: more tests require stronger evidence. Keep a research log that records every variant you tried, not only the final one.

Plateaus versus peaks

A parameter grid shows the result for many parameter values. A robust rule shows a plateau: neighbouring values give similar results. An overfitted rule shows a sharp peak: one value works and its neighbours do not. If the edge were real, a 9-month or 11-month average should not behave completely differently from a 10-month average.

Rules for robustness

• Fix parameters from the hypothesis before testing; optimise as little as possible.
• Prefer fewer rules. Each filter must have its own reason, not only a better backtest.
• Choose a parameter from the middle of a plateau, not its highest point.
• Test the same rule on related markets without changes; a real effect usually appears in several.
• Expect live results to be worse than the optimised backtest and plan for that.
• Keep untouched data aside for a final check (lesson 7).

Worked example

Hypothetical example: two systems were each tested with seven parameter values. The table shows the annualised Sharpe ratio (lesson 5) of each backtest; costs are included as a fixed estimate, taxes are excluded.

Step 1, best single value: Y at 8 (0.95) beats X at 10 (0.47). A naive tester picks Y.

Step 2, neighbourhood average (the value and its two neighbours):

• X at 10: (0.44 + 0.47 + 0.45) ÷ 3 ≈ 0.45.
• Y at 8: (0.12 + 0.95 + 0.10) ÷ 3 = 0.39.

Step 3, share of values with Sharpe above 0.30: X 6 of 7; Y 1 of 7.

Step 4, multiple testing: 14 variants were tried. With 14 independent no-edge tests, 1 − 0.95^14 ≈ 51% chance of at least one false "winner". Y's isolated 0.95 looks like such a winner. X shows a plateau and is the more credible system, although it still needs a test on unseen data.

Common mistake

Practice activity

Hypothetical grid of annualised Sharpe ratios for five breakout lengths (20, 40, 60, 80, 100 days). Rule A: 0.30, 0.34, 0.36, 0.33, 0.31. Rule B: 0.10, 0.62, 0.08, 0.05, 0.12. On paper: (a) calculate the neighbourhood average around each rule's best value, (b) decide which rule is more robust, (c) calculate the chance of at least one false winner among 10 independent no-edge tests, and (d) the Bonferroni threshold for 10 tests at 5%.

Key takeaways

• Overfitting means fitting past noise; it makes backtests look better than future results.
• The more variants you test, the more likely a chance winner; record every test.
• Prefer a plateau of good results over a single sharp peak.
• Few parameters, reasons for every rule and tests on related markets improve robustness.

Next step

In the next lesson you will use out-of-sample and walk-forward testing to check a rule on data it has never seen.