Out-of-sample and walk-forward testing

What you will learn

• The difference between in-sample and out-of-sample data, and why the test period may be used only once
• How a walk-forward analysis works, with rolling and anchored windows
• How to calculate walk-forward efficiency and a stitched out-of-sample result
• Practical rules for window lengths, trade counts and parameter stability

Explanation

The best defence against overfitting (previous lesson) is testing rules on data that played no part in building them.

In-sample and out-of-sample

In-sample (IS) data is the period you use to develop the rules and choose parameters, such as the length of a moving average. Out-of-sample (OOS) data is a later period that you keep locked away and use only to check the finished rules. A common simple split is about 70% in-sample and 30% out-of-sample, with the out-of-sample period at the end, because the future always comes after the past.

The out-of-sample period has value only while it is unseen. If you look at the result, change a rule and test again, it has become in-sample data. This is called data snooping: reusing the same data until something looks good by chance.

Walk-forward analysis

One holdout period gives only one verdict, and that verdict depends heavily on the market conditions of that period. Walk-forward analysis repeats the process many times:

1. Optimise the parameters on an in-sample window, for example four years.
2. Apply the chosen parameters, unchanged, to the next out-of-sample window, for example one year.
3. Move both windows forward by the length of the out-of-sample window and repeat.
4. Join all out-of-sample results into one stitched out-of-sample equity curve.

In a rolling walk-forward, the in-sample window keeps a fixed length and drops old data. In an anchored walk-forward, the window always starts at the first date and grows. Rolling windows adapt faster; anchored windows use more data and usually give steadier parameters.

The number of windows is: (total years − in-sample years) ÷ out-of-sample years.

Walk-forward efficiency

Walk-forward efficiency (WFE) compares unseen performance with optimised performance:

WFE = average annualised OOS return ÷ average annualised IS return

A WFE close to 100% means the system performed almost as well on new data as on the data it was tuned on. A low WFE means much of the in-sample result came from fitting noise. Some practitioners use roughly 50% as a minimum; this is a convention, not a proven threshold. Also check how many windows were profitable.

Practical rules of thumb

• Each out-of-sample window should contain enough trades to mean something; aim for at least 20–30 trades in total across all OOS windows, and more if possible.
• Each in-sample window should cover rising, falling and sideways phases.
• Prefer parameters from a stable "plateau", where neighbouring values also work, over a single sharp peak.
• Write down the window lengths and the selection rule before you start, so you cannot change them after seeing results.

Worked example

Hypothetical example: a trend-following system on a share index is tested on 10 years of daily data. Assumptions: rolling walk-forward, 4-year in-sample window, 1-year out-of-sample window, step of 1 year. Returns are hypothetical and exclude fees, spreads, slippage and taxes.

Number of windows: (10 − 4) ÷ 1 = 6 windows.

Step 1: average IS return = (14 + 12 + 15 + 11 + 13 + 12) ÷ 6 = 77 ÷ 6 ≈ 12.8%.

Step 2: average OOS return = (6 − 2 + 8 + 4 − 5 + 7) ÷ 6 = 18 ÷ 6 = 3.0%.

Step 3: WFE = 3.0 ÷ 12.8 ≈ 23%.

Step 4: stitched OOS growth = 1.06 × 0.98 × 1.08 × 1.04 × 0.95 × 1.07 ≈ 1.186. Only the unseen years count, so €10,000 grows to about €10,000 × 1.186 ≈ €11,860 over 6 years, about 2.9% per year compounded.

Interpretation: four of six OOS years were positive, but less than a quarter of the optimised performance survived. The in-sample figures strongly overstate the rules. Next, simplify them and test again on genuinely new data.

Common mistake

Practice activity

On paper: you have 12 years of hypothetical data and use a rolling walk-forward with a 3-year in-sample window and a 1-year out-of-sample window. (a) How many windows are there? (b) The first four windows give IS returns of 10%, 9%, 11% and 10%, and OOS returns of 5%, 6%, −1% and 4%. Calculate the WFE for these four windows. (c) Calculate the stitched OOS growth of €10,000 over these four years.

Key takeaways

• Out-of-sample data is valid only while unseen; reusing it turns it into in-sample data.
• Walk-forward analysis repeats optimise-then-test many times and joins only the unseen results.
• Walk-forward efficiency shows how much optimised performance survives; low values point to overfitting.
• Fix window lengths, selection rules and minimum trade counts before you look at any result.

Next step

In the next lesson you will add realistic costs and slippage to your backtest and see how much of a gross result remains.