A Complete Research-Notebook Project from Data to Report, with a Code-Review Checklist against Look-Ahead Bias and Overfitting
Organise a full strategy study as a reproducible project, run it from raw data to a written result, and review the code with a checklist that catches the errors that flatter backtests.
What you will learn
- A folder and notebook layout that makes a study reproducible by you in six months or by anyone else.
- The order of work: data, cleaning, features, signals, backtest, metrics, out-of-sample check, report.
- A line-by-line code-review checklist for look-ahead bias, survivorship bias and overfitting.
- How to write a result honestly, including what was tried and did not work.
Explanation
Out-of-sample testing, parameter plateaus and the cost of many trials were covered in Systematic Trading (lessons 6 and 7). This lesson is about the engineering that makes those ideas hold: a project structure, a fixed workflow and a review another person could repeat.
Project layout
project/data/raw/ # downloads, never edited by handdata/clean/ # parquet files produced by 01_data.ipynbnotebooks/ # 01_data, 02_features, 03_backtest, 04_reportsrc/ # indicators.py, engine.py, metrics.py (lessons 6, 8, 9)tests/ # test_metrics.py, test_engine.pyconfig.yaml # tickers, dates, parameters, cost assumptionsrequirements.txt # pinned versions: pandas==2.x, numpy==…
Reused code lives in src/; notebooks only call functions and show results. Pin package versions, because a pandas update can change a default (frequency strings, resampling labels) and move your numbers. Seed any randomness and record the data download date in the report.
The workflow in order
- Hypothesis first. One sentence of economic reasoning and the exact rules before touching data (Systematic Trading, lesson 2). Fix the in-sample / out-of-sample split now and do not open the out-of-sample file until step 7.
- Data. Download, store raw, document source and adjustment method (lesson 4). A list of "today's index members" tested back ten years is survivorship bias.
- Clean and validate. Gaps, duplicates, zero prices, time zones; save to
data/clean/. - Features and signals with the tested functions from
src/; every feature at bar t uses data up to t only. - Backtest with the lesson 8 engine, including the cost model (Systematic Trading, lesson 8).
- Metrics and charts (lessons 9 and 10), on the in-sample period, with a small parameter grid, not a search for the best cell.
- Out-of-sample run once, with the parameters frozen. Record the result whatever it is.
- Report: hypothesis, data, rules, costs, both result tables, charts, number of variants tried, decision.
Code-review checklist
Review your notebook as if it were someone else's; each line is a yes-or-no question.
| Area | Check | Typical fix |
|---|---|---|
| Timing | Is every signal shifted by one bar before it sets a position? | pos = signal.shift(1) |
| Timing | Do rolling windows use center=False (the default) and no bfill()? |
Remove centring and backward fills |
| Timing | Are normalisations (z-scores, ranks, scalers) fitted only on past data? | Expanding or rolling fits |
| Data | Is the universe defined as it was known at each date? | Point-in-time constituent lists or an honest caveat |
| Data | Are adjusted prices used for returns and unadjusted for stop distances where relevant? | Keep both columns |
| Fills | Do gaps fill at the open, not at the stop or signal price? | min(open, stop) |
| Costs | Are commissions, spread and slippage applied per trade and per side? | Cost model in config |
| Overfitting | How many parameter sets, rules and universes were tried? Is the count in the report? | Log every run |
| Overfitting | Do neighbouring parameters give similar results (a plateau)? | Heatmap of the grid |
| Overfitting | Was out-of-sample data touched more than once? | If yes, it is now in-sample; find new data |
| Reproducibility | Does "Restart kernel and run all" reproduce the numbers in the report? | Fix hidden state and cell order |
Rule of thumb for significance: the t-statistic of an annualised Sharpe ratio is roughly Sharpe × √years. A Sharpe of 0.8 over four years gives 1.6, below the usual threshold of about 2, and that is before allowing for the number of variants tried; among twenty trials the best is expected to look good by chance.
Worked example
Hypothetical example, all numbers invented. A trader tests "buy a 100-day breakout, exit on a 3 × ATR trailing stop" on 30 large-cap shares, 2014–2025, with 0.1 % costs per side.
- Split. In-sample 2014–2021 (8 years), out-of-sample 2022–2025 (4 years), decided before the first run.
- Grid. Breakout lengths 50, 100, 150 × ATR multiples 2, 3, 4 = 9 variants, logged. In-sample Sharpe ranges from 0.6 to 1.1 with the best cell at (100, 3) = 1.1; its neighbours are 0.9 and 1.0, a plateau, so the choice is kept.
- Review. The checklist finds that the ATR column used
rolling(14).mean()on today's true range for today's stop. The engine is changed to use the previous bar's ATR (lesson 8). In-sample Sharpe falls from 1.1 to 1.0; maximum drawdown is unchanged at −22 %. - Out-of-sample. Run once: Sharpe 0.6, CAGR 7.2 %, maximum drawdown −19 %, 118 trades. Significance: 0.6 × √4 = 1.2; the in-sample figure gave 1.0 × √8 = 2.8. The out-of-sample result is positive but statistically weak on its own.
- Decision in the report. "The rule kept about 60 % of its in-sample Sharpe out-of-sample (0.6 / 1.0), on a plateau, 9 variants tried, one look-ahead bug fixed. Evidence is modest; next step is six months of paper trading with the lesson 12 script, not live capital." Fees modelled; taxes and financing excluded.
Common mistake
Practice activity
On paper: review this notebook fragment and the result that followed it.
df['z'] = (df.close - df.close.mean()) / df.close.std()df['signal'] = df.z < -1.5df['pos'] = df.signal.astype(int)df['ret'] = df.close.pct_change() * df.pos
In-sample Sharpe 2.4 over 6 years; out-of-sample Sharpe −0.3 over 2 years. (a) Identify two look-ahead problems in the code. (b) Compute the approximate t-statistics for the in-sample and out-of-sample Sharpe ratios. (c) Rewrite the four lines so that they are free of look-ahead (describe the changes if you prefer prose).
Show solution
(a) First, df.close.mean() and df.close.std() use the whole sample, including future prices, so the z-score at any date "knows" the future average. Second, pos is not shifted: today's return is earned on a signal computed from today's close.
(b) In-sample: 2.4 × √6 = 5.9; out-of-sample: −0.3 × √2 = −0.4. The large in-sample value was produced by the bugs, which is why it collapsed.
(c) Use a rolling or expanding window and a shift:
m = df.close.rolling(250).mean(); s = df.close.rolling(250).std()df['z'] = (df.close - m) / sdf['signal'] = df.z < -1.5df['pos'] = df.signal.shift(1).fillna(False).astype(int)df['ret'] = df.close.pct_change() * df.pos
Limitations: removing look-ahead makes the test honest, not profitable; the corrected rule still needs costs, an engine with stops and a fresh out-of-sample period.
Key takeaways
- Keep reusable code in
src/with tests, pin versions, and make "run all" reproduce the report. - Fix the hypothesis, rules and in-sample / out-of-sample split before the first backtest; run out-of-sample once.
- Review timing, data, fills, costs and trial counts with a checklist; log every variant you tried.
- Report significance roughly as Sharpe × √years, the number of variants and the decision, including "not enough evidence".
Next step
This completes Python for Traders & Investors; the next course, "Prop Firms, Funded Accounts & Trading as a Business", looks at evaluation rules, drawdown types and the practical and German tax side of trading as a business, where a tested, scripted process like the one you built here is the starting point.
Self-check
Answer the questions, then check your answers. You can retry as often as you like.
Automatic self-check for your own revision. Your answers are checked in your browser and are not seen or assessed by us. It is not an exam and not a qualification.
Your progress
“Complete” only means you marked the lesson as done. It is not a test of your understanding. Progress and notes are saved only in this browser on this device — not in an account.