Strategy Lab Audit Tutorial

How we stopped building strategies that do not work

A practical walkthrough of the checks that turned a spectacular backtest into a credible, reproducible research result.

What you will learn

Detect look-ahead bias

Verify that every decision uses only information available at that timestamp.

Model execution honestly

Separate signal generation from the price at which a trade can actually execute.

Include trading friction

Apply fees and slippage on every turnover event, not only on the final result.

Challenge statistical luck

Use holdout data and permutation tests before accepting an apparent edge.

The backtest that looked too good

The first version appeared exceptional. That was the warning, not the victory. We froze development and audited the data path, timing assumptions, costs and statistical validation before changing another parameter.

Reported return
+620%
Sharpe ratio
4.8
Maximum drawdown
−8%
Win rate
78%

Five audit findings

Each finding changed the result and became a permanent audit control.

1

The biggest bug: same-bar execution

The signal used the closing price and the trade also executed at that close. The model was therefore trading on information it could not yet possess.

Correction: shift the position by one bar, or execute at the next bar's open using a clearly documented convention.
2

Zero trading costs

Frequent rebalancing looked free. For a high-turnover strategy, modest fees and slippage can consume most of the apparent edge.

Correction: charge explicit fees and slippage whenever the absolute position change is non-zero.
3

Indicator warm-up leakage

Rows with incomplete rolling indicators were allowed into evaluation. That made early observations inconsistent with the fully formed model.

Correction: calculate indicators, remove incomplete warm-up rows, then begin both signal generation and performance measurement.
4

Parameter selection on the full history

The same period was used to choose parameters and to advertise performance. This turns the test set into training data.

Correction: define train, validation and untouched test periods before optimization, and report the holdout result separately.
5

No test against chance

A positive backtest alone does not show that the signal contains information. Random alignment can produce impressive equity curves.

Correction: compare the observed statistic with a distribution produced by shuffled or otherwise null signals.

Corrected results

Less spectacular, but now useful for making decisions.

Audit passed
Net return
+84%
Sharpe ratio
1.3
Maximum drawdown
−24%
Win rate
54%

A ten-second timing test

The fastest first check is to compare a position with its lagged version. If performance collapses only when the signal is shifted, inspect the execution timing before doing any more research.

# Signal known after the close
position = signal.shift(1)
turnover = position.diff().abs().fillna(0)
strategy_return = position * asset_return
net_return = strategy_return - turnover * cost_rate

Reusable strategy audit protocol

  1. Step 1

    Trace every timestamp

    Document when inputs become available, when the signal is known and when an order can execute.

  2. Step 2

    Reconcile returns manually

    Check a small sample row by row, including position changes, fees, slippage and compounding.

  3. Step 3

    Protect the holdout

    Lock the test period before tuning and do not repeatedly optimize against its result.

  4. Step 4

    Stress the conclusion

    Vary costs, execution delay and nearby parameters, then compare performance with a null distribution.