Quant Validation

Validating a financial model — the five ways the number lies

A backtest that looks excellent and loses money live is not usually a bad strategy. It is a good measurement of the wrong thing. Each section below is one mechanism by which a number becomes convincing without becoming true.

On sourcing. The methods here are standard and attributable — most of them to Marcos López de Prado's Advances in Financial Machine Learning, with the information-ratio framing from Grinold & Kahn. This file states the MECHANISM and what to check, and deliberately does not restate formulas from memory. Where an implementation needs an exact expression — the deflated Sharpe ratio in particular — verify it against the primary source before shipping a number that depends on it. A formula recalled approximately is worse here than no formula: it produces a specific, wrong, confident figure.

1. Purged cross-validation with an embargo

The leak. In a normal k-fold split, training and test rows are disjoint. In a financial series they are not independent: a label at time t is computed from data spanning t to t+h. A training observation inside that window has seen the future the test observation is being asked to predict.

Purging. Drop from the training set every observation whose label window overlaps the label window of any test observation. Not the observation's timestamp — its label window. This is the step people skip, because a plain timestamp split looks like it already separates them.

The embargo. Purging is not enough when features are serially correlated: a training row immediately AFTER the test set still carries information about it. Drop a further band after each test fold. The band is a fraction of the total sample; there is no universal value, so state the one used and why.

Combinatorial purged CV. A single train/test split yields one backtest path and one Sharpe. Splitting combinatorially yields many paths and therefore a distribution, which is what you actually want: a strategy whose single path looks good and whose distribution straddles zero has told you something a point estimate hid.

What to check: is the split purged, is there an embargo, is its size stated, and is the reported figure a distribution or a single draw.

2. Triple-barrier labelling

The problem with fixed-horizon returns. Labelling "the return over the next five days" assumes you would have held for five days. You would not: a stop-loss would have taken you out on day two. The model is trained on an outcome that could not have happened.

The method. Three barriers per observation — a profit-take level, a stop-loss level, and a time limit. The label is which barrier was touched first. Levels are usually set from a volatility estimate rather than fixed, because a 2% move means different things in different regimes.

What to check: are the barriers volatility-scaled, is the time limit stated, and does the label record which barrier ended the observation rather than only the sign.

3. Sample uniqueness under overlapping labels

The problem. Overlapping label windows mean two rows can describe largely the same outcome. Standard learning assumes independent draws; here they are not, so the effective sample is far smaller than the row count and every confidence interval computed from that count is too narrow.

Two responses: weight each observation by its average uniqueness (how much of its label window it does not share), or draw with a sequential bootstrap that prefers observations overlapping little with those already drawn.

What to check: is a uniqueness weighting or effective sample size reported. A row count offered as a sample size is a wrong number, not a rough one.