Quant Methods · advanced · 8 min

Multiple Testing

There is a way a backtest can be honest, well-run, statistically significant by the textbook, and still be worthless. It happens not because any single test was wrong, but because of how many tests were run before this one surfaced. This is the multiple-testing problem, and it is the reason Visor holds strategies to a bar that looks, at first, strangely high.

A Backtest Results panel showing a strategy that beat buy-and-hold on headline numbers but failed robustness testing

One test, and the illusion of rigour

Statistics has a standard tool for "is this average return distinguishable from zero, or just noise?" — the t-statistic. Roughly, it is the size of the average trade's return measured in units of its own noisiness. The textbook threshold is |t| > 2.0: clear that, and there is only about a 5% chance a result this strong came from pure luck. A one-in-twenty fluke. On a single, pre-committed test, that is a reasonable bar.

The trouble is that nobody runs a single test.

Twenty shots at a one-in-twenty fluke

Suppose a result has a 5% chance of clearing |t| > 2.0 by luck alone. Test one strategy and a false positive is unlikely. Test twenty strategies, and — even if every one of them is genuinely worthless — you should expect about one to clear the bar by chance. Test forty, expect two. The threshold that meant "one-in-twenty fluke" for a single test means "almost certain to be fooled at least once" across a search.

This is why the equity curves in strategy marketplaces look so good: you are shown the survivor of a search you never saw. The winner was not selected for being real. It was selected for having the best luck among many. The naive t-test cannot tell the difference between a real edge and the luckiest of forty coin-flippers — they produce the same number.

The corrected bar: |t| ≥ 3.5

Visor follows Harvey, Liu and Zhu (2016), who worked through exactly how much the significance bar has to rise once you account for the sheer number of strategies the finance industry has tried over the decades. Their answer, roughly, is that |t| ≥ 3.5 — not 2.0 — is the honest threshold for a "discovery" in trading strategy research. The higher bar is simply the price of having gone looking for a winner at all. A strategy can clear the naive 2.0 and still fail this one, and when that happens the report says so explicitly rather than quietly passing it.

The deflated Sharpe: same disease, the Sharpe ratio

The same logic attacks the Sharpe ratio. If you tried forty variants and kept the best Sharpe, that best figure is inflated purely by the search — the luckiest of forty skill-less attempts posts a flattering Sharpe with no skill behind it. The deflated Sharpe (Bailey and López de Prado, 2014) discounts the observed Sharpe by the Sharpe that the luckiest of N worthless attempts would be expected to post, plus a correction for skew and fat tails — because selling tail risk buys a flattering Sharpe, and this claws that back. What comes out the other side is a Sharpe that has already paid for the search.

The registry: making N honest

Both corrections need one number: N, how many variants were tried under the same idea. And N is exactly the number a person is tempted to forget. "I only ran the one that worked" is the most natural lie in backtesting, and it is usually told in good faith — the forty failures are quietly dropped from memory.

Visor's hypothesis registry fixes this by making you state the claim before the run, and then counting every backtest made under it. That count is the multiple-testing correction. The consequence is deliberately uncomfortable: the same backtest — identical trades, identical return — becomes less credible the more variants you tried to find it. Nothing else stops p-hacking, because the search is invisible unless it is recorded. Register no hypothesis and there is no admitted search and no discount — but then the honest reading is that the number is unproven, not that it is good.

Why this belongs in every verdict

Multiple testing is the quiet engine behind two of the four robustness gates: the |t| ≥ 3.5 significance bar and the deflated Sharpe. It is also the deep reason The Overfitting Trap is so easy to fall into — every knob you turn is another test, another shot at a false positive, and the maths counts them whether you do or not. The defence is not cleverness. It is discipline: decide the claim first, count every attempt, and read the corrected number, not the raw one.