Using Visor · intermediate · 8 min

How to Read a Robustness Report

A backtest return is the number everyone reads, and it is the least reliable thing about a strategy. Visor runs every backtest through four robustness gates and reports an honest verdict: pass, fail, or not proven. This lesson explains what each gate asks. Once you can read it, the ⚡ verdicts throughout the Strategies track make sense — and so does why so many famous strategies fail.

A Backtest Results panel flagging that the strategy did not survive robustness testing, above its stats and equity curve

The whole idea rests on one question. Not "did this make money?" — in a market that went up, almost anything long made money. The real question is "did this strategy's decisions beat luck?" Each gate is a different way of asking that.

Gate 1 — The random control

This is the heart of it. The engine takes your exact strategy and re-runs it 500 times with its entry timing scrambled — the same number of trades, the same exit rule, the same market, but the alignment between the signal and the price destroyed. Then it asks: where does your real result land among those 500 randomly-timed versions of itself?

To pass, your real return has to beat 95% of them. If a scrambled version of your strategy does about as well as the real one, then your entry timing wasn't doing measurable work — the return was a by-product of being in the market at all, not of the signal being smart.

This is the gate that catches the long-only bull-market trap. A strategy that is simply long during an uptrend makes money by being long, not by being clever, and randomly-timed entries make money too. Beating zero is trivial; beating a scrambled version of yourself is not.

Gate 2 — Out-of-sample

The engine splits the timeline by time — the first 70% as "in-sample", the last 30% held out — and checks whether the edge survives on the data it wasn't measured on. A strategy that is profitable in-sample and loses money out-of-sample shows the classic curve-fitting signature: it was tuned (knowingly or not) to the history it was tested on, and falls apart on anything new.

This gate needs enough trades on each side of the split to mean anything. With too few, it honestly returns not proven rather than guessing.

Gate 3 — Significance

Was the average trade's return distinguishable from zero, or just noisy luck? The engine computes a t-statistic and holds it to a demanding bar: |t| ≥ 3.5, not the textbook 2.0. Why the higher bar? Because when you try many strategies and keep the winner, the naive |t| > 2 threshold produces false positives constantly — this is the Harvey-Liu-Zhu correction, and the higher bar is the price of having gone looking for a winner at all. A strategy can clear the naive bar and still fail this one; the report says so explicitly when that happens.

Gate 4 — Deflated Sharpe

The Sharpe ratio measures return per unit of risk. But if you tried forty variants of an idea and kept the best Sharpe, that best number is inflated by the search itself — the luckiest of forty coin-flippers looks skilled. The deflated Sharpe discounts the observed Sharpe for how many variants were tried under the same hypothesis. If you register no hypothesis, there is no admitted search and no discount — but the honest way to use the tool is to state the claim before you go hunting.

The verdict

The gates don't average. A strategy is only as credible as its weakest gate, so any outright failure fails the whole run — a strong random-control score can't launder an out-of-sample collapse. Not proven means the sample was too small to judge some gate, which is itself useful information: it means the headline numbers are unproven, not that they're good.

Why this is the honest way round

Most places that teach a strategy sell you the version where it works. Visor teaches the strategy and then shows you, with the same engine every time, whether it holds up — and frequently it does not. That is not a bug in the strategy or the tool. It is what honest testing looks like. A verdict that often says fail is the opposite of a performance promise, which is exactly why it can be trusted.