V I S O R

Strategies · intermediate · 7 min

The Overfitting Trap

Everything else in this track shows strategies that fail honest testing. This lesson is about the more dangerous case: strategies that pass a naive test spectacularly — because they were, knowingly or not, fitted to the very data they were tested on. This is overfitting, and it is the single most common way a backtest lies.

A Backtest Results panel warning that a good-looking strategy did not survive robustness testing — overfitting caught in the act

How it happens

You have a rule with a few knobs — a moving-average length, an oversold threshold, a stop distance. You try 20/50; it's okay. You try 18/44; better. You nudge the threshold; better still. After an afternoon of tweaking you have a version with a beautiful equity curve and a huge return.

What you have actually done is search the history for the settings that happened to fit its particular wiggles. The market's past has a specific, never-to-repeat sequence of noise in it, and with enough knobs you can always find a rule that carved that exact noise perfectly. It tells you nothing about the next stretch of market, because the next stretch has different noise.

The signature

Overfitting has a fingerprint, and Visor's out-of-sample gate is built to catch it. The engine splits the timeline — the first 70% to measure on, the last 30% held out — and compares. The classic overfit signature is:

When a run shows that pattern, the report calls it out by name: profitable in-sample only — the curve-fitting signature. A strategy tuned to its test data will show exactly this, because the thing it learned — the noise — isn't there in the held-out period.

Why the deflated Sharpe matters here too

Out-of-sample catches overfitting after the fact. The deflated Sharpe attacks the cause: the search itself. If you tried forty variants and kept the best, that best result is inflated simply because you went looking — the luckiest of forty coin-flippers looks like a genius. The deflated Sharpe discounts a strategy's Sharpe for how many variants were tried under the same hypothesis, which is why Visor asks you to register a hypothesis before you go hunting. Stating the claim up front makes the size of the search visible, and the size of the search is exactly what has to be discounted.

The honest habit

The defence against overfitting isn't a clever trick; it's a discipline:

  1. Decide the rule before you look at the outcome. Register the hypothesis first.
  2. Never judge on in-sample alone. The held-out period is the only opinion that counts.
  3. Count your attempts. Ten variants of an idea is ten shots at a false positive, and the maths knows it even if you'd rather forget.
  4. Read the whole verdict, not the return. A great in-sample number with a failing out-of-sample gate is not a good strategy — it is a warning.

The reason Visor bakes an honest verdict onto every strategy lesson is precisely this trap. It is very easy to produce a backtest that looks like a licence to print money, and the golden cross and trend-following examples show how ordinary rules dissolve under scrutiny. Overfitting is how the impressive ones dissolve — usually right after you start trading them.

What to read next