The best of many: why your best backtest is mostly the number of tries
hypothesess.com, AI-assisted writing ยท
The best of many. Short answer: the more strategies you try, the better your best backtest looks, even when none of them works. Test 100 strategies that are secretly coin flips on two years of data and the winner will usually show a Sharpe ratio of about 1.7 and a gain of roughly two thirds [1]. Then it meets new data and goes back to zero, because it never had an edge. This notebook entry runs the experiment in numbers.
The experiment: strategies with no edge
Suppose each strategy trades once a day for two years, 500 trading days, gaining or losing 1% at random. None has any edge. Keep the one with the best backtest and let it trade one more year, 250 days, of new data.
With one strategy there is no selection, and what you see is what luck gives a single try: a Sharpe ratio near zero, as often negative as positive, and a yearly return that wanders about 16 percentage points either side of nothing [1]. With 100 strategies, the best backtest has a Sharpe ratio of about 1.7 in a typical batch, and the luckiest batch in ten reaches 2.2 [1]. Every one of the hundred was a coin.
On the new data, the chosen strategy's year is a fresh coin flip. Its typical gain is zero, and its spread is the same 16 points any single strategy had [1]. The backtest told you which coin had been lucky. It could not tell you which coin would be.
How good does luck get?
Here is the same question for every batch size: the Sharpe ratio of the best backtest when nothing works. A Sharpe above 1 is what most people would call a real strategy.
| Strategies tested | Typical best backtest Sharpe | Luckiest batch in ten | Where it comes from |
|---|---|---|---|
| 1 | 0.0 | 0.9 | best of k draws from the no-edge Sharpe distribution over two years [1] |
| 10 | 1.1 | 1.6 | the same [1] |
| 20 | 1.3 | 1.8 | the same [1] |
| 100 | 1.7 | 2.2 | the same [1] |
| 300 | 2.0 | 2.4 | the same [1] |
| 500 | 2.1 | 2.5 | the same [1] |
With ten tries, luck clears a Sharpe of 1 more often than not: 56% of the time [1]. With a hundred, it almost always does (99.97% of the time), and one batch in five produces a backtest Sharpe above 2 [1]. These are the numbers a strategy has to beat before its backtest means anything, and the bar rises with every variation you tried.
Nerd corner
Each strategy's backtest Sharpe ratio is noise around zero with a spread of about 1 / √(years) per year, so about 0.7 over two years [2]. The best of k such draws sits roughly √(2 ln k) spreads above zero: it grows with every strategy you add, slowly but surely [2]. For k = 100 that formula gives 2.15 against an exact median of 1.74; it overshoots at small k, and the table above uses the exact quantiles of the maximum of k normal draws, Φ(x / s)^k = q, instead.
Defences, from cheapest: count every strategy and variation you tried and say the number; keep a stretch of data you never look at until the end, the out-of-sample test; and judge the winner against the luck line above, which is what the deflated Sharpe ratio of Bailey and López de Prado does formally [2].
Check yourself
You tweak one strategy's settings 50 times and keep the best version. How many strategies did you test? Fifty. Each setting is a separate try, and the luck line counts every one; the best of 50 coins shows a two-year Sharpe of about 1.6 [1].
A friend shows you a backtest with a Sharpe of 2 and says it is the only one they tried. Another shows 2 as the best of 300. Which is more convincing? The first, if it is true. The best of 300 coin flips reaches a Sharpe of about 2 by luck alone [1].
What does the best strategy usually earn on new data in this experiment? About nothing. It had no edge, so its future is another coin flip, with the same 16-point spread as any single strategy [1].
An illustration, not trading advice. A lab notebook entry for hypothesess.com, written with AI and checked by hand; all strategies here are coin flips, the table comes from the exact distribution of the best of k no-edge backtests, and the box above applies the rough rule to your own numbers.