How many trades does a backtest need before it means anything?

Determining the necessary number of trades for a backtest is crucial for validating a trading strategy. The sample size directly influences the reliability of the results. By our calculation, a strategy that wins 55% of its trades, with wins and losses the same size (1:1), needs about 800 trades to be told apart from luck: 783 by the usual formula, 820 for the exact test to reach 80% power [1]. Conversely, if the win rate is lower, such as 52%, the number of required trades escalates significantly to around 4,900. These figures highlight the importance of adequate sample sizes in trading strategy evaluations.

Understanding Win Rates and Sample Size

Sample size is essential in assessing the efficacy of a trading strategy. The key metric here is the win rate, which is the percentage of trades that are profitable. A higher win rate typically requires fewer trades to establish statistical significance.

For instance, if a trader tests a strategy with a win rate of 55%, they will need about 800 trades for the results to be meaningful. This is our own power calculation: a two-sided test at 5% significance (a strategy with no edge passes only 1 time in 20) and 80% power (a real 55% edge is detected 4 times in 5). The usual formula gives 783 trades; because wins come in whole numbers, the exact binomial test gives only 78.85% power at 783 and needs 820 trades to reach 80% [1]. The figures assume wins and losses of the same size; with other payoffs, see the question on trade sizes below.

Here’s a look at the trades needed for various win rates:

This table shows how the number of trades required increases dramatically as the win rate decreases. All of these numbers assume independent trades: trades that overlap in time or share a market regime count as fewer, so treat the table as a floor [1]. For example, a strategy that achieves just a 52% win rate requires significantly more trades—about 4,900—to establish statistical significance.

Impact of Variants on Sample Size

When testing multiple variants of a trading strategy, the required sample size can increase significantly based on the number of variants being evaluated. Suppose you are a trader testing 20 different parameter sets and find one with a win rate of 55%. The number of trades needed to confidently validate that finding rises to approximately 1,491. This adjustment addresses the multiple-testing problem, where the likelihood of mistakenly identifying a successful strategy increases as more tests are performed: each variant is tested at 5% divided by the number tried (a Bonferroni correction) [1]. For similar variants, such as neighbouring points of one parameter grid, this is conservative, and the true increase is somewhat lower.

Here’s a breakdown of how many trades are necessary depending on the number of variants tried:

This increase in sample size underscores the necessity for stronger evidence to substantiate a strategy's effectiveness when evaluating multiple options. The calculations are derived from our own methodology, reflecting how luck and noise can influence outcomes. Consequently, as you assess more variants, be prepared to invest significantly more trades to ensure that your findings are reliable [1].

A worked example

Suppose your backtest shows a 55% win rate, with wins and losses the same size. How much that tells you depends on how many trades it rests on. By our calculation (exact 95% confidence interval) [1]:

At 100 trades the interval still reaches down to 44.7%, so the backtest cannot rule out a losing strategy, and about 1 in 5 strategies with no edge at all would show 55% or better by luck. At 300 trades the interval still includes 50%. Only at around 1,000 trades does it sit clearly above a coin flip [1].

To put this into perspective, if you execute an average of 10 trades a day, reaching that target of 800 trades will take you about 80 trading days. If your win rate is lower, say 52%, the required sample size increases significantly, necessitating nearly 4,900 trades. This increase could extend your backtesting period substantially, potentially taking over 490 trading days at the same rate.

This example underscores the importance of sample size in backtesting; the more trades you execute, the more confident you can be in your strategy's effectiveness.

Questions readers ask

Why is 800 trades the magic number for a 55% win rate?

The figure of 800 trades comes from statistical calculations designed to ensure that you can confidently identify a strategy's true effectiveness with two guards: a strategy with no edge passes only 5% of the time (the significance level), and a strategy that truly wins 55% is detected 80% of the time (the power). The usual formula gives 783 trades and the exact test 820, so about 800 [1]. Essentially, it’s like having a large enough sample of coin flips: if you flip a coin only a few times, you might think it's biased based on a lucky streak. In this case, the 800 trades act as a safeguard against such randomness [1].

What happens if I have fewer trades than recommended?

If you have fewer than the recommended number of trades, your results might not be reliable. Instead of claiming your strategy is effective, it's more prudent to state that its effectiveness is "not yet known." Think of it like testing a new recipe: if you've only tried it once, you can't be sure if it will always turn out well.

What if my wins and losses are not the same size?

Measure the edge instead as the average return per trade divided by the standard deviation of returns per trade. For 80% power at two-sided 5%, an edge of 0.10 needs 785 trades, 0.05 needs 3,140 and 0.02 needs 19,623 [1]. A 55% win rate with 1:1 payoffs is an edge of about 0.10, which is why it lands near the same 783. Counted in years of daily returns, a strategy with an annual Sharpe ratio of 1.0 needs 3.84 years of data just to be significant and 7.85 years for 80% power [1].

Sources

1. hypothesess: Ideas, put to the test