How we test a trading system before we say it works
Daniel Okafor, Strategy analyst (AI-assisted) ยท
In trading, the difference between success and failure often lies in how a system is tested before being declared effective. Many traders rely on backtests that can mislead due to luck or overfitting. To combat this, a systematic approach called the Hypothesess Testing Protocol was developed. This process confirms whether a trading rule holds up across various stages of testing, allowing only the most robust systems to persist.
The Need for a Protocol
Backtesting can produce enticing results, often showcasing the best performance of a trading strategy without accounting for the number of trials, costs, or future performance. For instance, a study of 4,096 moving-average strategies applied to 20 years of the S&P 500 revealed that none of these strategies consistently outperformed a simple buy-and-hold strategy when considering the number of trials conducted [1]. This highlights the importance of a rigorous protocol to filter out unreliable systems.
A well-structured protocol not only demands a clear hypothesis but also implements systematic testing across multiple stages, which have defined criteria for a strategy to advance or be terminated. For example, at the first stage, a strategy must exceed a statistical threshold based on its past performance, known as the luck bar. This ensures that only those systems demonstrating genuine skill progress, while others that may have succeeded purely by chance are discarded early. The protocol's design is crucial for preserving the integrity of the testing process and ensuring that the results are reflective of real potential rather than mere happenstance.
The Testing Protocol Stages
The Hypothesess Testing Protocol consists of five stages, each designed to test a trading system progressively. Here’s a breakdown of each stage:
| Stage | What Runs | Kill Rule | Cost |
|---|---|---|---|
| 0. Claim | The rule is coded with parameters defined. | If there's no testable rule, stop. | an hour |
| 1. In-sample | One market over 10 years, costs accounted for. | Stop if the Sharpe ratio falls below the luck threshold or fewer than 100 trades occur. | an evening |
| 2. Walk-forward | Multiple markets with anchored training windows. | Stop if walk-forward efficiency is under 50% or a year of unseen data underperforms. | a weekend |
| 3. Paper-forward | Demo account results after publication. | Stop if three months are below the 25th percentile of previous performance. | months, automatic |
| 4. Held | Results over twelve months at or above median prior results. | Return to Partly Held if a later year underperforms. | ongoing |
This structured process allows for an efficient allocation of resources, with the first stage filtering out the majority of weak systems for the price of one run.
Why Early Stages Are Critical
The first two stages of the testing protocol are essential for filtering out trading systems that struggle under real-world scrutiny. In stage one, a critical factor is the "luck bar," which serves as a statistical threshold. This threshold is the Sharpe ratio the best of N skill-less strategies would be expected to reach for the N parameter sets actually tried; a system has to clear it before anyone speaks of skill [1]. For instance, to achieve statistical significance, a strategy with a 55% win rate requires approximately 783 trades [1]. Without reaching this threshold, the strategy's reliability remains questionable.
Moving to the second stage, walk-forward testing evaluates a strategy’s performance across various market conditions. This phase is vital as it typically uncovers instances of overfitting, where a strategy appears robust in backtests but falters when applied to new, unseen data. The walk-forward approach is akin to a rigorous fitness test, ensuring that a strategy can adapt and maintain its effectiveness outside of a controlled environment. This method's importance cannot be overstated, as it distinguishes between systems that are merely lucky and those that demonstrate genuine skill.
The Importance of Forward Testing
Only systems that survive the first two stages undergo paper-forward testing, which records real-time performance in a demo environment. This step is crucial because it reflects the actual potential of the system in the market without retroactive adjustments. The forward record is updated monthly, providing a transparent history of performance.
The forward testing stage is where you can truly gauge a strategy's robustness. Imagine this as training for an athlete who has passed initial trials but must now compete in live events. Just as the athlete's performance in real competitions reveals their true capabilities, forward testing exposes any weaknesses in the trading strategy that might not have been apparent during earlier evaluations.
The purpose of this ongoing tracking is to ensure that even strategies that appeared successful in controlled conditions can adapt to the dynamic nature of market conditions. The necessity of this forward record highlights that validation doesn't end after the initial tests; it requires vigilance over time to confirm efficacy and reliability in practice.
Limitations of the Protocol
While the Hypothesess Testing Protocol is robust, it does not guarantee profit. It does not test execution aspects like slippage or requotes, relying instead on a single broker's historical data. Additionally, a system can pass all stages and still fail later, underscoring the need for ongoing performance monitoring. The protocol is designed to combat survivorship bias by publishing both successful strategies and those that failed, offering a broader view of system viability.
A Worked Example
To illustrate the protocol’s application, consider a hypothetical trading strategy that claims to profit from price movements based on a specific indicator. The claim must be clearly defined before any data is analyzed. For instance, suppose the strategy states it will buy when an indicator crosses above a certain threshold and sell when it crosses below.
- Stage 0: If the rule cannot be expressed quantitatively, it’s discarded.
- Stage 1: Suppose ten years in-sample, costs on, give a Sharpe ratio of 0.5 after 40 parameter sets were tried; the best of 40 skill-less tries is expected to score higher than that, so the luck bar is not cleared and it stops [1].
- Stage 2: If it had progressed, it would be tested across five markets, requiring at least a 50% walk-forward efficiency to continue.
- Stage 3 and 4: If it passed, it would enter a demo account for real-time monitoring for at least three months.
This step-by-step approach illustrates how rigorous testing can filter out unreliable strategies before they reach the market.
Questions Readers Ask
What is the luck bar, and how does it affect testing?
The luck bar is a threshold that determines if a trading strategy's performance is likely due to skill or mere chance. If a strategy's performance falls below this bar in the initial testing stage, it is discarded.
How can I track the performance of a system after testing?
Once a system passes all necessary stages, it should be continuously monitored through a forward record, which logs performance outcomes monthly. This helps ensure that it continues to meet or exceed the thresholds set in earlier tests.