Material: The hypothesess testing protocol, version 1 (hypothesess.com), written 2026-11-02 by claude-platform-q7c2
Material for the site's writer, not an article. The owner decided (2026-11-02) that hypothesess.com is a public lab for trading systems: every system we test gets a page on a register with a hypothesis written before the test, a fixed protocol, the result, and a forward record updated monthly. This is the protocol every register entry links to. Figures below marked "ours" come from our own published tests on this site; the rest are the method's rules with their sources.
Why a protocol (what went wrong without one)
Backtests published by vendors and forums report the best result of many tries with no count of the tries, no costs, one market, one period, and no record of what happened after publication. Our own earlier tests showed how large the luck term is:
- 4,096 moving-average strategies on 20 years of the S&P 500: none beat buy-and-hold after the number of tries is accounted for; the best in-sample strategy spent 36.8% of days in the market and switched 28 times (ours, /lab/4096-strategies) [1].
- Re-picking the best rule each year (walk-forward) turned $1 into $2.63 while holding turned it into $10.13 (ours, didithold.com, "Re-picking trading rules trailed holding") [2].
- A 55% win rate on 1:1 trades needs about 800 trades before a backtest can tell it from a coin flip (783 by the normal formula, 820 for exact 80% power); the best of 20 variants needs about 1,500 (ours, "How many trades does a backtest need") [3].
- The expected best in-sample Sharpe ratio of N skill-less strategies grows with N: with 5 years of data, about 45 independent configurations are enough to produce an in-sample Sharpe of 1 with an expected out-of-sample Sharpe of zero (Bailey, Borwein, Lopez de Prado and Zhu, 2014) [4].
The register: what every entry carries
- The claim, as people say it, and the rule it becomes (entry, exit, sizing, instrument, timeframe), written before any data is touched.
- The prior: what a skill-less rule with the same trade frequency would produce on the same data (the coin).
- The protocol fields: data source and period, costs (spread at the hourly 95th percentile from our broker's ticks, commission, swap), the number of parameter sets tried, the walk-forward windows, the correction for the number of comparisons.
- The result, with its uncertainty: profit factor and its 95% interval from the trade count, maximum drawdown, deflated Sharpe ratio, walk-forward efficiency, and the stage reached (below).
- The forward record: dated paper results after publication, added monthly, and the verdict as it stands: Held, Partly held, Didn't hold, or Too early. A verdict can change; the page shows its history.
- The code and data needed to rerun it, and the corrections line when anything above changes.
The ladder: just the necessary tests, in order, with kill rules
| Stage | What runs | Kill rule | Cost |
|---|---|---|---|
| 0. Claim | The rule written as code with every parameter named; the prior computed | No testable rule, or the claim is about a feeling, not a number: stop | an hour |
| 1. In-sample, one market, costs on | One market, 10 years, costs at the p95 spread; the count of parameter sets tried is recorded; deflated Sharpe computed against that count | Deflated Sharpe below the luck bar for the trials tried, or fewer than 100 trades: stop | an evening |
| 2. Walk-forward, several markets | Anchored windows (2 years train, 1 year test) across 5 markets; parameters re-picked on train only | Walk-forward efficiency under 50%, or one unseen year that loses more than the in-sample maximum drawdown: stop | a weekend |
| 3. Paper-forward | The rule runs on a demo account from the day of publication; monthly entries on the register | Three months below the prior's 25th percentile: Didn't hold | months, automatic |
| 4. Held | Twelve forward months at or above the prior's median | A later year that fails stage 2's rule returns it to Partly held | ongoing |
Why this order: stage 1 kills most systems for the price of one run; stage 2 is where overfit systems die; stage 3 is the only test that cannot be gamed, because the data did not exist when the rule was written. Only stage 3 survivors get an article that says "worked so far", and every article names its stage.
Numbers a reader can hold (ours and sourced)
- Luck bar at stage 1: for N independent tries the expected maximum Sharpe of skill-less strategies is about the (1 - 1/N) quantile of a normal distribution scaled by the annualisation; a table from N = 1 to 1,000 goes on the protocol page (Bailey et al. formula) [4].
- Trades needed: 783 (55% win rate, 1:1), 4,904 (52%), 194 (60%); the best of 20 variants at 55%: 1,491 (ours) [3].
- Costs: a system that earns 0.10% per trade loses its edge entirely at a round-trip cost of 0.10%; our costs-versus-edge piece gives the break-even cost for a 55% strategy (ours, "Costs versus edge") [5].
What the protocol does not do
It does not promise profit, it does not test execution (slippage beyond the measured spread, requotes), it uses one broker's history, and a system can pass all four stages and fail later; that is what the forward record is for. We publish the kills as well as the survivors, because a list with only survivors is the survivorship bias the protocol exists to remove.
Sources
- hypothesess.com, "4,096 trading strategies, 20 years of the S&P 500: none beat luck", lab page https://hypothesess.com/lab/4096-strategies (ours, 2026-10).
- didithold.com, "Re-picking trading rules trailed holding; $1 to $2.63 vs $10.13", https://didithold.com/ (ours, 2026-10).
- hypothesess.com, "How many trades does a backtest need before it means anything?", https://hypothesess.com/blog/how-many-trades-does-a-backtest-need-before-it-means-anything (ours, 2026-10-30).
- D. H. Bailey, J. M. Borwein, M. Lopez de Prado, Q. J. Zhu (2014), "Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance", Notices of the AMS 61(5), 458-471. https://www.ams.org/journals/notices/201405/rnoti-p458.pdf (authors' copy: https://www.davidhbailey.com/dhbpapers/backtest-pseudo.pdf)
- hypothesess.com, "Costs versus edge: how much of a 55% strategy survives the spread", https://hypothesess.com/ (ours, 2026-10).
Writer: the three "ours" figures must be quoted exactly as above; do not add trading advice; the piece is the rulebook, in the site's plain voice, with the ladder table kept whole.