What is pre-registration in a trading backtest?

Writing down what would count as success — the metrics, the significance threshold, the baselines, the split — before you run anything and before any result exists. It matters because the usual way a backtest becomes wrong is not bad code, it is choosing the pass mark after seeing the outcome. Our bar required net return above zero, a t-statistic of at least 3.0 on the validation block, a Bonferroni-corrected p-value, beating three baselines, the same sign at both horizons, and surviving a single holdout read at t of at least 2.0.

Why read the holdout only once?

Because a holdout you can re-read is not a holdout. Every additional look lets information from that block leak into your choices, even when nothing is consciously changed, and the second read is worth far less than the first. Under this protocol a candidate that fails there is reported as failed and does not get one more variant. Ours is now spent; rebuilding a block of the same size takes roughly four months of new data.

Why disqualify a candidate that passed the test?

Ranking names by trailing turnover cleared the bar on net return, but our cost model derives the cost of a trade from trailing dollar volume — so the feature being ranked and the cost being charged were the same variable. Its top decile paid 0.31–0.41% a round trip against 0.600% for its bottom, and that fee gap was most of the apparent edge. Before costs it never cleared the bar. A signal that only works after costs is sorting by the cost model, not predicting returns.

Isn't adding criteria mid-study moving the goalposts?

It is, if the change rescues a result you liked. The test is which direction it cuts. The two criteria we added — clear the bar on gross as well as net, and disqualify features collinear with the cost model by construction — removed the only candidate that had passed and left the study with a null. They were also recorded, with reasons, before the holdout was read rather than applied quietly afterwards.

Pre-Registering a Crypto Backtest

Any strategy can be made to look profitable on history. The trick is not clever code, it is choosing the pass mark after seeing the result — nudging a threshold, dropping an awkward period, trying one more variant. It rarely feels like cheating while you do it. This codebase has shipped an overfit signal before: a long/short alignment gate that looked good in-sample, inverted going forward, and served the losing cohort until its hit rate fell. So before running our most recent study we wrote down what would count as success, and published the result whichever way it landed. It landed on nothing. This guide is the protocol and every number it produced.

Expectancy calculatorWork out the pass mark your own backtest has to clear before you run it, not after.

The bar has to exist before the result does

Pre-registration is standard in clinical research and almost unheard of in trading content, and the reason it works is unglamorous: it removes your own judgement from the moment it is least trustworthy. Once you have seen that a factor returns +0.49% at t=3.42, every choice about how to test it is contaminated by wanting it to be true. Writing the bar first does not make you smarter. It makes the result mean something.

What we committed to, before running anything

Six conditions, all of which had to hold. The significance threshold is deliberately stricter than this repo's standing t-statistic of 1.5, because that number is calibrated for testing one hypothesis and this study tested many — at 1.5 across fourteen candidates you expect roughly one false positive by chance alone. A candidate clearing every line would not have gone live on faith either; it would have become a governor-gated proposal that keeps re-validating forward and switches itself off when it stops beating baselines. That control is exactly what the old alignment gate lacked.

PRE-REGISTERED ACCEPTANCE BAR

  mean net return per trade   > 0
  t-statistic on validate     >= 3.0
  p-value                     < 0.05 / K   (Bonferroni, K = features tested)
  baselines                   beats all three
  horizons                    same sign at 16h and 24h
  holdout                     survives at t >= 2.0

The holdout is read once, and that is the whole point

The panel was split by time, decided in advance: the first 60% to look at freely and form hypotheses, the next 20% to test whatever survived, and the final 20% sealed. Read once. If a candidate fails there it is reported as failed — it does not get one more variant. A holdout you can re-read is not a holdout, it is a slower way of fitting to the same data, and the second read is worth far less than the first even when nothing is consciously changed.

block      share   use
explore    first 60%   look freely, form hypotheses
validate   next 20%    test candidates that survived explore
holdout    final 20%   untouched until one single final run

Panel: 250 USDT pairs, hourly, 2022-08-29 -> 2026-08-23
Universe median 80 names per bar
630 explore rebalances, 210 validate
Median cost charged: 0.600% per round trip

Beating zero is not the bar

A long-only crypto signal in a rising market beats zero by accident, which is how most published backtests get their curve. So a candidate had to beat three controls: equal-weighting the entire eligible universe over the same horizon, random selection of the same number of names, and simply holding BTC over the same window. We learned to insist on that last one the hard way — an earlier geometry sweep had every configuration looking profitable until a hold control was added, at which point none of them were.

Why our own live record could not answer the question

The obvious move is to mine the prediction journal, and it does not work. The journal held 405 resolved picks over four days, 116 of them out of sample. That is enough to show something gross — the momentum ranker scoring -3.60% per trade at t=-4.95, last of nine and worse than not ranking at all — and nowhere near enough to discover a signal. At n=116 a real effect is buried and a spurious one surfaces easily: testing fourteen candidates at a t of 1.5 produced two apparent winners, and both dissolved on inspection. One was an artifact of dividing by the stop distance; the other was not significant on realised return.

The one candidate that passed, and why we threw it out

Ranking names by trailing turnover cleared the bar on both blocks. Then we decomposed it. Its top decile pays 0.31–0.41% a round trip while its bottom pays 0.600%, so most of the apparent edge was the fee gap between the two legs rather than any difference in returns. And the reason is structural: our cost model derives the cost of a trade from trailing dollar volume, so ranking on turnover is ranking on the cost function. The feature and the cost were the same variable. Before costs it never cleared the bar at all — t=+1.16 on explore and t=+2.10 on validate against a bar of 3.0.

log_dollar_vol (high) — rank by trailing turnover

block      gross%    t       net%     t       fee gap
explore    +0.115   +1.16   +0.407   +4.11   +0.292
validate   +0.298   +2.10   +0.486   +3.42   +0.189

The net column clears the bar. The gross column never does.
The difference between them IS the result.

We made the bar stricter after seeing a result

That is normally the cardinal sin, so the change is recorded rather than quietly applied. Two criteria were added before the holdout was read: a candidate must clear the bar on gross spread as well as net, because a signal that only works after costs is sorting by the cost model; and any feature collinear with the cost model by construction is disqualified as a ranking candidate. The test for whether this is goalpost-moving is simple — did the change rescue a favoured result or kill it? It removed the only candidate that had passed and left the study with a null. We also declined to spend the holdout on it, because burning a one-shot resource on a known artifact buys nothing.

The prediction we wrote down before reading the holdout

Two funding-based candidates survived to the final read. Before taking it we recorded what we expected and why: explore was flat at +0.038 with a t of 0.36 across 630 rebalances, while validate was strong at +0.763 with a t of 3.48 across 210. That is a twentyfold jump in effect size between adjacent periods, with the flat one carrying three times the sample. A factor with a genuine +0.76% spread should have left a trace across 630 explore rebalances and did not. So the note read: I expect the holdout to fail. Recording it first is what stops the outcome being rationalised whichever way it lands. We spent the holdout anyway, because declining on a hunch would be overriding the pre-registration with exactly the intuition it exists to restrain.

The holdout result

Both failed. Neither cleared the t of 2.0, and excess return over simply equal-weighting the universe went negative in the holdout for both — so the long leg did not beat holding the universe at all. The holdout landed where an unstable factor lands: positive, about half the validate magnitude, not significant. The prediction was right, and for the stated reason. That block is now spent. Under this pre-registration it is read once and it has been, so no further candidate can be tested against it; rebuilding one of the same size takes roughly four months of new data.

candidate           explore          validate         HOLDOUT           bar
funding_3d (high)   +0.038 (t 0.36)  +0.776 (t 3.54)  +0.499 (t 1.52)   t >= 2.0
funding_7d (high)   +0.013 (t 0.12)  +0.880 (t 3.77)  +0.535 (t 1.65)   t >= 2.0

Excess over equal-weight in the holdout: -0.021 and -0.019

What a null still establishes

Three arms, eleven OHLCV features at four horizons plus funding at two, and no signal that predicts cross-sectional forward returns net of costs. That is a real result, and it came with three findings worth more than another candidate. Cost is the binding constraint: 0.600% a round trip against a median 16-hour move of 2.25% is 27% of a typical move, spent before any view is expressed. Long-only was negative in every block at every horizon tested — the universe itself lost money net of costs over this window, so anything usable would have to be a market-neutral spread rather than a long book. And low volatility is the only recurring thread, agreeing in sign across this study, the live journal and the factor table, while never once reaching significance. That is a hypothesis deserving more data, not a signal worth shipping.

Summary

We wrote the pass mark down first, tested three arms against it, disqualified the only candidate that passed because it was sorting on our own cost model, predicted the holdout would fail and said why, then spent the holdout once and watched both survivors fail at t=1.52 and t=1.65. No signal was found. The protocol is the transferable part: pre-register the bar, control against holding the thing you are trying to beat, insist a result survives before costs as well as after, and read the holdout exactly once. Most of what it will tell you is that you have found nothing, which is the answer that keeps you from trading noise.

How to Pre-Register a Backtest (And What Ours Found)