Before paying for an evaluation with a strategy, ask it these 12 questions. Three are essential: real costs, enough trades and a forward test. If it fails any of those three, no past result is worth what it seems.
The idea behind it: the past gets used up
Every time you look at some data and change something, that data stops being useful to judge the result. That's why the method is about keeping data you haven't looked at and always measuring against what chance would give. It's what we did with thousands of strategies in our lab; these questions are the summary.
The 12 questions
Mark your answers. They are not saved or sent anywhere.
- 1Did you write down the rules and the success criterion before looking at the data?
If you decide after looking, any result will look like confirmation.
If not: Write them down now and test them on data you haven't seen.
- 2Did you split the data into fitting, validation and test, and look at them in that order?
Passing three periods is not enough on its own: of 4,000 random strategies, 9.8% passed them.
If not: What you have is a design, not a test.
- 3Do you have a test on different years that is used only once?
Markets change regime. A test on other years, looked at only once per setup, is the closest thing to the future.
If not: Extend the test forward before risking anything.
- 4Does it include commission and slippage, on entry and exit? key
On micro Nasdaq they are about 4.4 ticks per trade: with small targets they take more than half.
If not: Subtract the costs: many strategies stop making money here.
- 5Did you also test it with no costs, as a diagnosis?
If it doesn't make money without costs either, there's nothing to optimize; if it makes a lot without costs and nothing with them, it trades too much.
If not: Do it before touching a single parameter.
- 6Are your bars at least as fine as your trading?
If the target and the stop fit inside one bar, the backtest doesn't know which was hit first.
If not: Repeat with finer bars or with ticks.
- 7Are there hundreds of trades and a t-stat of 2 or more, computed by day? key
With a $25 edge and a $249 standard deviation per trade you need about 380 trades to tell it apart from chance.
If not: Edge not proven: the pass rate any simulator gives you will be optimistic.
- 8Does it clearly beat a control group that knows nothing?
On 2023-2024 micro Nasdaq, buying blind made +$4,164. A long-only strategy has to beat that.
If not: Its edge is something else: the market rising, the time of day…
- 9Do neighboring settings also make money, in both halves of the data?
The correlation between results in fitting and in the test was -0.47: what looked best in the past tended to do worse. A solid strategy made money in 100% and 96% of its variants in each half; another, in 24% and 73%.
If not: It's a peak: don't trade it. Pick the center of a plateau.
- 10Do you size with a Monte Carlo by day and at today's prices?
Your backtest's worst drawdown is just one of the possible ones. And prop firm rules are in fixed dollars while the Nasdaq has multiplied: a 2019 backtest with 1 contract underestimates today's risk.
If not: Your real risk is larger than you think: reduce size.
- 11Does your platform take exactly the same trades as your simulation?
When we checked, they matched on 99-100% of trades. If the platform's backtest looks much better than your simulation, it's most likely broken.
If not: Fix the execution before going further.
- 12Have you tested it forward, in real time, with simulated money? key
It's the only data nobody could have looked at before, not even you.
If not: It's not ready for real money or for paying for an evaluation yet.
Why so much rigor
Because almost everything that looks like an edge is something else: the luck of having tried many variants, a rising market, a backtest bug or a regime that has passed. In our lab, most strategies that shone on the data they were designed with didn't survive these questions. The few that did made money in every period, in different years and against chance.
Where the figures come from
From the lab tests: random strategies run through the same filters, buying blind on 2023-2024 micro Nasdaq, consistency of variants across halves of the data and comparing backtests with the platform. Simulated and hypothetical results.
Simulated and hypothetical results, with the limitations that implies. Not investment advice. Firm rules change: check them on each firm's website. This article contains no affiliate links.