If you test 100 strategies with no edge at all, the best one will look good (a t of 2 or more) 9 times out of 10. A Strategy Analyzer optimization tests hundreds or thousands of combinations without you noticing.
The coin and the thousand people
Ask a thousand people to flip a coin ten times. Someone will get nine or ten heads. They don't have a gift: that's what happens when a thousand try. If you're only shown that person, it looks like magic. Strategies are the same: testing many things and keeping the one that did best is the most widespread mistake in the industry, and no backtest warns you how many tests you ran before.
What the t is, in one sentence
The t measures how many times your average profit exceeds its own noise: average profit per day ÷ standard deviation per day × √days. Below 2, a result is easily explained by chance. But that 2 only holds for a single idea decided before looking at the data.
What chance gives you when you test a lot
We generated strategies with no edge at all (random daily results with zero mean, 400 days each, about 19 months) and kept the best one, as anyone testing variants would:
| Tested | Best t (median) | Prob. of t > 2 | Prob. of t > 3 | Bar (5%) |
|---|---|---|---|---|
| 1 | 0.0 | 2% | 0% | 1.6 |
| 10 | 1.5 | 20% | 2% | 2.6 |
| 100 | 2.5 | 91% | 15% | 3.3 |
| 1,000 | 3.2 | 100% | 77% | 3.9 |
| 10,000 | 3.9 | 100% | 100% | 4.4 |
Testing 100 variants is very easy: 5 values of one parameter, 5 of another and 4 time windows already make 100. The last column is the t you should require to keep the chance of fooling yourself with a no-edge strategy at 5%. It's a conservative rule (it assumes variants are independent, and in practice they look alike), but it's a useful guide. There are finer tools for the same thing, like the deflated Sharpe ratio of Bailey and López de Prado (2014).
Calculator: can you believe your result?
Count every combination the optimizer tested (or you tested by hand) and enter the best one's t. It's calculated in your browser; nothing is sent.
I don't know my t: work it out from my daily results
t = average profit per day ÷ standard deviation per day × √days. Use days, not trades.
The best method: the same control with chance
Since our variants resemble each other, in the lab we prefer to measure directly what chance gives with the same machine: we repeat each search with the trade direction chosen at random and count how many rules "pass" anyway.
- One-day rule generator on micro Nasdaq (59,400 rules): 269 pass, and 25 with chance. There's something. But in the 2006-2019 Nasdaq test, 4 survive versus 1.
- 17 technical signals on micro Nasdaq (9,792 variants): 64 versus 11.
- Generator with a "don't look like anything known" filter (62,000 rules): 22 versus 15. Indistinguishable from chance.
- 4,000 random strategies passing fitting, validation and test: 9.8% do. Three periods are not enough.
The right question is never "how many pass?", but "how many more than chance?".
The tests you don't count
The number that matters isn't the one you wrote down, but all the ones you ran: the ones you eyeballed on the chart before writing the rule, the versions you deleted because they did badly, the time or stop changes "just to see"... and ideas taken from videos and forums, which are already the winners of many tests you don't know about. Their author can be honest and the result still inflated by selection.
What to do before paying for an evaluation
- Decide beforehand how many variants you'll test and write it down. If you test more later, add them.
- Test families, not points: check what share of neighboring combinations also make money. An isolated peak is easy to manufacture by chance; a wide plateau isn't.
- Keep data the optimizer doesn't see and use it only once.
- Discount the chosen one's result: if you picked it among many, its future will be worse than its past.
How it was calculated
Our own simulation with a fixed seed: random daily results with zero mean, 400 days per strategy, 300 repetitions per size (60 with 10,000). The bar is the value that the best of N no-edge strategies only exceeds 5% of the time; the calculator uses its formula with the normal distribution, which gives the same values. The lab counts come from our futures searches, with the random-direction control.
Simulated and hypothetical results, with the limitations that implies. Not investment advice. Firm rules change: check them on each firm's website. This article contains no affiliate links.