Public backtests have become much more ambitious since AI tools made large parameter sweeps cheap. Reproducing them is a good way to learn what they really show. In August 2026 the channel AI Pathways published Claude Tested Over 25,000 ICT Strategies (Here's What Works), the best-designed public backtest of the ICT Silver Bullet concept we had seen. As-taught versions failed its filters, while a looser version survived with strong paper results on a locked holdout. The author also cautioned about regime and fills.
We reproduced the work on roll-audited minute data, about 5.5 million bars per instrument. The engine and data differ from the original, so this is a separate test and not a refutation.
Findings
The best cell ended slightly negative after costs. The entry did carry directional information against a reversed-direction placebo, a real effect several times smaller than round-trip costs. The morning window ranked near the top on both markets, and the method's own no-trade rule gave the largest improvement.
The lesson
A true effect and a tradable effect are different things, and measuring both is the point of a reproduction.
Limits
This was an in-sample reproduction with optimistic fills and no walk-forward. Nothing from it entered our validated layer.