Launch memo
- paper trading
- the first three years live
held-back test
paper trading
three years live
All three figures are Sharpe ratios: yearly return divided by yearly bumpiness.
The Incubator
Do it the careful way, and let luck through anyway.
Careful quants don't trust a backtest. They hold back data for a final test, then trade new strategies with paper money for a year before launch. Run that whole process here on 1,000 strategies. One of them has a real edge.
0strategies scored
0looks at the held-back data
0passed the final test
- score on ten years minus the held-back part
- top 50, kept
- passed the held-back test
- failed
- not checked yet
- one line per strategy
- the one launched
Why the careful process still fools you
Each gate is a good idea on its own. The trouble is what happens between them.
The held-back test gets used up
Holding back data works once. When too few strategies pass, the team tweaks them and checks the same held-back data again. Every look leaks a little of it into the search. This run looked times before it had enough winners. A strategy with no edge passes a single check about of the time.
Paper trading is too short to tell
A year of results is very noisy. A strategy with a real Sharpe ratio of 1 needs about years of results before a standard test says it is not luck. A Sharpe ratio of 0.5, more typical of real strategies, needs years.
Picking the best incubated strategy is the old problem again
Incubate several and launch the best, and you have run a new search, only slower. The best of many ghosts looks good by luck alone. Judging the best of them fairly takes even longer.
Years of results before the best of the strategies you incubated clears a 5 percent test, allowing for how many you incubated. This is the Bonferroni rule, which is mild.
The Sharpe ratio the luckiest of your incubated strategies should show when none of them has any edge, by years of paper trading.
You never hear about the ones that died
Fund companies seed many funds in private and only open the winners to the public. The record you see is the one that survived.
This happens with real money
Evans looked at mutual funds started this way. While incubated, they beat comparable funds by 3.5 percent a year. Once they opened to the public, the extra return was gone. Vanguard checked 637 indexes that ETFs were built on: 87 percent beat the market before the fund launched, 51 percent after. Banks' own sold strategies showed a median Sharpe ratio of 1.20 in the backtest and 0.31 live.
Before you trust a track record, ask: how many did you incubate, and how often did you look at the test data?
Sources
- Evans, "Mutual Fund Incubation", Journal of Finance 65(4) (2010). Incubated funds beat by 3.5 percent a year while incubated; the edge is gone after they open.
- Vanguard, "Joined at the hip: ETF and index development". 637 indexes; 87 percent beat the market before launch, 51 percent after.
- Suhonen, Lennkh and Perez, "Quantifying Backtest Overfitting in Alternative Beta Strategies", Journal of Portfolio Management (2017). 215 bank strategies; median Sharpe ratio 1.20 in backtest, 0.31 live.
- Wiecki, Campbell, Lent and Stauth, "All that glitters is not gold", Quantopian (2016). 888 algorithms; the backtest Sharpe ratio explains less than 2.5 percent of the live one.
- Bailey, Borwein, López de Prado and Zhu, "Pseudo-Mathematics and Financial Charlatanism", Notices of the AMS (2014). A held-back test ignores how many tries were made.
- Dwork and colleagues, "The reusable holdout", Science 349 (2015). Re-using held-back data to choose what to try next makes it unreliable.
- Lo, "The Statistics of Sharpe Ratios", Financial Analysts Journal (2002). How noisy a measured Sharpe ratio is, used here for the years needed.
- McLean and Pontiff, "Does Academic Research Destroy Stock Return Predictability?", Journal of Finance (2016). Returns 26 percent lower out of sample, 58 percent lower after publication.