Why Your Backtest Is Lying to You (and How to Spot It)
Beautiful backtests are easy to make and dangerous to trust. Three biases — overfitting, look-ahead, and survivorship — show up in nearly every amateur backtest. If you cannot articulate why your backtest is free of all three, assume it is not.
1. Overfitting: The 21.6-Period RSI Problem
If you keep tweaking parameters until the backtest looks great, you are no longer testing a strategy. You are decorating one. The market has a lot of noise, and noise contains patterns that look meaningful but are not.
A common smell test: change one parameter slightly and see whether the result collapses. If your strategy needs a 21.6-period RSI to work but breaks at 21 or 22, the strategy is fitting noise. Real edges are robust to small parameter changes.
2. Look-Ahead Bias: Using Tomorrow's Data Today
This bug is subtle and almost always accidental. The classic version: you compute a daily indicator using closing data, then use it to enter trades on the same day. By the time you have the close, the day is over. You used information that was not yet available.
Other versions: using a stock's later-known sector classification on earlier dates, computing a regression including out-of-sample data, or using forward-fill on missing data in a way that leaks the next valid value backward.
Look-ahead is the bias that makes a backtest look spectacular and live trading look broken. If your live results are dramatically worse than your backtest, this is the first place to check.
3. Survivorship Bias: Only the Winners Are Left
If you backtest "buy the dip on the S&P 500's largest 50 stocks" using today's largest 50 names, you are only testing the names that survived to today. The companies that went bankrupt or got acquired are gone from your dataset.
Result: your backtest looks amazing because every name in it is a winner by definition. Real-time, you would also be holding the names that became zeros.
Crypto has the same problem in compressed time. Backtesting on "the top 100 coins" today is wildly different from holding the top 100 in 2018.
The Smell Tests
- Does the equity curve look like a perfect 45-degree line? Suspicious.
- Does the strategy trade rarely but always profitably? Probably overfit.
- Does it depend on a specific year being included? Almost certainly not robust.
- Did you tune parameters using the same data you measured on? Yes, you overfit.
What Honest Backtests Look Like
Honest backtests have realistic costs deducted, use point-in-time data, separate parameter tuning from final measurement, and report drawdowns honestly. They also tend to look less impressive than the polished ones — and that is the point. The strategies that look modest in backtest are the ones that survive contact with the live market.