PlumblineStage II:
Leave blank for automatic 95th-percentile calculation.
Parameter Sensitivity
A robust strategy performs well across a range of parameter values, not just one optimized set. If shifting a moving average from 20 to 22 periods destroys your edge, you don’t have an edge — you have a curve fit.
Test ±20% around every parameter. If performance collapses, the parameter is fragile.
Keep It Simple (Degrees of Freedom)
Every parameter you add is a degree of freedom the optimizer can exploit to fit noise. Rule of thumb: aim for at least 100–200 trades per free parameter.
A strategy with 5 parameters needs 500–1,000+ trades minimum. Fewer inputs = harder to overfit = more likely to survive live trading.
Sample Size & Statistical Power
Small samples lie. 50 trades can look incredible by pure luck. This dashboard’s p-value tests help, but they work best with 200+ trades.
If your strategy has fewer than 100 trades, treat ALL results with extreme skepticism — even passing scores.
Out-of-Sample Discipline
OOS data must be truly unseen — never peek, never “just check,” never iterate on it. The moment you adjust parameters based on OOS results, it becomes in-sample.
Use this tool’s Walk Forward Analysis to enforce honest OOS testing. One-shot validation is the gold standard.
Market Regime Diversity
Backtest across multiple market conditions: trending, ranging, high-volatility, low-volatility, and crisis periods.
A strategy that only works in 2020–2021 bull runs is not robust — it’s a regime artifact. If your data window doesn’t include at least one major drawdown period, your stress-test is incomplete.
Beware the Equity Curve Illusion
A smooth backtest equity curve is seductive but often a sign of overfitting. Real edges are messy — they have drawdowns, flat periods, and ugly months.
If your backtest looks too perfect, it probably is. The Monte Carlo section above stress-tests this by randomizing trade order.
Overfitting Warning Signs
- Win rate above 70% on intraday strategies
- Sharpe ratio above 3.0 in backtesting
- Performance collapses with ±10% parameter changes
- Strategy only works on one specific instrument / timeframe
- Fewer than 100 trades in the backtest
- WFA efficiency ratio below 30% (severe IS/OOS gap)
- Adding more indicators keeps “improving” results
Trade the Edge, Not the Curve
Your goal is NOT to maximize backtest profit. Your goal is to find the smallest, simplest edge that survives every stress test you throw at it.
A strategy that passes all five tests on this dashboard with modest returns is infinitely more valuable than one that shows 500% returns but fails statistical validation. Trade the edge, not the curve.