A free, open-source toolkit that catches overfit backtests before they cost you real money — with Monte Carlo, walk-forward, and permutation testing.
Each stage solves a specific problem in going from trading idea to deployed algo. Together they form the pipeline: spec → validate → portfolio-fit → prop firm.
Turn your trading idea into a written, testable specification.
Prove the results aren't overfit before you risk real money.
See how your new strategy interacts with your existing ones. Correlation, drawdown stacking, and allocation review across multiple systems.
Simulate whether your strategy would pass a Topstep, Apex, or Lucid evaluation.
A 6-step pipeline from strategy idea to validated portfolio, with pass/fail gates at every stage. The full guide walks through each stage, the exact thresholds, and the decision logic.
Most retail backtests are overfit. A strategy that looks profitable across years of historical data often performs that way for one reason: it was tuned, by hand or by optimizer, to match the noise in the sample. When deployed forward, the edge disappears.
The statistical methods used by institutional quants to detect this — permutation testing, Monte Carlo resampling, walk-forward validation — are well-documented in the academic literature but rarely applied in retail workflows. The math is straightforward. The tooling was missing.
Plumbline Trading Suite is an attempt to close that gap. It is not a trading platform. It is not a signal service. It is a set of small, focused tools that answer one question: given the trades I observed, how confident should I be that the underlying strategy has a real edge?
Plumbline Stage II: Backtest Verification applies three independent statistical tests. Passing one is not enough. A strategy that fails any of them is flagged for further investigation.
Randomly flips the sign of each trade's P&L thousands of times to estimate the distribution of outcomes under a null hypothesis of no edge. If the observed result is not significantly above the null distribution, the apparent edge may be noise.
Resamples the trade sequence with replacement to estimate the range of plausible equity curves and worst-case drawdowns. Reveals sequence risk and stress-tests the strategy against orderings that did not appear in the original backtest.
Splits the data into rolling in-sample and out-of-sample windows. A strategy that performs well in-sample but poorly out-of-sample is almost always overfit. The efficiency ratio between the two windows is the headline metric.
A strategy with strong-looking surface metrics that fails statistical validation. The kind of result that is easy to miss when only watching win rate and profit factor.
Clarity matters more than marketing. These tools are useful for one specific thing. They are not a substitute for the other parts of a trading practice.
Honest accounting of the current state. The shipped tools are usable today. Everything else is work in progress, no guarantees on timing.