An end-to-end walkthrough of the six-Step validation pipeline — from vague trading idea to a statistically-vetted, portfolio-fit, prop-firm-ready strategy. Four of the six Steps run inside the Plumbline Trading Suite (its four software Stages, I–IV); the other two run inside NinjaTrader 8.
Six Steps. One Decision Gate. Four Plumbline software Stages. Every strategy that reaches your portfolio has survived statistical scrutiny.
| Step | Name | Software | Plumbline Stage |
|---|---|---|---|
| 1 | Build (Prompt Builder) | Plumbline Trading Suite | Stage I |
| 2 | Compile strategy code | NinjaTrader 8 (NinjaScript Editor) | — |
| 3 | Backtest strategy | NinjaTrader 8 (Strategy Analyzer) | — |
| 4 | Backtest Verification | Plumbline Trading Suite | Stage II |
| 5 | Portfolio Fit | Plumbline Trading Suite | Stage III |
| 6 | Prop Firm Verification | Plumbline Trading Suite | Stage IV |
Vocabulary: Steps (1–6) describe the workflow you follow. Stages (I–IV) describe the four Plumbline software modules that run inside four of those Steps. The Decision Gate between Step 4 and Step 5 is not a Step — it is the logical checkpoint you apply based on Stage II's pass/fail verdict.
No strategy enters your portfolio without passing statistical validation first. The Decision Gate after Step 4 (Backtest Verification) is the single most important checkpoint in this pipeline — it prevents curve-fit, fragile, or random strategies from contaminating your portfolio and, later, your funded account. Only validated, evidence-backed strategies proceed to Step 5 (Portfolio Fit) and Step 6 (Prop Firm Verification). Most strategies will not pass. That does not mean the software is broken. Most strategies simply do not have edge.
Specification, code compilation, and backtesting.
The Prompt Builder enforces NinjaScript development discipline by generating structured, scope-locked prompts for AI coding tools. Each prompt includes workflow presets, core rules, scale profile, domain modules, and explicit deliverables.
Key ActionsA complete, structured NinjaScript development prompt ready for AI tool input.
The AI tool receives the structured prompt and generates NinjaScript (C#) code. Paste it into the NT8 NinjaScript Editor, compile, and resolve errors. The structured prompt from Step 1 dramatically reduces compile failures, hallucinated methods, and scope drift.
Key ActionsA compiled, error-free NinjaScript strategy loaded in NinjaTrader 8.
NinjaTrader's built-in Strategy Analyzer runs the compiled strategy against historical market data. Use a minimum of 8 years of data to capture multiple market regimes. After the backtest completes, export the trade list as a CSV file for external validation.
Key ActionsA CSV file containing all trade data (entry/exit times, profit, MAE, MFE, instrument).
Statistical validation, portfolio analysis, and prop-firm rule simulation.
Backtest Verification performs rigorous statistical analysis on exported trade data to determine whether a strategy's edge is real or the result of randomness and curve-fitting. It applies five independent tests — positive expectancy, sign-flip permutation, one-sample t-test, Monte Carlo bootstrap, and anchored walk-forward analysis. Full statistical procedure, assumptions, and worked examples live on the Methodology page.
Key ActionsA comprehensive statistical validation report with pass/fail on all 5 tests.
The Decision Gate is not software and not a numbered Step. It is the logical checkpoint you apply after Step 4 based on Stage II's verdict — the single most important architectural decision in this pipeline. A strategy failing any of the five tests is rejected before it can enter your portfolio.
For diagnosis: low expectancy points to weak entry/exit logic; a high p-value means the edge may be random noise; high ruin probability indicates inadequate risk management; a failed walk-forward signals overfitting to in-sample data.
Portfolio Fit receives only validated, Stage II-passing strategies. It combines them into a portfolio view and evaluates three portfolio-context tests: cross-strategy correlation (Test 6), SPY regime alpha (Test 7), and portfolio vs. SPY buy-and-hold benchmark (Test 8). Stage III does not re-validate individual strategies; it assumes each has already passed Stage II. Full statistical procedure lives on the Methodology page.
Key ActionsPortfolio analysis: correlation matrix, regime breakdown, SPY benchmark comparison, and a plain-English verdict on whether the active portfolio adds risk-adjusted value over passive SPY.
Prop Firm Verification simulates a firm's evaluation rulebook against your strategy's historical trade log — daily loss limits, trailing drawdown, maximum static drawdown, minimum profit target, and consistency requirements. A strategy with a perfect Stage II report can still fail evaluation on a single unlucky Wednesday; Stage IV surfaces that risk before you fund an account. Full intended methodology lives on the Methodology page.
Key ActionsA prop-firm-specific report: pass probability, failure-mode breakdown by rule, expected days-to-pass conditional on passing, and a compatibility verdict per firm on your candidate list.
The exact thresholds that decide whether a strategy advances, iterates, or is discarded. The Decision Gate is not software and not a numbered Step. It is the logical checkpoint you apply after Step 4 based on Stage II's verdict.
| Validation Test | Pass Threshold | What It Measures |
|---|---|---|
| Positive Expectancy | > $0 per trade | Average profitability per trade (Curtis Faith formulation). |
| Sign-Flip Permutation p-value | < 0.05 | Edge is not due to random win/loss sign structure. |
| One-Sample t-Test p-value (one-tailed) | < 0.05 | Statistical significance of mean per-trade P&L. |
| Monte Carlo P(ruin) | < 5% | Likelihood of hitting the drawdown threshold under bootstrap resampling. |
| Anchored Walk-Forward | median efficiency ≥ 0.50 and ≥ 60% OOS windows profitable | Edge persists on unseen data across expanding folds. |
| Portfolio Metric | Reading | What It Measures |
|---|---|---|
| Cross-strategy Pearson ρ | |ρ| < 0.33 preferred | Diversification benefit vs. redundant exposure. |
| Regime Alpha coverage | Profitable in ≥ 66% of active buckets = Strong edge | Whether edge is regime-agnostic or regime-conditional. |
| Annualized alpha vs. SPY (OLS) | > +2% AND portfolio Sharpe > SPY Sharpe = Real edge | Risk-adjusted outperformance vs. passive buy-and-hold. |
| Beta vs. SPY | |β| < 0.3 market-neutral; 0.3–1.3 market-like; > 1.3 leveraged | Market exposure profile of the aggregate portfolio. |
| Up / Down Capture | Ideal: up high, down low or negative | Asymmetric participation in SPY rallies vs. drawdowns. |
| Simulated Metric | Pass Threshold (target) | What It Measures |
|---|---|---|
| P(pass) — Monte Carlo | > threshold (firm/user-defined) | Fraction of resampled paths reaching the profit target without rule violation. |
| P(fail by daily loss) | Report only | Fraction of paths violating the firm's max daily loss. |
| P(fail by total drawdown) | Report only | Fraction of paths violating static or trailing max drawdown. |
| P(fail by consistency rule) | Report only | Fraction of paths violating "no single day exceeds X% of profit"-style rules. |
| Expected days-to-pass | Report only (conditional on passing) | Typical time from account start to profit-target hit. |
The Decision Gate between Step 4 (Stage II) and Step 5 (Stage III) is the single most important architectural decision in this pipeline. A strategy that fails statistical validation has no business in your portfolio — adding it would contaminate correlation analysis, inflate the combined equity curve with unreliable returns, and mask real risk.
When a strategy fails, the specific failure tells you what to fix: low expectancy points to weak entry/exit logic; a high p-value means the edge may be random noise; high ruin probability indicates inadequate risk management; a failed walk-forward signals overfitting to in-sample data. Use the Prompt Builder's diagnostic mode to generate a targeted debugging prompt, then restart at Step 1. Each iteration produces a more robust candidate. The goal is a portfolio of independently validated, low-correlation strategies that survive every statistical test — and, at Step 6, a specific prop firm's rulebook.
One-line summary of every Step. Print this, tape it to your monitor.
| Step | Name | Software (Stage) | Key Action & Output |
|---|---|---|---|
| 1 | Build — Prompt Builder | Plumbline (Stage I) | Define requirements → generate structured NinjaScript prompt. |
| 2 | Compile Strategy Code | NinjaTrader 8 | Paste prompt → AI generates code → compile in NT8 editor. |
| 3 | Backtest Strategy | NinjaTrader 8 | Backtest 8+ years → Trades tab → Export CSV. |
| 4 | Backtest Verification | Plumbline (Stage II) | Drop CSV → check expectancy, p-values, Monte Carlo, WFA. |
| 5 | Portfolio Fit | Plumbline (Stage III) | Import validated CSVs → correlation, regime alpha, SPY benchmark. |
| 6 | Prop Firm Verification Planned | Plumbline (Stage IV) | Load ruleset → deterministic pass + Monte Carlo → P(pass), failure modes. |