How Plumbline validates a strategy.

Test 1 — Positive Expectancy

What it asks

Before any statistics, the strategy must clear the most basic hurdle: does its expected value per trade come out positive under the Curtis Faith formulation? A strategy that fails this test has no edge to validate.

Governing formula

The engine also reports expectancy per dollar risked as E / avg_loss, which normalizes across position sizes and lets you compare strategies with different average loss magnitudes.

Pass/fail threshold

Pass when E > 0. This is a hard gate. A strategy with non-positive expectancy will not be advanced to the remaining four tests in any meaningful sense — those tests still run, but the overall verdict is already a fail.

Reported metrics

• expectancy_per_trade — the value of E above, in dollars
• expectancy_per_dollar — E divided by avg_loss
• win_rate, loss_rate — fraction of trades in each category
• avg_win, avg_loss — mean of winners and mean absolute value of losers
• win_loss_ratio — avg_win / avg_loss
• total_trades — count of non-zero trades

Assumptions and failure modes

• Assumes each trade P&L is realized net of costs. If the trade log excludes commissions and slippage, positive expectancy on this test can vanish live.
• Not sample-size sensitive. A strategy can produce positive E on 20 lucky trades. Tests 2–5 exist to guard against that.
• Zero-P&L trades excluded. Break-even trades do not contribute to E. If your strategy has many exact break-evens, the reported win_rate and loss_rate reflect only non-zero trades.

Test 2 — Sign-Flip Permutation

What it asks

If the win/loss pattern were random — every trade a fair coin flip with the observed magnitudes — how often would a random assignment produce a net P&L at least as high as what actually occurred?

Null hypothesis

Procedure

Test statistic: net P&L — the sum of realized trade P&L. Iteration count: 10,000 permutations. Random number source: numpy-compatible PCG64 with seed 42, ensuring reproducibility across runs and across the Python and JavaScript implementations.

1. Take the observed trade P&L series x = (x₁, ..., x_n).
2. Compute the observed test statistic T_obs = Σ x_i.
3. For each of N = 10,000 iterations:
   1. Independently draw s_i ∈ {−1, +1} via rng.integers(0, 2), each with probability 0.5.
   2. Compute the permuted statistic T_k = Σ s_i · |x_i|.
4. The one-sided p-value is:

Pass/fail threshold

Pass when p < 0.05. This is a strict binary gate in the engine (the significant_sign_flip_005 flag). There is no warn tier — the engine reports the raw p-value alongside a boolean pass/fail.

Worked example

x = [+120, -80, +45, +200, -60, +90, -110, +75, +150, -50]
T_obs = 380
|x| = [120, 80, 45, 200, 60, 90, 110, 75, 150, 50]

Enumerating all 2¹⁰ = 1024 sign assignments exactly: 139 produce a signed sum ≥ 380. Exact p-value = 139 / 1024 ≈ 0.136. With p above the 0.05 threshold, this trade log fails the sign-flip test at n = 10. The engine would run 10,000 sampled permutations instead of enumerating exhaustively, but at n = 10 the sampled p-value converges to the exact one within ~0.005.

Assumptions and failure modes

• Assumes trade independence. Overlapping positions or highly correlated signal firings inflate variance the null does not model. P-value comes back optimistically low, letting fragile strategies pass.
• Assumes magnitudes are exogenous. Signs permute; the |x_i| sequence is preserved. If your risk management systematically sizes losers larger than winners, that asymmetry bleeds into every permutation and can distort the p-value in either direction.
• Not robust to survivorship. A pre-filtered trade log (deleted "unrealistic" trades, cherry-picked periods) produces a p-value that means nothing.

Test 3 — One-Sample t-Test

What it asks

Sign-flip is a non-parametric test on the sign structure. The one-sample t-test is a parametric test on the mean. Both interrogate "does this strategy have a real edge?" but they attack the question from different angles and can disagree in informative ways. Requiring both to pass tightens the gate.

Null hypothesis

Test statistic

Procedure

1. Compute the two-tailed p-value from the Student-t CDF: p_two = 2 · (1 − F_t,df(|t|)).
2. Convert to a one-tailed p-value against H_a: μ > 0:
   • If t > 0: p_one = p_two / 2
   • If t ≤ 0: p_one = 1 − p_two / 2

The JavaScript implementation computes F_t,df via the regularized incomplete beta function (Lanczos log-Gamma for stability, Lentz continued fraction for the beta CDF), which matches scipy.stats.ttest_1samp to double-precision.

Pass/fail threshold

Pass when p_one < 0.05.

Edge cases

• If n < 2: return t = 0, p = 1.0 (automatic fail).
• If the sample standard error is zero (all trades identical): return t = 0, p = 1.0 (automatic fail).

Why both this and sign-flip?

A strategy with one huge outlier winner and many small losers can have very positive mean P&L (t-test says pass) but very few actual winning trades (sign-flip says fail, because random sign assignment easily produces the same sum from the outlier alone). Conversely, a strategy with a strong win rate but tiny average edge can pass sign-flip and fail the t-test. Requiring both catches both pathologies.

Assumptions and failure modes

• Assumes i.i.d. trades with finite variance. Autocorrelated trade series inflate false-positive rates. The t-test does not know your strategy overlaps positions.
• Not robust to fat tails. The t-statistic is anchored on a normality assumption that breaks with heavy-tailed P&L distributions. In practice this makes the test conservative (harder to pass) for fat-tailed strategies, which is not the worst failure mode.
• Single-outlier sensitivity. One massive winner or loser can dominate both &xmacr; and s. If your backtest contains a single anomalous trade, the t-result is driven by that one row.

Test 4 — Monte Carlo Bootstrap

What it asks

The observed backtest is one specific ordering of one specific sample. If the same trades had occurred in a different sequence — or with slight variations in composition — what would the distribution of terminal equities, max drawdowns, and Sharpe ratios look like? And how deep is the drawdown that materializes in unlucky-but-not-unrealistic paths?

Procedure

Iteration count: 10,000. Resampling scheme: i.i.d. bootstrap with replacement — not block bootstrap. Random source: PCG64, seed 42.

1. Take the observed trade P&L series x = (x₁, ..., x_n).
2. Compute the observed equity curve, terminal net profit, max drawdown, and Sharpe ratio (mean / population std, i.e. ddof = 0). These are the "observed" reference values.
3. For each of N = 10,000 iterations:
   1. Draw n samples with replacement from x via rng.integers(0, n) → x*^(k).
   2. Compute the resampled equity curve: E_t^(k) = Σ_i≤t x*_i^(k).
   3. Record: terminal equity E_n^(k), max drawdown DD^(k), Sharpe μ^(k) / σ^(k).
4. After all N iterations, compute the ruin threshold (see below) and the P(ruin) statistic.

The ruin threshold: auto vs. user-supplied

This is the most important design choice in Test 4 and the one most likely to be misunderstood.

If the user supplies a threshold (e.g., "−$5,000, the max drawdown my prop firm allows"), R is fixed to that value and P(ruin) becomes an absolute measure. The engine reports the ruin_source field as either "auto_95pct" or "user" so the interpretation is unambiguous downstream.

Reported statistics

• net_profit_median, _mean, _5pct, _95pct — central tendency and 90% CI of terminal equity
• max_drawdown_95pct — 95th percentile of simulated max drawdowns
• prob_of_ruin — the P(ruin) statistic above
• ruin_threshold, ruin_source — the R value used, and whether it was auto-derived or user-set
• observed_net_profit, observed_max_drawdown, observed_sharpe — the reference values from the un-resampled backtest
• sharpe_median — median Sharpe across simulated paths

Pass/fail threshold

Pass when prob_of_ruin < 0.05. Under the auto ruin threshold, this is a test of drawdown-distribution shape: a strategy passes if the tail of the drawdown distribution is short (few paths hit the 95th-percentile ceiling repeatedly). Under a user-supplied threshold, this is a test of absolute risk exposure against your operational limit.

Assumptions and failure modes

• i.i.d. resampling breaks temporal clustering. Real strategies cluster wins and losses (trending vs. ranging regimes). Standard bootstrap destroys those clusters, which usually makes the simulated max drawdown distribution optimistic relative to what a live account would experience. The engine does not use block bootstrap, so this understatement is real for autocorrelated strategies.
• Auto ruin threshold can hide risk. Under the auto default, P(ruin) is roughly 5% by construction. A strategy with catastrophically deep tails can still pass the P(ruin) gate because R scales up with the tail. Users trading under a hard drawdown limit (funded accounts, personal max-loss rules) should always pass a user-supplied R.
• Sharpe uses population std (ddof = 0). This matches numpy's default but differs from the sample-std Sharpe some traders expect. Comparisons across tools require matching conventions.
• No path-dependent trade sizing. If your strategy uses Kelly, martingale, or any equity-dependent sizing, the resampled paths do not respect that logic — they assume the same trade magnitudes regardless of prior equity.

Test 5 — Anchored Walk-Forward Analysis

What it asks

Split the trade history into a sequence of in-sample and out-of-sample windows. Compare per-trade expectancy between the two. Does the strategy's edge persist as the sample rolls forward? Do enough out-of-sample windows come out profitable to trust the pattern?

Anchored vs. sliding

Configuration

• WFA_N_WINDOWS = 5 — target number of OOS windows
• WFA_MIN_TRADES = 50 — strategies with fewer trades skip the test entirely and return pass_wfa: false
• fold_size = floor(n / (n_windows + 1)) — if this drops below 10, the engine falls back to fold_size = max(10, floor(n / 3)) and adjusts n_windows accordingly
• WFA_EFF_CLIP_LO = −2.0, WFA_EFF_CLIP_HI = 3.0 — per-window efficiency is clipped to this range before aggregation, to prevent a single degenerate window from dominating the median

Procedure

1. Reject the test if n < 50 (auto-fail with a "skipped: insufficient trades" flag).
2. Compute fold_size as above.
3. For window k = 0, 1, ..., n_windows − 1:
   1. IS slice: trades[0 : (k+1) · fold_size] — expanding from the anchor
   2. OOS slice: trades[(k+1) · fold_size : (k+2) · fold_size] — the next non-overlapping chunk
   3. Compute per-trade expectancy on each slice (win_rate · avg_win − loss_rate · avg_loss).
   4. Per-window efficiency: e_k = OOS_expectancy / IS_expectancy, clipped to [−2, 3].
4. Aggregate across all windows.

The two headline metrics

Pass/fail thresholds — both gates required

Overall pass_wfa = true requires both gates. Median efficiency alone can be gamed by one very strong OOS window; the profitability gate exists to require breadth as well as depth.

Worked example

A strategy with 300 trades, 5 windows, fold_size = floor(300 / 6) = 50:

Window 0: IS trades 0-49    (E_IS = $18.20)   OOS trades 50-99    (E_OOS = $9.10)   eff = 0.50
Window 1: IS trades 0-99    (E_IS = $15.40)   OOS trades 100-149  (E_OOS = $8.50)   eff = 0.55
Window 2: IS trades 0-149   (E_IS = $13.10)   OOS trades 150-199  (E_OOS = $6.80)   eff = 0.52
Window 3: IS trades 0-199   (E_IS = $11.90)   OOS trades 200-249  (E_OOS = -$2.10)  eff = -0.18
Window 4: IS trades 0-249   (E_IS = $10.20)   OOS trades 250-299  (E_OOS = $7.60)   eff = 0.75
median_efficiency = median([0.50, 0.55, 0.52, -0.18, 0.75]) = 0.52  → passes 0.50 gate
pct_oos_profitable = 4 / 5 = 0.80                                    → passes 0.60 gate
pass_wfa = TRUE

Note: window 3 lost money OOS but did not fail the overall test because the other four windows carried the median and the profitability gate. This is the intended behavior — a single unlucky window should not veto an otherwise robust strategy, but a pattern of unlucky windows will.

Assumptions and failure modes

• Correlated folds. Because the IS window expands, later folds' IS statistics share most of the trades that appeared in earlier folds. Per-window efficiencies are not statistically independent, so aggregate statistics (median, mean) have an inflated effective sample size. The reported thresholds are calibrated as descriptive gates, not as inputs to formal statistical inference across folds.
• Unequal statistical power across folds. Fold 0's IS is estimated from a small sample and is noisy; fold K−1's IS is estimated from most of the history and is tight. The efficiency ratio's denominator has different variance across folds. Clipping to [−2, 3] limits the damage but does not eliminate it.
• Minimum 50 trades is a floor, not a target. At exactly 50 trades with 5 windows, fold_size drops to 8, hits the <10 fallback, and reduces to 3 windows of 16 trades each. Small folds produce noisy per-window expectancies. Treat WFA output as directional below ~150 trades.
• Regime changes look like overfit. A strategy that legitimately works in trending markets but fails in ranging markets will fail WFA if OOS windows happen to land in the wrong regime. The test cannot distinguish overfit from regime-dependence; both produce the same signature.
• Expectancy, not Sharpe, is the per-window statistic. The test asks whether the edge size persists, not whether the risk-adjusted edge persists. A strategy that keeps positive expectancy but with growing per-trade variance can pass WFA and still be dangerous to trade.

Test 6 — Cross-Strategy Correlation

What it asks

Do these strategies' returns move together? Two strategies with correlated returns provide less diversification than uncorrelated ones, regardless of how individually strong each is.

Procedure

1. For each pair of strategies (A, B), align their trade sequences on a common time index. Plumbline aligns by daily bucket keyed on trade exit date (UTC, YYYY-MM-DD). All trades that closed on the same calendar day are summed into a single per-strategy P&L observation for that day.
2. Compute per-period P&L series r_A,t and r_B,t over the union of all dates that appear in any enabled strategy's curve. Missing periods (a strategy took no trades on that day) are zero-filled. Zero-fill is defensible for a mechanical trade log — no trade means no realized P&L — but it does drag the correlation estimate toward zero when trade frequencies differ substantially between strategies, and users should read the metric with that in mind.
3. Compute Pearson correlation:

Interpretation bands

The engine does not emit a Pass/Warn/Fail verdict for correlation. It renders a heat-shaded matrix using the bands below, plus a per-strategy diversification-benefit bar. These are visual guides, not summary tests.

Positive and negative correlations receive symmetric bands (brass for positive, red for negative). The diversification-benefit bar is a rough summary computed as max(0, (1 − max(0, mean off-diagonal ρ)) · 100%), not a formal test.

Assumptions and failure modes

• Pearson assumes linear relationships. Two strategies with a non-linear relationship (one profits from volatility spikes, one from vol compression) can show low Pearson while being deeply linked.
• Correlation is not causation of joint drawdown. Two zero-correlation strategies can still draw down at the same time by coincidence. Pearson ρ alone will not surface that risk.
• Sample size matters. ρ estimated from < 30 aligned periods has wide confidence intervals.

Test 7 — SPY Regime Analysis

What it asks

Is the strategy's edge conditional on a specific market regime, or does it persist across regimes? A strategy with strong overall statistics can still be concentrated in one type of market — only bullish, only volatile, only trending — and that concentration is a portfolio-fit signal Stage II cannot see.

Classifier design

Plumbline classifies every month in the historical SPY window on three independent axes. The classifier is a fixed rules-based system, not a model fit. Classifier version 1.0.0 ships with the bundle; percentile-based cuts are re-computed at build time from the SPY sample and pinned into the shipped data.

Realized volatility is annualized as stdev(daily_returns) · √252 over a trailing 21-trading-day window ending on the month's last trading day. Efficiency ratio (Kaufman) is |monthly_return| / Σ|daily_returns| — a value near 1.0 indicates a straight-line month, values near 0 indicate directionless chop.

Trade tagging

Every trade is tagged to a regime by its exit date, sliced to a UTC year-month key. This choice is deliberate and matches Stage II's convention: P&L is realized at exit, so the regime in which the exit occurred is the regime that produced the P&L — even if the trade opened in a different regime.

Per-bucket statistics

For each (strategy, axis, bucket) triple with at least one trade, the engine reports:

• count — number of trades tagged to this bucket
• win_rate — fraction of trades with P&L > 0
• pf — profit factor (gross win / gross loss), clamped to 99.99 when there are no losing trades, with a no_losses flag set
• expectancy — mean P&L per trade in the bucket
• grossWin, grossLoss, netProfit — dollar totals for the bucket

The headline statistic: Regime Alpha

Regime Alpha is reported per strategy per bucket and is the primary signal for identifying regime dependence. A strategy with a large positive Reg-α in one bucket and a large negative Reg-α in the complementary bucket has a regime-conditional edge. A strategy with Reg-α near zero across all buckets has a regime-agnostic edge.

Verdict bands

The engine does not emit a binary Pass/Fail on regime data. It renders a verdict panel and a heat-shaded per-strategy breakdown. Practical reading:

Assumptions and failure modes

• SPY as a market proxy. The classifier uses SPY. For futures strategies (ES/NQ/CL/GC), SPY is a reasonable proxy for the equity-vol regime but not for the trading instrument's own regime. A gold-only strategy whose edge depends on precious-metals volatility will be misclassified.
• Price-only SPY. The shipped data is Stooq price-only. Dividend drag of ~1.5%/yr is documented in the data meta but not added back. Total-return SPY is deferred to classifier v2. Directional buckets are affected marginally; volatility and trendiness are not.
• Percentile-based cuts are sample-dependent. Low/Normal/High volatility bands are defined relative to the SPY sample in the shipped bundle. If that sample changes (rebuild against a longer window), the thresholds move. The classifier_version and window dates are surfaced in every report so a Stage III output can be pinned to a known reference frame.
• Monthly granularity. Regimes are monthly. A strategy that switches between trending and choppy conditions within a single month gets assigned to whichever the month's aggregate looked like, and its per-bucket statistics blur across the two.
• PF clamp asymmetry. Buckets with zero losing trades report pf = 99.99 with the no_losses flag set. Regime Alpha computed against a clamped PF is not a meaningful comparison; the flag is surfaced in the UI for this reason.
• Reg-α is descriptive, not inferential. No p-value is attached to Regime Alpha. Small buckets can produce large Reg-α values by chance. The trade count is displayed alongside every bucket so users can weight Reg-α by sample size.

Test 8 — Portfolio vs SPY Benchmark

What it asks

Would you have been better off just buying and holding SPY? A validated multi-strategy portfolio should outperform passive buy-and-hold on absolute return, on risk-adjusted return, or both. If it does neither, the portfolio's active work is not being compensated.

Setup

• SPY daily close series is bundled offline via data/spy_daily.js. No runtime fetch.
• SPY is normalized to the portfolio's first trade date, not SPY's first date. Both curves start at the same initial capital on the same date. This keeps the comparison fair even when the portfolio started years after the shipped SPY window began.
• Trading days per year: 252.
• Time-in-Market is measured on exit-day basis: a SPY trading day counts as "in market" if the portfolio had at least one trade exit on that day.

Alpha and Beta (OLS)

Alpha and Beta are estimated by ordinary least squares regression of portfolio daily returns on SPY daily returns over the aligned window:

Up and Down Capture

The 9-metric panel

The engine reports each metric for the portfolio, for SPY buy-and-hold, and the delta:

Beta interpretation bands

Verdict logic

The engine collapses the panel to one of five plain-English verdicts based on annualized alpha and the Sharpe delta against SPY:

• Real edge confirmed — α_ann > +2% AND portfolio Sharpe > SPY Sharpe.
• Outperformed at higher risk — beat SPY on total return but not on Sharpe.
• Lost on return, won risk-adjusted — underperformed SPY on total return but higher Sharpe. Often indicates deliberately low Time-in-Market.
• Marginal — positive but small alpha; edge may not be statistically distinguishable from noise over the window.
• SPY buy-and-hold beat the portfolio — on both raw and risk-adjusted return. Question whether the active portfolio adds value beyond passive exposure.

Assumptions and failure modes

• Price-only SPY. Dividend drag of ~1.5%/yr is not added back. The comparison is therefore slightly conservative against SPY on total return — a total-return SPY would beat price-only SPY by roughly that amount per year, and reported alpha is inflated by the same amount.
• OLS assumes stationarity. A portfolio whose strategy mix changed mid-window will produce noisy alpha and beta, because the regression is estimated on a mixture of behaviors.
• Beta is estimated on aligned daily returns. A strategy that trades in bursts (long flat periods punctuated by short active periods) will have most of its aligned days at zero return, biasing beta toward zero. Read beta together with Time-in-Market.
• Sharpe uses zero risk-free rate. The engine's Sharpe is annualized mean / annualized stdev with no risk-free adjustment. For comparisons to Sharpe values that assume a T-bill hurdle, subtract the T-bill rate manually before comparing.
• Sample-window shrinkage. The aligned window is the intersection of portfolio and SPY dates. A portfolio with a very short live history produces alpha and beta estimates with wide confidence intervals, even if the point estimates look clean.
• Time-in-Market is exit-based, not position-based. A strategy that holds one position for six months and closes it once will show one in-market day, not 120. This is a known limitation of using closed-trade logs rather than tick-level equity paths.

What the Stage IV pipeline will do.

The problem

Prop firms do not evaluate a strategy on its long-run expected return. They evaluate it on a specific rulebook — usually some combination of daily loss limits, trailing drawdown, maximum static drawdown, minimum profit target, and consistency requirements. A strategy with a perfect Stage II report can fail evaluation on a single unlucky Wednesday.

Intended procedure

1. Rulebook ingestion. Load a firm's ruleset from a structured template. Fields include: starting balance, profit target, max total drawdown (static or trailing), max daily loss, minimum trading days, consistency rule, news event restrictions, and payout eligibility triggers.
2. Deterministic pass over historical trades. Replay the trade log tick-by-tick against the ruleset. Record the terminal state: passed, failed (with reason and timestamp), or in-progress.
3. Monte Carlo variation. Bootstrap-resample the trade sequence (as in Stage II Test 4) and re-run the deterministic pass on each resampled path. Report:
   • P(pass) — fraction of paths that reach the profit target without violating any rule
   • P(fail by daily loss)
   • P(fail by total drawdown)
   • P(fail by consistency rule)
   • Expected days-to-pass, conditional on passing

Planned rulesets

Initial coverage targets Topstep, Apex, and Lucid. Additional firms will be added based on user request. Rulesets will live as JSON templates so users can define custom rulebooks without code changes.

Known open design questions

• Intraday drawdown reconstruction. Prop firm daily loss rules trigger on intraday equity, not end-of-day. If the trade log records only closed trades, the intraday equity path is unknown. Stage IV will need to assume worst-case ordering within each day — deliberately pessimistic.
• Rule versioning. Firms change rules. The tool will pin a ruleset version and note the effective date.
• Consistency-rule interpretation. "No single day exceeds X% of total profit" is defined slightly differently by every firm. The template schema will need explicit fields for each interpretation.

What Stage IV will not do

• Model execution slippage against a firm's specific fill rules.
• Model latency or connectivity failures.
• Predict future rule changes.
• Replace reading the actual rulebook. The tool is a check, not a shortcut.