What this report shows

MCPower’s scenario analysis deliberately stresses a planned study: instead of simulating the idealised design you specified, each scenario perturbs it — slopes wobble between observations, residual variance trends with the predictors, correlations differ from the plan, predictor and residual distributions get swapped, group allocation becomes random. Each perturbation is controlled by a knob in configs/scenarios.json, and each knob documents a precise statistical law for the disturbance it injects.

This report checks that every knob generates exactly its documented law — no more, no less. It is the L5 layer of the validation charter: the L3 reports prove the optimistic (unperturbed) generator faithful against the spec as a point oracle; here the spec value is deliberately randomised, so the oracle is the perturbation law itself (including its documented distortions: the ±0.8 correlation clamp, PSD repair, the censored t(3) table, the stale heteroskedasticity anchor).

The gate doctrine. Every gate is set == get on the realised magnitude: the knob’s value, recovered from the generated data by a probe, must match the documented law within an SE-of-mean z-band of 4 (each case draws K independent perturbation blocks; a mis-scaled knob lands tens of σ out, chance alone exceeds the band once in ~10,000 gates). Two checks are explicitly not gates: monotone power across presets (a readable summary that washes out real faults) and a global error-variance invariant (heterogeneity is supposed to inflate it). The β̂-unbiasedness backstop (B1) — including the intercept — is kept as a cheap mean-leak tripwire: a mis-centred swapped distribution lands in β̂₀ while every effect estimate stays clean.

Results at a glance

72 of 72 gates pass. Golden reproducibility: all cases reproduce.

Tier A — one knob at a time

Each case turns on a single knob on an otherwise-optimistic design, draws K independent perturbation blocks, and recovers the knob’s magnitude with a probe matched to its law.

Heterogeneity (slope wobble)

Effects vary per observation: βⱼ + N(0, (h·βⱼ)²). The probe regresses squared true-β residuals on each squared predictor — the slope recovers h²βⱼ² per predictor, separating He from anything that only moves pooled moments.

Heteroskedasticity (residual-variance trend)

Residual variance follows Var(εᵢ) = σ²·exp(γzᵢ)/exp(γ²/2) with γ = ln(λ)/4, z the standardised driver. log e² is then linear in z with slope exactly γ — shape-blind, so the same probe serves B3. The realised ±2σ ratio is exp(4γ̂).

Realised λ̂ = exp(4γ̂) = 4.022 (set: 4); the raw ±2σ binned variance ratio reads 3.95 (reported only — finite bins make its law approximate; the slope is the gate).

Correlation noise

Per block the off-diagonals get symmetrised Gaussian noise — symmetrisation halves the variance, so the per-block ρ law is N(ρ, s²/2) censored at ±0.8, plus finite-n sampling noise in quadrature. The low-ρ case gates the exact law; the high-ρ case sits against the clamp, gating the censored-normal truncation law itself (mean visibly below the naive ρ).

At ρ = 0.75, s = 0.15: the censored law predicts mean 0.7281 — the −0.0219 shift below the nominal ρ is the documented clamp truncation, and the realised mean lands on it.

PSD repair (3 predictors, high ρ)

Repair cannot fire at p = 2 (any clamped 2×2 is PD), so a 3-variable all-0.6 design under s = 0.3 is where eigenvalue-floor + diagonal-renormalisation distortion lives. There is no closed-form law; the empirical per-pair moments are frozen as MCPower-golden (table below) and re-checked on every run.

Distribution swaps (predictors)

Each continuous-normal column is swapped per block with probability q to a uniform pick from the pool; every pool candidate is standardised (mean 0, var 1), so a swap perturbs shape only — a mis-centred candidate would be a mean leak straight into β̂₀.

Block classifications: left_skewed 124, normal 376, right_skewed 158, uniform 142.

Custom pool: high_kurtosis

The only swappable marginal outside the presets. Its engine identity is a censored standardised t(3): a 2048-knot inverse-CDF table on percentiles [0.00121, 0.99879], normalised at build to the censored table’s own SD (1.5958 raw, vs √3 ≈ 1.7321 for the full t(3)) — so the marginal has exactly unit variance, excess kurtosis ≈ 6.39, and support ±6.0 SD. The censoring is deliberate: it bounds every synthetic marginal at ±6 SD while keeping this the heaviest-tailed shape (t(3)’s own kurtosis is infinite). v1 — and this engine until 2026-06 — divided by √3, which standardises the full t(3) and left the censored marginal at var ≈ 0.858, a silent ~14% effect-size shrink for every high-kurtosis predictor; the L5 gate below caught it, and the table is now normalised by construction.

Residual swaps

With probability q_r the block’s residual distribution is replaced — distribution and df — by a pool pick (t(df)·√((df−2)/df) or (χ²(df)−df)/√(2df)). The shape laws (skew = √(8/df), excess kurtosis = 6/(df−4)) recover the df, proving the scenario’s df is carried, not the spec’s.

The pinned case (scen_re_replace) verifies the swap-eligibility rule: it pins the spec residual with set_residual_distribution("high_kurtosis"), then configures a forced right_skewed swap (q_r = 1). Because pick_residual only swaps an unpinned default-normal residual, the swap is inert — every draw keeps the pinned, symmetric censored-t3 high_kurtosis residual (skew ≈ 0, table-normalised to var 1), never the χ²(6) the config asks for. The skew = 0 gate is the tripwire: a fired swap would force skew = √(8/6) ≈ 1.15.

Factor-proportion sampling

sampled_factor_proportions = FALSE (the optimistic default) assigns factor levels by a deterministic largest-remainder walk — counts are a pure function of (n, p), identical across draws, each within 1 of n·p. TRUE draws levels per row: counts are Binomial(n, p) with variance n·p(1−p).

Fixed counts at n = 1000: 500/300/200 against expected 500/300/200. The MLE rows are the one scenario knob the mixed-model estimator admits (every other knob is rejected by the engine’s estimator gate — a deterministic L1 assertion owned by the engine test suite, not re-tested here).

Tier B — knob interactions

The shipped presets co-vary every knob, so B2–B4 isolate single interactions through custom scenario pairs on shared seeds (P1 pairing: two scenarios at the same seed draw the same raw noise streams, so cross-scenario deltas are knob-attributable).

B0 — Optimistic ≡ baseline

The optimistic member of a three-scenario paired call is bit-identical to the plain non-scenario call — the DGP-level companion to the orchestrator’s call-seed test (power here: 0.948, 0.28).

B1 — β̂ unbiased across the presets (leak backstop)

OLS rows gate on the z-band, intercept included — the mean-leak tripwire (linear averaging keeps OLS β̂ exactly unbiased under every knob). Logit rows need a different law: the heterogeneity β-jitter is drawn once per study, so each study’s data is a clean logit at its own β_eff and the MLE recovers it — averaged over the K studies the fitted coefficient → E[β_eff], not the attenuated coefficient a per-observation population-averaged marginal would show. The clip toward zero (s_j = h·|β_j|) nudges each slope’s magnitude up by ×(Φ(1/h)+h·φ(1/h)) ≈ ×1.0008 at doomer’s h = 0.4; the symmetric unclipped intercept jitter leaves β_0 unchanged. So the logit law is this per-study pseudo-true value (glm_perstudy_beta), gated on the absolute band ±0.02. The GLM calibration gates remain Tier A and the flip rate below.

B2 — He × Hs separation

The β-jitter variance ∝ xᵢⱼ²βⱼ² must be present at λ = 1 and unchanged by a λ toggle (λ is driven by the clean linear predictor, never the jittered one). The λ driver is pinned to x2, so the x1²-decomposed jitter variance is uncontaminated by the λ channel’s even cosh component; the paired Δ across {h = 0.4, λ = 1} vs {h = 0.4, λ = 4} isolates the interaction.

The driver column shows why the pin matters: x2²-slope reads 0.0095 at λ = 1 (law h²β₂² = 0.01) but inflates to 0.0689 at λ = 4 — the cosh(γz) contamination the naive probe would misread as an He × Hs interaction.

B3 — Hs × Re preservation

Under a forced t(10) or χ²(10) residual the log-e² slope still reads ln(λ)/4 — the multiplicative variance trend amplifies the swapped tails without bending the ratio.

B4 — Co × Px (NORTA)

Correlation is induced on the latent normals; a swapped marginal transforms them, so the realised Pearson r follows the NORTA law, not the latent spec value. The oracle is computed numerically in R (Gauss–Hermite over the latent bivariate normal; it matches the closed forms (e^ρ−1)/(e−1) and (6/π)asin(ρ/2) to 7 digits).

B5 — Heteroskedasticity-anchor drift (P6), measured and bounded

het_coeffs (the driver moments that standardise the λ driver) is computed once from the base spec; correlation noise moves the realised driver SD σ′ per block while the anchor σ₀ stays put, so the realised ±2σ′ ratio drifts to λ′ = exp(4γσ′/σ₀). This is an accepted approximation — the gates prove the mechanism and bound the drift; they do not fail on the drift itself.

The staleness gate is the discriminating one: per block, the measured ratio λ̂′ tracks the moment-predicted λ′ = exp(4γσ̂′/σ₀) with regression slope ≈ 1. (A per-block recompute of the anchor would pin λ̂′ at λ, slope ≈ 0 — ~11σ away.) Note the mean slope alone cannot tell the two designs apart (γ·E[1/σ′] ≈ γ/σ₀ to 0.1%); the per-block tracking is what evidences the stale anchor. The preset table quantifies the documented drift: under realistic/doomer the effective λ′ wobbles around the set λ by the quantiles shown — second-order next to the perturbations themselves.

Family gating

Knobs must be live or inert exactly per family.

• GLM λ / residual swaps — inert by bit-identity. apply_hsk requires a continuous outcome and consumes no RNG, so a λ toggle is an exact no-op under logit; a forced residual swap only consumes scenario-stream draws after every other consumer, leaving the Bernoulli outcomes bit-identical.
• GLM heterogeneity — live, latent. The per-row log-odds jitter is hidden behind a single Bernoulli draw, so the observable is the paired h-toggle flip rate: X and the uniforms are drawn before the jitter normals, the pair shares them bit-identically, and P(flip) = E|p_h − p₀| — predicted by numerical integration row by row. Realised flip rate 0.0827 vs predicted 0.0804. The Jensen mean-rate shift (+0.0051) is expected and reported, not gated — median latent-rate invariance is unobservable.
• MLE — only sampled_factor_proportions is live (see the Fa section above); every other knob is rejected by the engine’s estimator gate.

Verdict table — every gate

All 72 gates pass and every case reproduces its golden baseline. Each scenario knob generates its documented law: the realised slope wobble, variance trend, correlation noise (with its clamp truncation), swap frequencies, pool moments, allocation behaviour, NORTA coupling, and heteroskedasticity-anchor drift all land on their predicted values, and the β̂ backstop shows no mean leak.

scen_re_multi — multi-grouping RE knobs (M2)

The uniformity law on a crossed extra beside the primary: random_effect_dist / random_effect_df apply to every grouping’s draw, and icc_noise_sd jitters every grouping’s τ² independently. Measurement mirrors scen_re’s level-mean-variance recovery (solver-free); β̂ unbiased is the leak tripwire. The jitter-presence check is paired against a no-jitter (optimistic) control: the realised τ̂² spread under the knob must exceed the control’s Monte-Carlo floor on the extra grouping too.

scen_re_multi: set == get per grouping, independent jitter present, beta unbiased — PASS

How this was produced

Cases live in mcpower/validation/formulas.R (SCENARIO_CASES), probes in mcpower/validation/common.R, gates in mcpower/validation/tolerances.R (SCENARIO_TOL). Golden baseline: mcpower/validation/data/scenario_golden.rds (delete it to re-freeze after a deliberate DGP change). To reproduce, from the repository root:

rmarkdown::render("mcpower/validation/validation_scenarios.rmd",
                  output_dir = "mcpower/web/documentation/validation")