What this report shows

MCPower answers power-analysis questions by generating synthetic datasets from a formula you specify — for example “the outcome y equals 0.25 × x1 plus random noise” — and then simulating your analysis on those datasets many times. Everything downstream depends on one thing being true: the data it generates must actually embody the formula you asked for. If the generator is even slightly wrong, every power number built on top of it is wrong too.

This report checks exactly that. For each supported formula we chose the true coefficients, so we know the right answer in advance. The question is simple:

When an independent, standard statistical analysis is run on MCPower’s generated data, does it recover the true numbers we put in?

If it does — consistently, across every supported kind of model — the generator is faithful.

How the data is generated

For each formula MCPower builds a dataset of n rows:

• continuous predictors are drawn from a standard normal distribution (average 0, standard deviation 1);
• categorical predictors (factors) are assigned at the requested proportions (a 2-level factor becomes a single 0/1 column; a 3-level factor becomes two);
• clustered (mixed-effects) data groups the rows and gives each group a shared random offset, sized to hit the requested intra-cluster correlation (ICC) — specifically the conditional ICC, the correlation that remains after the predictors are accounted for (the raw outcome’s marginal ICC is lower whenever the predictors explain variance);
• the outcome is computed from the true formula and coefficients, then random variation is added — standard-normal noise for ordinary regression, a 0/1 draw at the model’s probability for logistic regression, a cluster offset plus noise for mixed models.

A single dataset is noisy, so we never judge from one. Instead each formula is regenerated many times with fresh random noise — 1,600 times for ordinary and logistic models, 800 for mixed models — and we look at the distribution of results across all of those draws.

The three checks

1. Data checks. The basic statistics of the generated data should match what was requested: predictor averages near 0, standard deviations near 1, the requested correlations and factor proportions, the noise variance, and the ICC. We average each statistic over all draws (which cancels the random scatter of any single dataset) and require it to land within a small fixed tolerance of the requested value.

2. Coefficient recovery — the headline check: is the formula actually true in the data? For every generated dataset we fit a completely independent, standard model in R (lm, glm, or lme4::lmer) and read off the estimated coefficients. Averaged over all draws, each estimate should land on the true value we put in. We measure how far the average estimate is from the truth in units of its own standard error (the estimate’s draw-to-draw scatter divided by the square root of the number of draws). For OLS and mixed-effects models, the ~103 per-coefficient z-scores are pooled across all cases and the false-discovery rate is controlled via Benjamini-Hochberg at q = 0.001 (a fixed per-coefficient z-threshold below ~4 would abort nearly every clean render on sampling noise alone). For logistic regression we use an absolute band instead (|mean − true| ≤ 0.02), because logistic MLE’s finite-sample bias swamps the z-score at this K — see below.

3. File reproduces. The dataset saved to disk for each formula is regenerated from its random seed, and its content fingerprint (a SHA-256 hash) is compared to the saved one. A mismatch would mean the generator’s output had drifted, so any later comparison against the saved file would be meaningless.

The thresholds

The data-check tolerances are tight because they apply to the average over all draws, not to a single dataset — averaging thousands of draws removes the random scatter, so anything outside these bands signals a real generation problem.

A note on logistic regression. Logistic-regression estimates carry a small, well-known finite-sample bias: with a limited number of rows the maximum-likelihood estimate is slightly off even when everything is correct. With this many draws that tiny bias becomes statistically visible (the logistic models below show recovery distances running a bit higher, around 3–4 standard errors), but it stays within tolerance at the sample sizes used here. It is an expected statistical artefact, not a flaw in the generator.

Bias vs. spread. For each formula we report two different things separately. Bias is whether the average estimate is off the true value (a centring problem). Spread is how much the estimates vary from draw to draw. When two predictors are correlated, their estimates naturally vary more — collinearity inflates the spread. That wider spread is expected and correct; it is not a failure, and we say so wherever it appears.

Results at a glance

Every row is one formula with one set of true coefficients. “Data checks” and “Coefficient recovery” summarise the per-formula detail that follows.

All 31 formulas pass all three checks. The generated data matches the requested statistics, an independent R model recovers every true coefficient, and every saved dataset reproduces from its seed.

Formula-by-formula detail

y = 0.25·x1 + noise

R formula y ~ x1 · ordinary least squares (R’s lm) · n = 400 · saved as data/ols_simple_a.rds

How it’s built: one continuous predictor (x1) drawn from a standard normal (mean 0, sd 1). The outcome is the formula above plus standard-normal noise. Sample size 400, random seed 2137.

Data checks — statistics of the generated data, averaged over all draws, vs. what was requested:

Coefficient recovery — an independent R model fitted to each generated dataset; the average estimate should land on the true value:

Every coefficient is centred on its true value — the formula holds in the generated data.

Each panel: the spread of one recovered coefficient across 1600 generated datasets. Red line = the true value, blue dashed = the average estimate. File reproduces from seed: yes.

y = 0.4·x1 + noise

R formula y ~ x1 · ordinary least squares (R’s lm) · n = 400 · saved as data/ols_simple_b.rds

How it’s built: one continuous predictor (x1) drawn from a standard normal (mean 0, sd 1). The outcome is the formula above plus standard-normal noise. Sample size 400, random seed 2138.

Data checks — statistics of the generated data, averaged over all draws, vs. what was requested:

Coefficient recovery — an independent R model fitted to each generated dataset; the average estimate should land on the true value:

Every coefficient is centred on its true value — the formula holds in the generated data.

Each panel: the spread of one recovered coefficient across 1600 generated datasets. Red line = the true value, blue dashed = the average estimate. File reproduces from seed: yes.

y = 0.25·x1 + 0.1·x2 + noise

R formula y ~ x1 + x2 · ordinary least squares (R’s lm) · n = 400 · saved as data/ols_two_a.rds

How it’s built: 2 continuous predictors (x1, x2) drawn from a standard normal (mean 0, sd 1). The outcome is the formula above plus standard-normal noise. Sample size 400, random seed 2137.

Data checks — statistics of the generated data, averaged over all draws, vs. what was requested:

Coefficient recovery — an independent R model fitted to each generated dataset; the average estimate should land on the true value:

Every coefficient is centred on its true value — the formula holds in the generated data.

Each panel: the spread of one recovered coefficient across 1600 generated datasets. Red line = the true value, blue dashed = the average estimate. File reproduces from seed: yes.

y = 0.4·x1 + 0.25·x2 + noise

R formula y ~ x1 + x2 · ordinary least squares (R’s lm) · n = 400 · saved as data/ols_two_b.rds

How it’s built: 2 continuous predictors (x1, x2) drawn from a standard normal (mean 0, sd 1). The outcome is the formula above plus standard-normal noise. Sample size 400, random seed 2138.

Data checks — statistics of the generated data, averaged over all draws, vs. what was requested:

Coefficient recovery — an independent R model fitted to each generated dataset; the average estimate should land on the true value:

Every coefficient is centred on its true value — the formula holds in the generated data.

Each panel: the spread of one recovered coefficient across 1600 generated datasets. Red line = the true value, blue dashed = the average estimate. File reproduces from seed: yes.

y = 0.25·x1 + 0·x2 + noise

R formula y ~ x1 + x2 · ordinary least squares (R’s lm) · n = 400 · saved as data/ols_zero_a.rds

How it’s built: 2 continuous predictors (x1, x2) drawn from a standard normal (mean 0, sd 1). The outcome is the formula above plus standard-normal noise. Sample size 400, random seed 2137.

Data checks — statistics of the generated data, averaged over all draws, vs. what was requested:

Coefficient recovery — an independent R model fitted to each generated dataset; the average estimate should land on the true value:

Every coefficient is centred on its true value — the formula holds in the generated data.

Each panel: the spread of one recovered coefficient across 1600 generated datasets. Red line = the true value, blue dashed = the average estimate. File reproduces from seed: yes.

y = 0.25·x1 + 0.1·x2 + noise

R formula y ~ x1 + x2 · ordinary least squares (R’s lm) · n = 400 · saved as data/ols_corr_a.rds

How it’s built: 2 continuous predictors (x1, x2) drawn from a standard normal (mean 0, sd 1); x1 and x2 correlated at 0.50. The outcome is the formula above plus standard-normal noise. Sample size 400, random seed 2137.

Data checks — statistics of the generated data, averaged over all draws, vs. what was requested:

Coefficient recovery — an independent R model fitted to each generated dataset; the average estimate should land on the true value:

Every coefficient is centred on its true value — the formula holds in the generated data. Because x1 and x2 are correlated (0.50), their estimates vary more from draw to draw; that wider spread is the expected effect of collinearity, not a problem.

Each panel: the spread of one recovered coefficient across 1600 generated datasets. Red line = the true value, blue dashed = the average estimate. File reproduces from seed: yes.

y = 0.4·x1 + 0.25·x2 + noise

R formula y ~ x1 + x2 · ordinary least squares (R’s lm) · n = 400 · saved as data/ols_corr_b.rds

How it’s built: 2 continuous predictors (x1, x2) drawn from a standard normal (mean 0, sd 1); x1 and x2 correlated at 0.30. The outcome is the formula above plus standard-normal noise. Sample size 400, random seed 2138.

Data checks — statistics of the generated data, averaged over all draws, vs. what was requested:

Coefficient recovery — an independent R model fitted to each generated dataset; the average estimate should land on the true value:

Every coefficient is centred on its true value — the formula holds in the generated data. Because x1 and x2 are correlated (0.30), their estimates vary more from draw to draw; that wider spread is the expected effect of collinearity, not a problem.

Each panel: the spread of one recovered coefficient across 1600 generated datasets. Red line = the true value, blue dashed = the average estimate. File reproduces from seed: yes.

y = 0.25·x1 + 0.1·x2 − 0.2·x1:x2 + noise

R formula y ~ x1*x2 · ordinary least squares (R’s lm) · n = 600 · saved as data/ols_interaction_a.rds

How it’s built: 2 continuous predictors (x1, x2) drawn from a standard normal (mean 0, sd 1); x1 and x2 correlated at 0.30. The outcome is the formula above plus standard-normal noise. Sample size 600, random seed 2137.

Data checks — statistics of the generated data, averaged over all draws, vs. what was requested:

Coefficient recovery — an independent R model fitted to each generated dataset; the average estimate should land on the true value:

Every coefficient is centred on its true value — the formula holds in the generated data. Because x1 and x2 are correlated (0.30), their estimates vary more from draw to draw; that wider spread is the expected effect of collinearity, not a problem.

Each panel: the spread of one recovered coefficient across 1600 generated datasets. Red line = the true value, blue dashed = the average estimate. File reproduces from seed: yes.

y = 0.4·x1 + 0.25·x2 + 0.15·x1:x2 + noise

R formula y ~ x1*x2 · ordinary least squares (R’s lm) · n = 600 · saved as data/ols_interaction_b.rds

How it’s built: 2 continuous predictors (x1, x2) drawn from a standard normal (mean 0, sd 1). The outcome is the formula above plus standard-normal noise. Sample size 600, random seed 2138.

Data checks — statistics of the generated data, averaged over all draws, vs. what was requested:

Coefficient recovery — an independent R model fitted to each generated dataset; the average estimate should land on the true value:

Every coefficient is centred on its true value — the formula holds in the generated data.

Each panel: the spread of one recovered coefficient across 1600 generated datasets. Red line = the true value, blue dashed = the average estimate. File reproduces from seed: yes.

y = 0.25·x1 + 0.5·g[2] + 0.8·g[3] + noise

R formula y ~ x1 + g · ordinary least squares (R’s lm) · n = 600 · saved as data/ols_factor_a.rds

How it’s built: one continuous predictor (x1) drawn from a standard normal (mean 0, sd 1); a 3-level factor g at proportions 50%/30%/20%. The outcome is the formula above plus standard-normal noise. Sample size 600, random seed 2137.

Data checks — statistics of the generated data, averaged over all draws, vs. what was requested:

Coefficient recovery — an independent R model fitted to each generated dataset; the average estimate should land on the true value:

Every coefficient is centred on its true value — the formula holds in the generated data.

Each panel: the spread of one recovered coefficient across 1600 generated datasets. Red line = the true value, blue dashed = the average estimate. File reproduces from seed: yes.

y = 0.4·x1 + 0.2·g[2] + 0.5·g[3] + noise

R formula y ~ x1 + g · ordinary least squares (R’s lm) · n = 600 · saved as data/ols_factor_b.rds

How it’s built: one continuous predictor (x1) drawn from a standard normal (mean 0, sd 1); a 3-level factor g at proportions 40%/35%/25%. The outcome is the formula above plus standard-normal noise. Sample size 600, random seed 2138.

Data checks — statistics of the generated data, averaged over all draws, vs. what was requested:

Coefficient recovery — an independent R model fitted to each generated dataset; the average estimate should land on the true value:

Every coefficient is centred on its true value — the formula holds in the generated data.

Each panel: the spread of one recovered coefficient across 1600 generated datasets. Red line = the true value, blue dashed = the average estimate. File reproduces from seed: yes.

y = 0.3·x1 + 0.4·g[2] + 0.6·g[3] + 0.2·x1:g[2] + 0.3·x1:g[3] + noise

R formula y ~ x1*g · ordinary least squares (R’s lm) · n = 800 · saved as data/ols_cf_a.rds

How it’s built: one continuous predictor (x1) drawn from a standard normal (mean 0, sd 1); a 3-level factor g at proportions 50%/30%/20%. The outcome is the formula above plus standard-normal noise. Sample size 800, random seed 2137.

Data checks — statistics of the generated data, averaged over all draws, vs. what was requested:

Coefficient recovery — an independent R model fitted to each generated dataset; the average estimate should land on the true value:

Every coefficient is centred on its true value — the formula holds in the generated data.

Each panel: the spread of one recovered coefficient across 1600 generated datasets. Red line = the true value, blue dashed = the average estimate. File reproduces from seed: yes.

y = 0.4·x1 + 0.5·g[2] + 0.8·g[3] + 0.25·x1:g[2] + 0.4·x1:g[3] + noise

R formula y ~ x1*g · ordinary least squares (R’s lm) · n = 800 · saved as data/ols_cf_b.rds

How it’s built: one continuous predictor (x1) drawn from a standard normal (mean 0, sd 1); a 3-level factor g at proportions 40%/35%/25%. The outcome is the formula above plus standard-normal noise. Sample size 800, random seed 2138.

Data checks — statistics of the generated data, averaged over all draws, vs. what was requested:

Coefficient recovery — an independent R model fitted to each generated dataset; the average estimate should land on the true value:

Every coefficient is centred on its true value — the formula holds in the generated data.

Each panel: the spread of one recovered coefficient across 1600 generated datasets. Red line = the true value, blue dashed = the average estimate. File reproduces from seed: yes.

y = 0.5·g1[2] + 0.4·g2[2] + 0.3·g1[2]:g2[2] + noise

R formula y ~ g1*g2 · ordinary least squares (R’s lm) · n = 800 · saved as data/ols_ff_a.rds

How it’s built: a 2-level factor g1 at proportions 50%/50%; a 2-level factor g2 at proportions 60%/40%. The outcome is the formula above plus standard-normal noise. Sample size 800, random seed 2137.

Data checks — statistics of the generated data, averaged over all draws, vs. what was requested:

Coefficient recovery — an independent R model fitted to each generated dataset; the average estimate should land on the true value:

Every coefficient is centred on its true value — the formula holds in the generated data.

Each panel: the spread of one recovered coefficient across 1600 generated datasets. Red line = the true value, blue dashed = the average estimate. File reproduces from seed: yes.

y = 0.2·g1[2] + 0.8·g2[2] + 0.5·g1[2]:g2[2] + noise

R formula y ~ g1*g2 · ordinary least squares (R’s lm) · n = 800 · saved as data/ols_ff_b.rds

How it’s built: a 2-level factor g1 at proportions 50%/50%; a 2-level factor g2 at proportions 55%/45%. The outcome is the formula above plus standard-normal noise. Sample size 800, random seed 2138.

Data checks — statistics of the generated data, averaged over all draws, vs. what was requested:

Coefficient recovery — an independent R model fitted to each generated dataset; the average estimate should land on the true value:

Every coefficient is centred on its true value — the formula holds in the generated data.

Each panel: the spread of one recovered coefficient across 1600 generated datasets. Red line = the true value, blue dashed = the average estimate. File reproduces from seed: yes.

log-odds(y = 1) = logit(0.30) + 0.5·x1

R formula y ~ x1 · logistic regression (R’s glm) · n = 600 · saved as data/glm_simple_a.rds

How it’s built: one continuous predictor (x1) drawn from a standard normal (mean 0, sd 1). The outcome is 1 or 0, drawn at the logistic probability of the formula above (baseline rate 30%). Sample size 600, random seed 2137.

Data checks — statistics of the generated data, averaged over all draws, vs. what was requested:

Coefficient recovery — an independent R model fitted to each generated dataset; the average estimate should land on the true value:

Every coefficient is centred on its true value — the formula holds in the generated data.

Each panel: the spread of one recovered coefficient across 1600 generated datasets. Red line = the true value, blue dashed = the average estimate. File reproduces from seed: yes.

log-odds(y = 1) = logit(0.50) + 0.8·x1

R formula y ~ x1 · logistic regression (R’s glm) · n = 600 · saved as data/glm_simple_b.rds

How it’s built: one continuous predictor (x1) drawn from a standard normal (mean 0, sd 1). The outcome is 1 or 0, drawn at the logistic probability of the formula above (baseline rate 50%). Sample size 600, random seed 2138.

Data checks — statistics of the generated data, averaged over all draws, vs. what was requested:

Coefficient recovery — an independent R model fitted to each generated dataset; the average estimate should land on the true value:

Every coefficient is centred on its true value — the formula holds in the generated data.

Each panel: the spread of one recovered coefficient across 1600 generated datasets. Red line = the true value, blue dashed = the average estimate. File reproduces from seed: yes.

log-odds(y = 1) = logit(0.30) + 0.5·x1 + 0.3·x2

R formula y ~ x1 + x2 · logistic regression (R’s glm) · n = 800 · saved as data/glm_two_a.rds

How it’s built: 2 continuous predictors (x1, x2) drawn from a standard normal (mean 0, sd 1); x1 and x2 correlated at 0.20. The outcome is 1 or 0, drawn at the logistic probability of the formula above (baseline rate 30%). Sample size 800, random seed 2137.

Data checks — statistics of the generated data, averaged over all draws, vs. what was requested:

Coefficient recovery — an independent R model fitted to each generated dataset; the average estimate should land on the true value:

Every coefficient is centred on its true value — the formula holds in the generated data. Because x1 and x2 are correlated (0.20), their estimates vary more from draw to draw; that wider spread is the expected effect of collinearity, not a problem.

Each panel: the spread of one recovered coefficient across 1600 generated datasets. Red line = the true value, blue dashed = the average estimate. File reproduces from seed: yes.

log-odds(y = 1) = logit(0.50) + 0.8·x1 + 0.5·x2

R formula y ~ x1 + x2 · logistic regression (R’s glm) · n = 800 · saved as data/glm_two_b.rds

How it’s built: 2 continuous predictors (x1, x2) drawn from a standard normal (mean 0, sd 1). The outcome is 1 or 0, drawn at the logistic probability of the formula above (baseline rate 50%). Sample size 800, random seed 2138.

Data checks — statistics of the generated data, averaged over all draws, vs. what was requested:

Coefficient recovery — an independent R model fitted to each generated dataset; the average estimate should land on the true value:

Every coefficient is centred on its true value — the formula holds in the generated data.

Each panel: the spread of one recovered coefficient across 1600 generated datasets. Red line = the true value, blue dashed = the average estimate. File reproduces from seed: yes.

log-odds(y = 1) = logit(0.30) + 0.5·x1 + 0.4·g[2] + 0.8·g[3]

R formula y ~ x1 + g · logistic regression (R’s glm) · n = 1000 · saved as data/glm_factor_a.rds

How it’s built: one continuous predictor (x1) drawn from a standard normal (mean 0, sd 1); a 3-level factor g at proportions 50%/30%/20%. The outcome is 1 or 0, drawn at the logistic probability of the formula above (baseline rate 30%). Sample size 1000, random seed 2137.

Data checks — statistics of the generated data, averaged over all draws, vs. what was requested:

Coefficient recovery — an independent R model fitted to each generated dataset; the average estimate should land on the true value:

Every coefficient is centred on its true value — the formula holds in the generated data.

Each panel: the spread of one recovered coefficient across 1600 generated datasets. Red line = the true value, blue dashed = the average estimate. File reproduces from seed: yes.

log-odds(y = 1) = logit(0.50) + 0.8·x1 + 0.5·g[2] + 0.8·g[3]

R formula y ~ x1 + g · logistic regression (R’s glm) · n = 1000 · saved as data/glm_factor_b.rds

How it’s built: one continuous predictor (x1) drawn from a standard normal (mean 0, sd 1); a 3-level factor g at proportions 40%/35%/25%. The outcome is 1 or 0, drawn at the logistic probability of the formula above (baseline rate 50%). Sample size 1000, random seed 2138.

Data checks — statistics of the generated data, averaged over all draws, vs. what was requested:

Coefficient recovery — an independent R model fitted to each generated dataset; the average estimate should land on the true value:

Every coefficient is centred on its true value — the formula holds in the generated data.

Each panel: the spread of one recovered coefficient across 1600 generated datasets. Red line = the true value, blue dashed = the average estimate. File reproduces from seed: yes.

log-odds(y = 1) = logit(0.30) + 0.5·x1 + 0.3·x2 + 0.3·x1:x2

R formula y ~ x1*x2 · logistic regression (R’s glm) · n = 1000 · saved as data/glm_interaction_a.rds

How it’s built: 2 continuous predictors (x1, x2) drawn from a standard normal (mean 0, sd 1). The outcome is 1 or 0, drawn at the logistic probability of the formula above (baseline rate 30%). Sample size 1000, random seed 2137.

Data checks — statistics of the generated data, averaged over all draws, vs. what was requested:

Coefficient recovery — an independent R model fitted to each generated dataset; the average estimate should land on the true value:

Every coefficient is centred on its true value — the formula holds in the generated data.

Each panel: the spread of one recovered coefficient across 1600 generated datasets. Red line = the true value, blue dashed = the average estimate. File reproduces from seed: yes.

log-odds(y = 1) = logit(0.50) + 0.8·x1 + 0.5·x2 + 0.4·x1:x2

R formula y ~ x1*x2 · logistic regression (R’s glm) · n = 1000 · saved as data/glm_interaction_b.rds

How it’s built: 2 continuous predictors (x1, x2) drawn from a standard normal (mean 0, sd 1). The outcome is 1 or 0, drawn at the logistic probability of the formula above (baseline rate 50%). Sample size 1000, random seed 2138.

Data checks — statistics of the generated data, averaged over all draws, vs. what was requested:

Coefficient recovery — an independent R model fitted to each generated dataset; the average estimate should land on the true value:

Every coefficient is centred on its true value — the formula holds in the generated data.

Each panel: the spread of one recovered coefficient across 1600 generated datasets. Red line = the true value, blue dashed = the average estimate. File reproduces from seed: yes.

y = 0.5·x1 + per-grp random intercept (ICC 0.20) + noise

R formula y ~ x1 + (1|grp) · linear mixed model (lme4::lmer) · n = 600 · saved as data/lme_simple_a.rds

How it’s built: one continuous predictor (x1) drawn from a standard normal (mean 0, sd 1). The outcome is the formula above, plus a shared offset for each of the 20 clusters (30 observations each, sized to an intra-cluster correlation of 0.20), plus standard-normal noise. Sample size 600, random seed 2137.

Data checks — statistics of the generated data, averaged over all draws, vs. what was requested:

Coefficient recovery — an independent R model fitted to each generated dataset; the average estimate should land on the true value:

Every coefficient is centred on its true value — the formula holds in the generated data.

The intra-cluster correlation you set is the conditional ICC — the correlation between two observations in the same cluster after accounting for the predictors — and the generator recovers it (the “intra-cluster correlation (ICC)” row above). The observed (marginal) ICC of the raw outcome is lower, because the predictors explain part of the total variance; the stronger the fixed effects, the larger the gap. This is the standard conditional-vs-marginal distinction — expected and correct, not a generation fault — and the “observed (marginal) ICC vs predicted” row confirms the observed value lands on what that distinction predicts.

Each panel: the spread of one recovered coefficient across 800 generated datasets. Red line = the true value, blue dashed = the average estimate. File reproduces from seed: yes.

y = 0.3·x1 + per-grp random intercept (ICC 0.30) + noise

R formula y ~ x1 + (1|grp) · linear mixed model (lme4::lmer) · n = 600 · saved as data/lme_simple_b.rds

How it’s built: one continuous predictor (x1) drawn from a standard normal (mean 0, sd 1). The outcome is the formula above, plus a shared offset for each of the 20 clusters (30 observations each, sized to an intra-cluster correlation of 0.30), plus standard-normal noise. Sample size 600, random seed 2138.

Data checks — statistics of the generated data, averaged over all draws, vs. what was requested:

Coefficient recovery — an independent R model fitted to each generated dataset; the average estimate should land on the true value:

Every coefficient is centred on its true value — the formula holds in the generated data.

The intra-cluster correlation you set is the conditional ICC — the correlation between two observations in the same cluster after accounting for the predictors — and the generator recovers it (the “intra-cluster correlation (ICC)” row above). The observed (marginal) ICC of the raw outcome is lower, because the predictors explain part of the total variance; the stronger the fixed effects, the larger the gap. This is the standard conditional-vs-marginal distinction — expected and correct, not a generation fault — and the “observed (marginal) ICC vs predicted” row confirms the observed value lands on what that distinction predicts.

Each panel: the spread of one recovered coefficient across 800 generated datasets. Red line = the true value, blue dashed = the average estimate. File reproduces from seed: yes.

y = 0.5·x1 + 0.3·x2 + per-grp random intercept (ICC 0.20) + noise

R formula y ~ x1 + x2 + (1|grp) · linear mixed model (lme4::lmer) · n = 750 · saved as data/lme_two_a.rds

How it’s built: 2 continuous predictors (x1, x2) drawn from a standard normal (mean 0, sd 1). The outcome is the formula above, plus a shared offset for each of the 25 clusters (30 observations each, sized to an intra-cluster correlation of 0.20), plus standard-normal noise. Sample size 750, random seed 2137.

Data checks — statistics of the generated data, averaged over all draws, vs. what was requested:

Coefficient recovery — an independent R model fitted to each generated dataset; the average estimate should land on the true value:

Every coefficient is centred on its true value — the formula holds in the generated data.

The intra-cluster correlation you set is the conditional ICC — the correlation between two observations in the same cluster after accounting for the predictors — and the generator recovers it (the “intra-cluster correlation (ICC)” row above). The observed (marginal) ICC of the raw outcome is lower, because the predictors explain part of the total variance; the stronger the fixed effects, the larger the gap. This is the standard conditional-vs-marginal distinction — expected and correct, not a generation fault — and the “observed (marginal) ICC vs predicted” row confirms the observed value lands on what that distinction predicts.

Each panel: the spread of one recovered coefficient across 800 generated datasets. Red line = the true value, blue dashed = the average estimate. File reproduces from seed: yes.

y = 0.3·x1 + 0.5·x2 + per-grp random intercept (ICC 0.30) + noise

R formula y ~ x1 + x2 + (1|grp) · linear mixed model (lme4::lmer) · n = 750 · saved as data/lme_two_b.rds

How it’s built: 2 continuous predictors (x1, x2) drawn from a standard normal (mean 0, sd 1). The outcome is the formula above, plus a shared offset for each of the 25 clusters (30 observations each, sized to an intra-cluster correlation of 0.30), plus standard-normal noise. Sample size 750, random seed 2138.

Data checks — statistics of the generated data, averaged over all draws, vs. what was requested:

Coefficient recovery — an independent R model fitted to each generated dataset; the average estimate should land on the true value:

Every coefficient is centred on its true value — the formula holds in the generated data.

The intra-cluster correlation you set is the conditional ICC — the correlation between two observations in the same cluster after accounting for the predictors — and the generator recovers it (the “intra-cluster correlation (ICC)” row above). The observed (marginal) ICC of the raw outcome is lower, because the predictors explain part of the total variance; the stronger the fixed effects, the larger the gap. This is the standard conditional-vs-marginal distinction — expected and correct, not a generation fault — and the “observed (marginal) ICC vs predicted” row confirms the observed value lands on what that distinction predicts.

Each panel: the spread of one recovered coefficient across 800 generated datasets. Red line = the true value, blue dashed = the average estimate. File reproduces from seed: yes.

y = 0.5·x1 + 0.3·x2 + 0.3·x1:x2 + per-grp random intercept (ICC 0.20) + noise

R formula y ~ x1*x2 + (1|grp) · linear mixed model (lme4::lmer) · n = 750 · saved as data/lme_interaction_a.rds

How it’s built: 2 continuous predictors (x1, x2) drawn from a standard normal (mean 0, sd 1). The outcome is the formula above, plus a shared offset for each of the 25 clusters (30 observations each, sized to an intra-cluster correlation of 0.20), plus standard-normal noise. Sample size 750, random seed 2137.

Data checks — statistics of the generated data, averaged over all draws, vs. what was requested:

Coefficient recovery — an independent R model fitted to each generated dataset; the average estimate should land on the true value:

Every coefficient is centred on its true value — the formula holds in the generated data.

The intra-cluster correlation you set is the conditional ICC — the correlation between two observations in the same cluster after accounting for the predictors — and the generator recovers it (the “intra-cluster correlation (ICC)” row above). The observed (marginal) ICC of the raw outcome is lower, because the predictors explain part of the total variance; the stronger the fixed effects, the larger the gap. This is the standard conditional-vs-marginal distinction — expected and correct, not a generation fault — and the “observed (marginal) ICC vs predicted” row confirms the observed value lands on what that distinction predicts.

Each panel: the spread of one recovered coefficient across 800 generated datasets. Red line = the true value, blue dashed = the average estimate. File reproduces from seed: yes.

y = 0.4·x1 + 0.3·x2 + 0.2·x1:x2 + per-grp random intercept (ICC 0.30) + noise

R formula y ~ x1*x2 + (1|grp) · linear mixed model (lme4::lmer) · n = 900 · saved as data/lme_interaction_b.rds

How it’s built: 2 continuous predictors (x1, x2) drawn from a standard normal (mean 0, sd 1). The outcome is the formula above, plus a shared offset for each of the 30 clusters (30 observations each, sized to an intra-cluster correlation of 0.30), plus standard-normal noise. Sample size 900, random seed 2138.

Data checks — statistics of the generated data, averaged over all draws, vs. what was requested:

Coefficient recovery — an independent R model fitted to each generated dataset; the average estimate should land on the true value:

Every coefficient is centred on its true value — the formula holds in the generated data.

The intra-cluster correlation you set is the conditional ICC — the correlation between two observations in the same cluster after accounting for the predictors — and the generator recovers it (the “intra-cluster correlation (ICC)” row above). The observed (marginal) ICC of the raw outcome is lower, because the predictors explain part of the total variance; the stronger the fixed effects, the larger the gap. This is the standard conditional-vs-marginal distinction — expected and correct, not a generation fault — and the “observed (marginal) ICC vs predicted” row confirms the observed value lands on what that distinction predicts.

Each panel: the spread of one recovered coefficient across 800 generated datasets. Red line = the true value, blue dashed = the average estimate. File reproduces from seed: yes.

y = 0.3·x1 + 0.3·g[2] + per-grp random intercept (ICC 0.20) + noise

R formula y ~ x1 + g + (1|grp) · linear mixed model (lme4::lmer) · n = 750 · saved as data/lme_factor_a.rds

How it’s built: one continuous predictor (x1) drawn from a standard normal (mean 0, sd 1); a 2-level factor g at proportions 50%/50%. The outcome is the formula above, plus a shared offset for each of the 25 clusters (30 observations each, sized to an intra-cluster correlation of 0.20), plus standard-normal noise. Sample size 750, random seed 2137.

Data checks — statistics of the generated data, averaged over all draws, vs. what was requested:

Coefficient recovery — an independent R model fitted to each generated dataset; the average estimate should land on the true value:

Every coefficient is centred on its true value — the formula holds in the generated data.

The intra-cluster correlation you set is the conditional ICC — the correlation between two observations in the same cluster after accounting for the predictors — and the generator recovers it (the “intra-cluster correlation (ICC)” row above). The observed (marginal) ICC of the raw outcome is lower, because the predictors explain part of the total variance; the stronger the fixed effects, the larger the gap. This is the standard conditional-vs-marginal distinction — expected and correct, not a generation fault — and the “observed (marginal) ICC vs predicted” row confirms the observed value lands on what that distinction predicts.

Each panel: the spread of one recovered coefficient across 800 generated datasets. Red line = the true value, blue dashed = the average estimate. File reproduces from seed: yes.

y = 0.4·x1 + 0.5·g[2] + 0.8·g[3] + per-grp random intercept (ICC 0.30) + noise

R formula y ~ x1 + g + (1|grp) · linear mixed model (lme4::lmer) · n = 900 · saved as data/lme_factor_b.rds

How it’s built: one continuous predictor (x1) drawn from a standard normal (mean 0, sd 1); a 3-level factor g at proportions 50%/30%/20%. The outcome is the formula above, plus a shared offset for each of the 30 clusters (30 observations each, sized to an intra-cluster correlation of 0.30), plus standard-normal noise. Sample size 900, random seed 2138.

Data checks — statistics of the generated data, averaged over all draws, vs. what was requested:

Coefficient recovery — an independent R model fitted to each generated dataset; the average estimate should land on the true value:

Every coefficient is centred on its true value — the formula holds in the generated data.

The intra-cluster correlation you set is the conditional ICC — the correlation between two observations in the same cluster after accounting for the predictors — and the generator recovers it (the “intra-cluster correlation (ICC)” row above). The observed (marginal) ICC of the raw outcome is lower, because the predictors explain part of the total variance; the stronger the fixed effects, the larger the gap. This is the standard conditional-vs-marginal distinction — expected and correct, not a generation fault — and the “observed (marginal) ICC vs predicted” row confirms the observed value lands on what that distinction predicts.

Each panel: the spread of one recovered coefficient across 800 generated datasets. Red line = the true value, blue dashed = the average estimate. File reproduces from seed: yes.

How this was produced

The datasets are generated by mcpower/validation/data_generation.r from the formula catalogue in mcpower/validation/formulas.R; this report regenerates them many times over and runs the checks above. To reproduce it, from the repository root:

rmarkdown::render("mcpower/validation/validation_data_generation.rmd",
                  output_dir = "mcpower/web/documentation/validation")