Tolerances and recorded conventions

Part of the numerical-validation suite. What each tolerance means, and every convention chosen where a statistic has more than one defensible definition.

Most statistics agree to 1e-4–1e-6. A looser tolerance always names a reason in the reference's source string. Where a statistic has more than one defensible definition, the reference is computed in the engine's convention and the choice is recorded here — these are agreements about definitions, not discrepancies:

Quantiles (Explore percentiles) — the engine uses the Hazen definition (plotting position (k − ½)/n); the reference uses numpy.percentile(method="hazen"). (NumPy's default "linear" is a different, equally valid convention.)
Mann-Whitney — the engine reports the asymptotic test from a tie-corrected z with no continuity correction; the reference derives both z and p from that same z, rather than SciPy's continuity-corrected p.
Type III SS (factorial GLM) — only well-defined under sum-to-zero contrasts; the reference fits C(group, Sum)*C(sex, Sum) so statsmodels' anova_lm(typ=3) reproduces the engine's main-effect SS in the presence of an interaction.
ARIMA — the engine fits AR by conditional least squares; the AR(1) reference is the OLS lag-1 regression (the exact CLS estimator), not statsmodels' unconditional ML, which would differ slightly.
Negative binomial — the dispersion α is ML-estimated, with a looser 5e-2 tolerance on the dispersion alone (the mean-model coefficients match tightly).
CFA — five recorded conventions, each pinned against the references. (1) Scale setting: each factor's first indicator is the reference, its loading fixed at 1; factor variances are free. (2) χ² = N·F with divisor N (not N−1) — pinned by reproducing lavaan's published Holzinger–Swineford χ² = 85.306 = 301·F. (3) Standard errors use the expected (Fisher) information with divisor N — the lavaan/semopy default; the observed-Hessian alternative disagrees with both references exactly where the model misfits, so it is deliberately not used. (4) The baseline model behind CFI/TLI is the independence model (diagonal Σ at the ML variances, closed form F_b = Σ ln s_jj − ln|S|); a saturated (df = 0) model reports CFI = TLI = 1 and no χ² p-value by convention. (5) The RMSEA CI is 90% by convention (not the analysis CI level), from inverting the noncentral χ². The sample covariance uses divisor N (weighted: Σw), matching the estimates the references produce.
CFA modification indices — the 1-df score test of each fixed parameter from the expected information (divisor N), lavaan's epc and sepc.all; non-identified candidates skipped (lavaan's NA rows); listing floor 3.84 by default. Pinned against lavaan::modindices() on Holzinger–Swineford, correlated and orthogonal — see CFA follow-ups.
Johnson–Neyman regions — the roots of |β₁ + β₃w| = t_crit·SE(w) (t on the residual df at the interval's α), reported inside the observed moderator range with the significant side read from the quadratic's sign; pinned against interactions::johnson_neyman — see Mediation follow-ups.
Three-way moderated mediation — Z joins W in the moderated model(s) with every product (PROCESS models 11 / 18), the Z-outer × W-inner probe grid, the index of moderated moderated mediation a7·b / a·b7 and the conditional index (a3 + a7z)·b / a·(b3 + b7z) (Hayes 2018); pinned against lm/vcov — see Mediation follow-ups.
Serial mediation — two mediators in causal order (PROCESS model 6): M₂ ~ X + M₁ for d₂₁, the specific indirects a₁b₁, a₂b₂, a₁d₂₁b₂ and c = c′ + Σ exactly; one moderator on the a- or b-paths with d₂₁ unmoderated and a per-effect index; pinned against lm — see Mediation follow-ups.
Meta-analysis — three recorded conventions. (1) The between-study variance is the DerSimonian–Laird moment estimate, truncated at zero (as is I²) — the metafor/textbook convention; statsmodels' combine_effects reports the raw (possibly negative) τ² and pools with it, so meta.csv is built deliberately heterogeneous (the generator asserts raw τ² > 0) to keep the two sides comparable — a homogeneous dataset would make the engine's truncated RE collapse onto FE while statsmodels' would not. (2) Pooled and per-study intervals are large-sample Wald (z-based); the Knapp–Hartung adjustment is out of scope for v1. (3) Egger's test is the classic OLS of the standardized effect on the precision with the intercept tested at k − 2 df (equivalent to WLS of effect on SE under 1/SE² weights). The dataset's Weight Cases setting is meaningless by design for meta-analysis — rows are studies and the inverse-variance weights are the statistic's own — so, like time series, it carries the standard unweighted note and has no weighted reference.
Case weighting — the engine treats Weight Cases as a frequency weight: a weight-w case behaves exactly like w coincident copies (the procedures weighted on 2026-06-18 — clustering, discriminant, RM-ANOVA/MANOVA, survival — are tested against that weight ≡ replication identity). One sub-case carries a convention: Cox with the Efron ties correction under non-integer weights. The tied-event discount steps over the weighted event total, which reproduces replication exactly for integer weights but is an approximation for fractional ones (the same approach lifelines takes); the default Breslow correction is exact for any weights. Time series (ACF/PACF, ARIMA) is deliberately unweighted — a frequency weight is undefined for a series taken in order — so it has no weighted reference. The REML τ² (0.11.0) is the standard fixed-point iteration on the restricted likelihood — metafor's default estimator — bounded at zero, iteration-capped with a hit cap reported in the output note; Q, its p and I² stay defined from the fixed-effect fit and are estimator-independent, so only τ², the random-effects pooling and the per-study random weights change with the tau_method choice.
Violin plot (kernel density) — four recorded conventions (0.12.0). (1) The bandwidth is Silverman's scaling rule as scipy implements it, h = (3W/4)^(−1/5)·σ with σ the ddof-1 standard deviation — deliberately not R bw.nrd0's 0.9·min(σ, IQR/1.34)·n^(−1/5) rule of thumb, so the unweighted comparison against scipy.stats.gaussian_kde(bw_method='silverman') is exact (the Rust unit tests pin scipy's own numbers to 1e-12, and the gate-4 ids pin them again end-to-end at 1e-9). (2) Weights are frequency weights, W = Σw in both the bandwidth and the variance, keeping the engine-wide weight ≡ replication identity — scipy's own weights= parameter is an importance-weight convention (Kish effective N, reliability-weights variance) and is deliberately not matched; the weighted identity is asserted in rust/src/analyze/graphs.rs's violin_tests rather than pinned here, since this suite runs unweighted. (3) The density curve is evaluated on a fixed 96-point grid over [min − 3h, max + 3h] (R density's cut = 3 convention), and the gate-4 density ids name grid rows, not quantiles, so no quantile convention can drift between the sides. (4) A group with fewer than two distinct values (or W ≤ 1) has no density by design — its statistics row keeps null bandwidth and the table note names the group; the run never fails for it.
Propensity score — five recorded conventions (0.13.0). (1) The score model is the same binomial IRLS fit logistic regression uses, pinned against statsmodels' independent Logit MLE at 1e-6. (2) Matching distance and the caliper live on the logit of the score; the caliper option is a multiple of the SD of the logit over the analyzed sample — the scale Austin (2011)'s 0.2 recommendation is stated on. (3) Matching is greedy 1:k nearest-neighbour without replacement, estimand ATT, treated processed in descending score order with every tie (processing order and equidistant candidates) broken by dataset order, so the result is deterministic; the gate-4 matching ids are pinned against an independent NumPy implementation of exactly these conventions, not against a package — MatchIt's nearest-neighbour defaults differ (distance scale, processing order), so exact parity with its documented examples is not expected and a transcribed number would corroborate nothing. (4) Balance is the standardized mean difference over the unadjusted pooled SD √((s²T+s²C)/2), the same denominator before and after adjustment; IPTW weights are 1/e vs 1/(1−e) for ate (default) and 1 vs e/(1−e) for att. (5) The whole procedure is computed unweighted by design: weight ≡ replication is not well defined for pairing, and IPTW layered on a frequency weight would compound two case multiplicities — a pre-existing Weight Cases variable only selects cases (mark_unweighted), and the weight the procedure emits is the product.
Panel regression — six recorded conventions (0.15.0), each decided before the core was written (the release plan's risk shape — docs/releases/release_0.15.0.md, kept private — demanded exactly this) and each verified against the published Stata results for the classic 10-firm Grunfeld data before being pinned. The reference package is linearmodels (pinned in the JSON header), whose defaults reproduce Stata's xtreg numbers on Grunfeld to publication precision. (1) RE variance components: Swamy–Arora, exactly as linearmodels.RandomEffects (default small_sample=False) implements it — σ²ₑ = within-SSR / (N − n − K); the between regression is OLS of entity means on entity means with constant, σ²ᵤ = max(0, between-SSR/(n − K − 1) − σ²ₑ/T̄) with T̄ the harmonic mean of the entity sizes; per-entity θᵢ = 1 − √(σ²ₑ/(Tᵢσ²ᵤ + σ²ₑ)). Wallace–Hussain and Amemiya give visibly different θ and are deliberately not offered. Verified: θ = 0.861224 on Grunfeld, Stata's published value. (2) FE degrees of freedom: the n − 1 absorbed entity effects are charged against the residual df — df = N − n − K, matching Stata xtreg, fe, plm's within default and linearmodels debiased=True. This is the single most common source of "your numbers don't match" reports, so the output note names the df. The FE intercept is reported Stata-style: the within regression runs on demeaned data with the grand means added back, so the constant is the grand-mean intercept (−58.744 on Grunfeld). (3) Hausman: the classic form, slopes only (no intercept), H = d′(V_FE − V_RE)⁻¹d on the conventional (homoskedastic) covariances — the classic test's efficiency argument requires the efficient RE estimator, so it is never computed from clustered covariances. df = the number of common slopes. The finite-sample negative-H case is reported as computed with an explanatory note, never clamped — a negative H is evidence the test's conditions failed, and hiding it would misreport. Verified: H = 2.3304 on Grunfeld (Stata: chi2(2) = 2.33). (4) Cluster-robust (by entity) small-sample correction: Stata's xtreg, fe factor c = G/(G−1) · (N−1)/(N−K) applied to the sandwich, with K = intercept + slopes only — for FE the absorbed entity effects are deliberately not charged in this factor. This is the famous xtreg-vs-areg split (areg charges them and gives larger SEs); the xtreg convention is pinned because it is the one the published Grunfeld clustered SEs come from, the one linearmodels (debiased=True, group_debias=True) reproduces, and the one panel users expect. Verified: FE clustered SEs 0.0151945 / 0.0527518 on Grunfeld. (5) t statistics on the residual df for all three estimators — linearmodels' presentation. Stata reports z for RE (GLS asymptotics); the t-based p is the more conservative choice and keeps one presentation across the estimator option. (6) R² definitions are linearmodels' _rsquared: within = 1 − SSR/TSS on entity-demeaned data, between = on entity means, overall = on raw data, each using the fitted model's full coefficient vector. Stata defines between/overall R² as squared correlations, which differ; the 1 − SSR/TSS family is used because it is the one the reference reports. The procedure is computed unweighted by design (mark_unweighted): the within/quasi-demeaning transformations under weight ≡ replication would need every entity mean re-derived under frequency weights, and no reference implements that combination to compare against — the weight selects cases only, and the note says so.
Moderated mediation — six recorded conventions (0.15.0), following the PROCESS (Hayes) definitions throughout so a user comparing against the PROCESS macro sees the same model shapes; the deterministic quantities are pinned against statsmodels OLS (mediation_mod/*). (1) Model shapes are PROCESS models 7 / 14 / 58: the moderator W and its product term enter only the moderated model(s) — moderation=a puts W and X·W in each mediator model and leaves the outcome model alone; moderation=b puts W and every M·W in the outcome model and leaves the mediator models alone; both does both. W is never silently added as a covariate to an unmoderated model. (2) Probe values: a moderator with ≤ 5 distinct values over the listwise sample is treated as categorical and probed at each value; otherwise at the mean − 1 SD / mean / mean + 1 SD (Aiken–West), the SD with the n − 1 denominator, unweighted like the rest of the procedure. Probe values are computed once on the full sample and held fixed across bootstrap resamples. (3) The index of moderated mediation (a₃·b for model 7, a·b₃ for model 14) is reported only when exactly one path is moderated; with both paths moderated the indirect effect is quadratic in W, no single index exists, and the output says so instead — the same choice PROCESS makes for model 58. (4) No total-effect model when a moderator is present: the decomposition c = c′ + ab does not hold conditionally, so a "total" row would misreport; the path-table note states this. (5) Conditional (simple slope) inference from the coefficient covariance: Var(β₁ + w·β₃) = V₁₁ + w²·V₃₃ + 2w·V₁₃, t on the model's residual df — pinned against statsmodels cov_params() (mediation_mod/a_cond_hi_se). (6) Bootstrap CIs are not pinned (seed-dependent), the convention established for mediation v1; the conditional indirect effects and the index reuse v1's exact resampling contract (shared PRNG, quantile rule and BCa points), so they stay byte-reproducible for a fixed seed. When the moderator option is absent the output is byte-identical to v1 (the branch is taken before any moderated code runs), so every existing journaled mediation command replays unchanged.
The 0.16.0 chart repertoire — four chart kinds that add no new statistic, so none is pinned here; what they add is reading conventions, and those are recorded because a user comparing against another package will ask about them. (1) Scatterplot matrix: cases are dropped listwise, not pairwise — a matrix is read by comparing panels, and panels drawn on different case subsets are not comparable — with the dropped count in the note; the diagonal carries the variable name (base R pairs' default, not a density), both triangles are drawn, and the variable count is capped at 8 by the max_cases convention (max_variables, 0 removes it). Panels are laid in the largest centred square of the plot, because a stretched scatter flattens its cloud and would make one relationship look different in different cells. A density on the diagonal and correlations in the upper triangle are deliberate v1 exclusions — the second would put a statistic in a chart tool. (2) ECDF: ties step once, by their combined weight, so the curve is right-continuous; drawing one point per case would ramp through a tie, which is a distribution the data does not have. Weights are frequency weights on the engine-wide weight ≡ replication identity, asserted in graphs.rs's repertoire_tests. Each group's curve is normalized within group, so every curve reaches 1. (3) Cleveland dot plot: categories are sorted by value, descending, by default (sort = value | category) — sorting is half of what makes a dot plot readable, and unlike a bar chart no baseline anchors the eye. The value axis deliberately does not start at zero (that is the whole argument for the form), which is why the leader line spans the full plot width rather than running from the axis to the dot: a partial leader draws a bar from the axis minimum, the exact lie the chart exists to avoid. (4) Bubble chart: area is proportional to the value, r ∝ √size — mapping the value to the radius inflates it by the square, and the rule lives in one exported bubbleRadius with a numeric test on it rather than inline in a paint loop. Non-positive sizes have no area and are dropped, with the count in the note rather than silently.
Complex survey designs — the design declaration, the covered-procedure list, the five estimation conventions and the v1 scope cuts are decided and recorded in their own section above, Complex survey designs (0.19.0 item 1, recorded before any pin — the 0.15.0 rule).
Equivalence tests (TOST) — the tool shape, the test conventions, the SMD-bound standardizers, the two recorded TOSTER divergences and the v1 scope cuts are decided and recorded in their own section above, Equivalence tests (TOST) (0.19.0 item 2, recorded before any pin). The pins ride their own R-backed suite — tool/validation/generate_tost_references.R → test/validation/tost_reference_values.json → test/tost_validation_test.dart (90 values on the sleep documented-example data plus a synthetic unequal-variance block, with the orphan guard) — because the reference implementation is R TOSTER, not a package in the pinned Python set. Degenerate bounds are error paths, not values, so they are pinned as errors-as-data in analysis_test.dart / tool_dispatcher_test.dart.
Bland–Altman method comparison — the difference direction, the fixed 1.96 multiplier, the t-based CI forms (the 1999 exact LoA variance with z = 1.96), the proportional-bias table and the v1 scope cuts are decided and recorded in their own section above, Bland–Altman method comparison (0.19.0 item 3, recorded before any pin). The pins ride their own R-backed suite — tool/validation/generate_bland_altman_references.R → test/validation/bland_altman_reference_values.json → test/bland_altman_validation_test.dart (54 values: dual references on the original Bland & Altman (1986) PEFR data plus a synthetic incomplete pair, with the orphan guard) — because the reference implementations are R blandr and BlandAltmanLeh, not packages in the pinned Python set. The one loosened tolerance: the LoA-CI pins against blandr carry 5e-3 absolute, because blandr computes the 1999 variance with z = Φ⁻¹(0.975) where our recorded convention (and its own lines) use the literal 1.96 — the divergence is recorded in the section above, and the same surfaces are also pinned at 1e-9 against R values computed in the recorded convention (the item-1 "R SEs + our t interval" precedent).
Latent class analysis — the reference (poLCA) and its documented examples, the estimator and starts policy, the fit-block formulas, the descending-π ordering (also the pin alignment rule) and the v1 scope cuts are decided and recorded in their own section above, Latent class analysis (0.19.0 item 4, recorded before any pin; spike-gated). The pins ride their own R-backed suite — tool/validation/generate_lca_references.R → test/validation/lca_reference_values.json → test/lca_validation_test.dart on poLCA's own values and carcinoma example data, with the orphan guard — because the reference implementation is R poLCA, not a package in the pinned Python set. aBIC and entropy are R-computed from poLCA's LL/npar/posterior (poLCA does not print them); the EM optimum is what is pinned, so tolerances are looser than closed-form suites (LL ~1e-6, probabilities ~1e-5).
Item response theory — the dual references (ltm + mirt, the CFA rule) on the canonical LSAT6/LSAT7 data, the Bock–Aitkin quadrature, the θ ~ N(0,1) identification, the reported parameterization and the v1 scope cuts are decided and recorded in their own section above, Item response theory (0.19.0 item 5, recorded before any pin; spike-gated). The pins ride their own R-backed suite — tool/validation/generate_irt_references.R → test/validation/irt_reference_values.json → test/irt_validation_test.dart with the orphan guard — because the reference implementations are R packages, not the pinned Python set. Both packages are pinned separately on the same cells at tolerances set from their own measured disagreement (~8e-4 on parameters, 5e-5 on the LL): a value inside both pins is inside the genuine reference uncertainty.
LOESS smoother (scatterplot) — the exact local-linear tricube algorithm (the q rule with its verified 10⁻⁷ epsilon, tricube over all points inside the q-th-nearest bandwidth, the weighted local line), the 101-point evaluation grid, the degenerate-window fallbacks and the v1 scope cuts are decided and recorded in their own section above, LOESS smoother on the scatterplot (0.19.0 item 6, recorded before any pin). The pins ride their own R-backed suite — tool/validation/generate_loess_references.R → test/validation/loess_reference_values.json → test/loess_validation_test.dart (on the base-R cars data plus seeded synthetic fixtures, with the orphan guard) — because the reference is R stats::loess (degree = 1, surface = "direct", default control), not a package in the pinned Python set. Verified agreement is machine precision, so the pins carry the standard 1e-9; the three recorded reference quirks (lowess defaults, statistics = "exact", singular-window pseudoinverse artifacts) live in the section above.
Permutation tests — the covered tests, the two-sided/observed- direction counting, the automatic exact threshold (1,000,000 arrangements), the (b+1)/(m+1) Monte-Carlo estimator with its 99% Clopper–Pearson CI, the Fisher–Yates draw contract and the v1 scope cuts are decided and recorded in their own section above, Permutation tests (0.19.0 item 7, recorded before any pin). The pins ride their own R-backed suite — tool/validation/generate_permutation_references.R → test/validation/permutation_reference_values.json → test/permutation_validation_test.dart with the orphan guard — because the references are R coin and perm (plus hand enumeration in R where coin's exact algorithms do not reach the correlation problem). Exact p's pin exactly; Monte-Carlo p's pin the exact/reference p at a tolerance derived from the binomial simulation SE at the pinned seed's m (named in the source); bitwise seed reproducibility is asserted engine-side, the bootstrap precedent. Two recorded reference divergences: coin reports the raw b/m estimator (we use the +1 form, perm's convention), and cor.test's exact Spearman is tie-free only (our midrank enumeration extends it; pins stay on tie-free data).
GLM quantile residuals — the per-family residual definitions, the seed policy, the reference lines and the panel names are decided and recorded in their own section above, GLM quantile residuals (0.16.0). The two pinning conventions that belong in this list: (1) Gamma is pinned end to end at 1e-9 — its residual is non-randomized, so Φ⁻¹(F_Γ(y)) is a deterministic function of the fit and SciPy reproduces it exactly. (2) The discrete families are pinned as containment, not as point parity: the residual depends on a PRNG draw no reference package can reproduce, so each glm_resid/* id pins the midpoint of the case's CDF bracket as value with half the bracket width as tol, and the Dart probe returns Φ(r) — the existing value ± tol machinery then expresses exactly the assertion that matters, that the residual's probability integral transform falls inside the bracket SciPy computes. This checks the CDF and the randomization range per case, which a single transcribed number could not. Reproducibility itself is asserted separately, by running the same command twice and requiring the tables to agree exactly.