ChakataStat
Download

Regenerating the reference values

Part of the numerical-validation suite. The generator, the pinned reference environment, and what to do when a reference package changes.

The reference toolchain is needed only to author the values — CI just reads the committed JSON (so the suite runs everywhere the Dart suite runs, with no Python dependency on the CI machines). To regenerate after adding a procedure or changing a number:

python3 -m venv /tmp/refenv
/tmp/refenv/bin/pip install -r tool/validation/requirements.txt
/tmp/refenv/bin/python tool/validation/generate_references.py
dart run tool/cstool.dart test test/numerical_validation_test.dart

Install from the lockfile, never by name. Until 2026-08-23 this page said pip install scipy statsmodels …, which resolves to whatever is newest that day — so the same command on two machines could produce two different reference values and neither would be wrong. tool/validation/requirements.txt pins the exact versions that produced the committed numbers.

The versions are also recorded in the JSON header (_versions) by the generator itself, and test/validation_environment_test.dart asserts that the header and the lockfile agree — so regenerating against a newer package without updating the pin fails the bar, and so does bumping the pin without regenerating. Two packages (pandas, semopy) were not recorded before 2026-08-23 and are therefore not yet pinned; the generator records them now, so the next full regen closes that gap.

For the fixture generators (the foreign-format corpus and the spreadsheet fixtures) use requirements-fixtures.txt instead — those are deliberately unpinned, because they write committed fixtures rather than compute reference values.

The survey-design, TOST, Bland–Altman, LCA, IRT, LOESS, permutation, CFA follow-up and mediation follow-up references have their own toolchain — R with the survey package (the canonical reference implementation for Taylor-linearized design-based estimation; none of the pinned Python packages implements it), the TOSTER package (the reference for equivalence testing), the blandr + BlandAltmanLeh pair (the dual references for method comparison; blandr also ships the original Bland & Altman 1986 PEFR data the fixture is written from), the poLCA package (the reference for latent class analysis; it ships the values and carcinoma example data the fixtures are written from), the ltm + mirt pair (the dual references for IRT; they ship the LSAT6/LSAT7 data), the coin + perm pair (the dual references for permutation tests, with hand enumeration in R where coin's exact algorithms do not reach), and — for the LOESS smoother — base R alone (stats::loess is the reference; the cars fixture is a base-R dataset), and — for the CFA follow-ups — lavaan (the reference SEM package; it ships the Holzinger–Swineford data the fixture is written from), and — for the mediation follow-ups — base R lm plus the interactions package (the Johnson–Neyman reference). Like the Python values, CI only reads the committed JSON:

# Pin the R side first — Rscript tool/validation/r-packages.R installs the
# exact package versions the committed values were produced under (R 4.5.3),
# and the same guard test checks that script against the JSON headers. The
# floors below are what the scripts *need*; the pins are what they were *run*
# with, and only the pins reproduce a committed number.
#
# any R ≥ 4 with survey ≥ 4.4, TOSTER ≥ 0.8, blandr ≥ 0.6,
# BlandAltmanLeh ≥ 0.3, poLCA ≥ 1.6, ltm ≥ 1.2, mirt ≥ 1.4,
# coin ≥ 1.4 and perm ≥ 1.0; a user-space conda-forge R works:
#   micromamba create -p ~/micromamba/envs/r-survey -c conda-forge \
#     r-base r-survey r-toster r-polca r-coin r-lavaan \
#     c-compiler cxx-compiler gfortran make
#   # CRAN source builds need the env's bin on PATH for its compilers:
#   PATH=~/micromamba/envs/r-survey/bin:$PATH Rscript -e \
#     'install.packages(c("blandr","BlandAltmanLeh","ltm","mirt","perm","interactions"),
#      repos="https://cloud.r-project.org")'
Rscript tool/validation/generate_survey_references.R
Rscript tool/validation/generate_tost_references.R
Rscript tool/validation/generate_bland_altman_references.R
Rscript tool/validation/generate_lca_references.R
Rscript tool/validation/generate_irt_references.R
Rscript tool/validation/generate_loess_references.R
Rscript tool/validation/generate_permutation_references.R
Rscript tool/validation/generate_cfa_followups_references.R
Rscript tool/validation/generate_mediation_followups_references.R
dart run tool/cstool.dart test test/survey_validation_test.dart \
  test/tost_validation_test.dart test/bland_altman_validation_test.dart \
  test/lca_validation_test.dart test/irt_validation_test.dart \
  test/loess_validation_test.dart test/permutation_validation_test.dart \
  test/cfa_followups_validation_test.dart \
  test/mediation_followups_validation_test.dart

The R and package versions used for the committed values are recorded in each JSON header (_versions).

The second oracle for the Python-referenced pins (0.23.0 item 8 — see validation.md) runs in the same pinned R environment plus metafor, ivreg, plm and MatchIt (installed from CRAN into it; MASS, nnet, survival, car, psych, lavaan and urca are already there):

PATH=~/micromamba/envs/r-survey/bin:$PATH Rscript -e \
  'install.packages(c("metafor","ivreg","plm","MatchIt"), repos="https://cloud.r-project.org")'
Rscript tool/validation/generate_second_oracle_references.R
dart run tool/cstool.dart test test/numerical_validation_test.dart

The generator prints the between-oracle agreement per family and lists any disagreement past a pin's tolerance that has no recorded divergence; the Dart suite fails on the same condition. Regenerate it whenever reference_values.json changes — a pin added to a covered family without an R value is a pin the second-oracle guard no longer sees, and a pin whose Python value moved past its R value fails as an unexplained disagreement.

The statistics-package format artifacts have their own, smaller toolchain (pyreadstat + pandas, and xlwt/odfpy for the spreadsheet fixtures):

python3 -m venv /tmp/fmtenv
/tmp/fmtenv/bin/pip install pyreadstat pandas
/tmp/fmtenv/bin/python tool/validation/generate_foreign_fixtures.py   # create-if-absent
/tmp/fmtenv/bin/python tool/validation/generate_foreign_corpus.py     # rewrites expected.json
dart run tool/cstool.dart test test/io_test.dart test/foreign_corpus_test.dart

The fixture writer is deliberately create-if-absent: .sav files embed a creation timestamp, so an unconditional rewrite would churn committed bytes on every run. Delete a fixture to genuinely regenerate it.

Regenerating the cross-OS agreement report

The cross-OS page is generated, never edited: a dispatched tier-2 run (gh workflow run ci) has every validation suite emit each pin's engine-computed value (VALIDATION_EMIT_PINS, set by the workflow; test/validation/pin_emission.dart) as one pins-<OS> artifact per leg, and the cross-os-agreement job scores them with tool/validation/cross_os_agreement.dart and uploads the Markdown as the cross-os-agreement artifact. To refresh the committed copy from a run:

gh run download <run-id> --pattern 'pins-*' --dir qa_output/validation/agreement
dart run tool/validation/cross_os_agreement.dart \
  --run-id <run-id> --sha <sha> \
  qa_output/validation/agreement/pins-Linux \
  qa_output/validation/agreement/pins-Windows \
  qa_output/validation/agreement/pins-macOS

The script exits non-zero when any pin's cross-OS spread exceeds its tolerance — that is a finding to fix or record, not a reason to rerun. The committed page must come from the release candidate's run, so the published table describes the pins that ship.

Adding a procedure

  1. In the generator, add a add("<family>/<stat>", value, tol, "<source>") line computing the statistic with a reference package; re-run it.
  2. In the Dart test, add a Probe (or an entry to an existing one) mapping each new id to the engine request and the result cell that holds it.
  3. Run the test. The orphan-guard will tell you immediately if the two sides are not in step.

When the reference disagrees, decide whether it is a real engine bug or a convention difference (a different-but-valid definition). If the latter, compute the reference in the engine's convention and record the choice in the Tolerances and recorded conventions list above — never just loosen the tolerance to make a genuine discrepancy pass.