ChakataStat
Download

Regenerating the reference values

Part of the numerical-validation suite. The generator, the pinned reference environment, and what to do when a reference package changes.

The reference toolchain is needed only to author the values — CI just reads the committed JSON (so the suite runs everywhere the Dart suite runs, with no Python dependency on the CI machines). To regenerate after adding a procedure or changing a number:

python3 -m venv /tmp/refenv
/tmp/refenv/bin/pip install scipy statsmodels scikit-learn lifelines pingouin
/tmp/refenv/bin/python tool/validation/generate_references.py
dart run tool/cstool.dart test test/numerical_validation_test.dart

The package versions used for the committed values are recorded in the JSON header (_versions) — at the time of writing SciPy 1.18, statsmodels 0.14, scikit-learn 1.9, lifelines 0.30, pingouin 0.6, NumPy 2.5.

The survey-design, TOST, Bland–Altman, LCA, IRT, LOESS, permutation, CFA follow-up and mediation follow-up references have their own toolchain — R with the survey package (the canonical reference implementation for Taylor-linearized design-based estimation; none of the pinned Python packages implements it), the TOSTER package (the reference for equivalence testing), the blandr + BlandAltmanLeh pair (the dual references for method comparison; blandr also ships the original Bland & Altman 1986 PEFR data the fixture is written from), the poLCA package (the reference for latent class analysis; it ships the values and carcinoma example data the fixtures are written from), the ltm + mirt pair (the dual references for IRT; they ship the LSAT6/LSAT7 data), the coin + perm pair (the dual references for permutation tests, with hand enumeration in R where coin's exact algorithms do not reach), and — for the LOESS smoother — base R alone (stats::loess is the reference; the cars fixture is a base-R dataset), and — for the CFA follow-ups — lavaan (the reference SEM package; it ships the Holzinger–Swineford data the fixture is written from), and — for the mediation follow-ups — base R lm plus the interactions package (the Johnson–Neyman reference). Like the Python values, CI only reads the committed JSON:

# any R ≥ 4 with survey ≥ 4.4, TOSTER ≥ 0.8, blandr ≥ 0.6,
# BlandAltmanLeh ≥ 0.3, poLCA ≥ 1.6, ltm ≥ 1.2, mirt ≥ 1.4,
# coin ≥ 1.4 and perm ≥ 1.0; a user-space conda-forge R works:
#   micromamba create -p ~/micromamba/envs/r-survey -c conda-forge \
#     r-base r-survey r-toster r-polca r-coin r-lavaan \
#     c-compiler cxx-compiler gfortran make
#   # CRAN source builds need the env's bin on PATH for its compilers:
#   PATH=~/micromamba/envs/r-survey/bin:$PATH Rscript -e \
#     'install.packages(c("blandr","BlandAltmanLeh","ltm","mirt","perm","interactions"),
#      repos="https://cloud.r-project.org")'
Rscript tool/validation/generate_survey_references.R
Rscript tool/validation/generate_tost_references.R
Rscript tool/validation/generate_bland_altman_references.R
Rscript tool/validation/generate_lca_references.R
Rscript tool/validation/generate_irt_references.R
Rscript tool/validation/generate_loess_references.R
Rscript tool/validation/generate_permutation_references.R
Rscript tool/validation/generate_cfa_followups_references.R
Rscript tool/validation/generate_mediation_followups_references.R
dart run tool/cstool.dart test test/survey_validation_test.dart \
  test/tost_validation_test.dart test/bland_altman_validation_test.dart \
  test/lca_validation_test.dart test/irt_validation_test.dart \
  test/loess_validation_test.dart test/permutation_validation_test.dart \
  test/cfa_followups_validation_test.dart \
  test/mediation_followups_validation_test.dart

The R and package versions used for the committed values are recorded in each JSON header (_versions).

The statistics-package format artifacts have their own, smaller toolchain (pyreadstat + pandas, and xlwt/odfpy for the spreadsheet fixtures):

python3 -m venv /tmp/fmtenv
/tmp/fmtenv/bin/pip install pyreadstat pandas
/tmp/fmtenv/bin/python tool/validation/generate_foreign_fixtures.py   # create-if-absent
/tmp/fmtenv/bin/python tool/validation/generate_foreign_corpus.py     # rewrites expected.json
dart run tool/cstool.dart test test/io_test.dart test/foreign_corpus_test.dart

The fixture writer is deliberately create-if-absent: .sav files embed a creation timestamp, so an unconditional rewrite would churn committed bytes on every run. Delete a fixture to genuinely regenerate it.

Adding a procedure

  1. In the generator, add a add("<family>/<stat>", value, tol, "<source>") line computing the statistic with a reference package; re-run it.
  2. In the Dart test, add a Probe (or an entry to an existing one) mapping each new id to the engine request and the result cell that holds it.
  3. Run the test. The orphan-guard will tell you immediately if the two sides are not in step.

When the reference disagrees, decide whether it is a real engine bug or a convention difference (a different-but-valid definition). If the latter, compute the reference in the engine's convention and record the choice in the Tolerances and recorded conventions list above — never just loosen the tolerance to make a genuine discrepancy pass.