Cross-OS agreement
Part of the numerical-validation suite. What the three operating systems' engines compute relative to each other, per pinned statistic — the statement the tolerances cannot make.
The validation suites hold every pin within its tolerance of the reference on each OS, and stop there: a pin that passes on one OS at one value and on another at a slightly different value is invisible, because the tolerance swallows the difference. This report makes the difference visible. With VALIDATION_EMIT_PINS set, every suite also writes the value the engine actually computed; the tier-2 matrix uploads one file set per OS and a job after it puts them side by side. Per pin, the score is the number of significant digits on which every OS agrees — −log₁₀(spread / max |value|), capped at 16, the double limit — so 16 means bit-identical. A spread wider than the pin's own tolerance is a finding: a real platform difference to fix, or a convention to record.
This run
| Engine | ChakataStat 0.23.0+24 |
| Commit | 17da57b |
| CI run | 33361732495 |
| Generated | 2026-08-31 |
| Linux | Linux 6.17.0-1022-azure #22-Ubuntu SMP Mon Jul 27 17:24:03 UTC 2026 |
| Windows | "Windows Server 2025 Datacenter" 10.0 (Build 26100) |
| macOS | Version 26.5.2 (Build 25F84) |
Summary
- 821 pins compared across 3 operating systems (821 emitted, of 1088 in the committed reference files — counted from
test/validation/*reference_values.json; the two counts differ). - 615 bit-identical on every OS (74.9 %).
- 206 differ in the last digits; the least agreement is 6.1 digits (
cfa_mi/x8_x9/mi). - 0 pins with a cross-OS spread wider than their tolerance — no findings.
Per family
The family is the pin id's prefix (the procedure), across all reference files.
| Family | Pins | Bit-identical | Min digits | Over tolerance |
|---|---|---|---|---|
acf |
3 | 3 | 16 (identical) | — |
adf |
2 | 2 | 16 (identical) | — |
anova |
5 | 5 | 16 (identical) | — |
arima |
1 | 1 | 16 (identical) | — |
ba_pefr1 |
14 | 14 | 16 (identical) | — |
ba_pefr1_c90 |
4 | 4 | 16 (identical) | — |
ba_pefr1_prop |
8 | 8 | 16 (identical) | — |
ba_pefr2 |
6 | 6 | 16 (identical) | — |
ba_synth |
14 | 14 | 16 (identical) | — |
ba_synth_prop |
8 | 8 | 16 (identical) | — |
binomial |
1 | 1 | 16 (identical) | — |
cfa |
27 | 0 | 6.8 | — |
cfa_mg |
58 | 7 | 6.6 | — |
cfa_mi |
18 | 0 | 6.1 | — |
cfa_mi_orth |
12 | 1 | 15.0 | — |
cfa_wlsmv |
25 | 7 | 6.8 | — |
correlate |
6 | 6 | 16 (identical) | — |
crosstabs |
5 | 5 | 16 (identical) | — |
discriminant |
3 | 3 | 16 (identical) | — |
ets |
2 | 2 | 16 (identical) | — |
explore |
10 | 10 | 16 (identical) | — |
factor |
4 | 1 | 15.5 | — |
friedman |
2 | 2 | 16 (identical) | — |
glm |
4 | 4 | 16 (identical) | — |
glm_resid |
17 | 14 | 14.2 | — |
granger |
4 | 4 | 16 (identical) | — |
hier |
1 | 1 | 16 (identical) | — |
irt_lsat6_2pl |
2 | 2 | 16 (identical) | — |
irt_lsat6_rasch |
15 | 6 | 7.4 | — |
irt_lsat7_2pl |
42 | 2 | 6.5 | — |
iv |
6 | 6 | 16 (identical) | — |
kpss |
1 | 1 | 16 (identical) | — |
kruskal |
2 | 2 | 16 (identical) | — |
lca_carcinoma2 |
1 | 1 | 16 (identical) | — |
lca_carcinoma3 |
11 | 10 | 13.2 | — |
lca_carcinoma4 |
1 | 0 | 15.1 | — |
lca_values1 |
2 | 2 | 16 (identical) | — |
lca_values2 |
13 | 13 | 16 (identical) | — |
levene |
2 | 2 | 16 (identical) | — |
loess_cars100 |
7 | 7 | 16 (identical) | — |
loess_cars40 |
7 | 7 | 16 (identical) | — |
loess_cars75 |
9 | 9 | 16 (identical) | — |
loess_sine30 |
7 | 7 | 16 (identical) | — |
loess_sine75 |
9 | 9 | 16 (identical) | — |
mann_whitney |
4 | 4 | 16 (identical) | — |
manova |
3 | 3 | 16 (identical) | — |
means |
2 | 2 | 16 (identical) | — |
med_3w |
11 | 11 | 16 (identical) | — |
med_jn |
8 | 8 | 16 (identical) | — |
med_serial |
11 | 11 | 16 (identical) | — |
mediation |
7 | 7 | 16 (identical) | — |
mediation_mod |
14 | 14 | 16 (identical) | — |
meta_analysis |
15 | 15 | 16 (identical) | — |
multinomial |
3 | 0 | 15.1 | — |
nls |
5 | 0 | 8.1 | — |
normality |
3 | 3 | 16 (identical) | — |
ordinal |
2 | 1 | 15.0 | — |
pacf |
1 | 1 | 16 (identical) | — |
panel |
27 | 27 | 16 (identical) | — |
partial |
1 | 1 | 16 (identical) | — |
perm_corr_exact |
3 | 3 | 16 (identical) | — |
perm_mc_corr |
2 | 2 | 16 (identical) | — |
perm_mc_ts |
2 | 2 | 16 (identical) | — |
perm_ts_exact |
3 | 3 | 16 (identical) | — |
ph_test |
3 | 3 | 16 (identical) | — |
propensity |
8 | 8 | 16 (identical) | — |
reg_gamma |
6 | 5 | 15.3 | — |
reg_linear |
16 | 16 | 16 (identical) | — |
reg_logistic |
4 | 0 | 14.3 | — |
reg_negbin |
2 | 2 | 16 (identical) | — |
reg_poisson |
3 | 3 | 16 (identical) | — |
reg_probit |
2 | 2 | 16 (identical) | — |
reliability |
1 | 1 | 16 (identical) | — |
rm_anova |
3 | 3 | 16 (identical) | — |
runs |
2 | 2 | 16 (identical) | — |
sign |
1 | 1 | 16 (identical) | — |
survey |
1 | 1 | 16 (identical) | — |
survival_cox |
2 | 2 | 16 (identical) | — |
survival_km |
2 | 2 | 16 (identical) | — |
svy_chisq |
8 | 8 | 16 (identical) | — |
svy_explore |
14 | 14 | 16 (identical) | — |
svy_freq |
22 | 22 | 16 (identical) | — |
svy_linreg |
11 | 11 | 16 (identical) | — |
svy_logreg |
11 | 1 | 12.8 | — |
svy_means |
8 | 8 | 16 (identical) | — |
tost_ind_sleep_a01 |
2 | 2 | 16 (identical) | — |
tost_ind_sleep_pooled |
7 | 7 | 16 (identical) | — |
tost_ind_sleep_welch |
9 | 9 | 16 (identical) | — |
tost_ind_y_asym |
7 | 7 | 16 (identical) | — |
tost_ind_y_pooled |
7 | 7 | 16 (identical) | — |
tost_ind_y_smd_pooled |
8 | 8 | 16 (identical) | — |
tost_ind_y_smd_welch |
8 | 8 | 16 (identical) | — |
tost_ind_y_welch |
7 | 7 | 16 (identical) | — |
tost_one_score |
9 | 9 | 16 (identical) | — |
tost_paired_prepost |
9 | 9 | 16 (identical) | — |
tost_paired_prepost_smd |
8 | 8 | 16 (identical) | — |
tost_paired_sleep |
9 | 9 | 16 (identical) | — |
ttest_ind |
7 | 7 | 16 (identical) | — |
ttest_one |
4 | 4 | 16 (identical) | — |
ttest_paired |
4 | 4 | 16 (identical) | — |
var1both |
3 | 3 | 16 (identical) | — |
var2 |
33 | 33 | 16 (identical) | — |
varselect |
16 | 16 | 16 (identical) | — |
violin |
4 | 4 | 16 (identical) | — |
wilcoxon |
2 | 2 | 16 (identical) | — |
Least agreement — the 30 worst pins
Every pin that is not bit-identical, least agreement first (the full list is in the run's cross-os-agreement artifact).
| Pin | Linux | Windows | macOS | Digits | Spread | Tolerance |
|---|---|---|---|---|---|---|
cfa_mi/x8_x9/mi |
14.946343697690370 | 14.946356831920207 | 14.946343697690494 | 6.1 | 0.000013134229837064026 | 0.0020000000000000000 |
cfa_mi/x8_x9/sepc |
-0.80520077044417204 | -0.80520122648552750 | -0.80520077044417560 | 6.2 | 4.5604135545662672e-7 | 0.00020000000000000001 |
cfa_mi/x8_x9/epc |
-0.42309501766166863 | -0.42309523645273617 | -0.42309501766167040 | 6.3 | 2.1879106754107980e-7 | 0.00020000000000000001 |
irt_lsat7_2pl/se_a1_ltm |
0.17719757137360106 | 0.17719760171457649 | 0.17719754859463796 | 6.5 | 5.3119938531187572e-8 | 0.0020000000000000000 |
irt_lsat7_2pl/se_a1_mirt |
0.17719757137360106 | 0.17719760171457649 | 0.17719754859463796 | 6.5 | 5.3119938531187572e-8 | 0.0020000000000000000 |
cfa_mi/textual_x3/mi |
9.1509138229906455 | 9.1509113147593713 | 9.1509138229906313 | 6.6 | 0.0000025082312742341628 | 0.0020000000000000000 |
irt_lsat7_2pl/se_b1_ltm |
0.26395969823941617 | 0.26395973837821002 | 0.26395966627950135 | 6.6 | 7.2098708669976475e-8 | 0.0020000000000000000 |
irt_lsat7_2pl/se_b1_mirt |
0.26395969823941617 | 0.26395973837821002 | 0.26395966627950135 | 6.6 | 7.2098708669976475e-8 | 0.0020000000000000000 |
cfa_mg/scalar/mean_visual_g2 |
0.14770040013931324 | 0.14770040013931290 | 0.14770036191084618 | 6.6 | 3.8228467058587512e-8 | 0.0010000000000000000 |
irt_lsat7_2pl/se_a2_ltm |
0.16876531801411210 | 0.16876535591960456 | 0.16876534529576920 | 6.6 | 3.7905492467071866e-8 | 0.0020000000000000000 |
irt_lsat7_2pl/se_a2_mirt |
0.16876531801411210 | 0.16876535591960456 | 0.16876534529576920 | 6.6 | 3.7905492467071866e-8 | 0.0020000000000000000 |
irt_lsat7_2pl/se_a5_ltm |
0.15113490923265327 | 0.15113491177615146 | 0.15113488000347852 | 6.7 | 3.1772672937080415e-8 | 0.0020000000000000000 |
irt_lsat7_2pl/se_a5_mirt |
0.15113490923265327 | 0.15113491177615146 | 0.15113488000347852 | 6.7 | 3.1772672937080415e-8 | 0.0020000000000000000 |
irt_lsat7_2pl/se_b5_ltm |
0.44624770766042177 | 0.44624771505661803 | 0.44624762925720990 | 6.7 | 8.5799408133802046e-8 | 0.0020000000000000000 |
irt_lsat7_2pl/se_b5_mirt |
0.44624770766042177 | 0.44624771505661803 | 0.44624762925720990 | 6.7 | 8.5799408133802046e-8 | 0.0020000000000000000 |
cfa_mi/x7_x8/mi |
34.145103772023667 | 34.145097398918985 | 34.145103772023703 | 6.7 | 0.0000063731047177384426 | 0.0020000000000000000 |
cfa_mg/scalar/mean_speed_g2 |
0.17735046158913068 | 0.17735046158913062 | 0.17735049372484546 | 6.7 | 3.2135714839176543e-8 | 0.0010000000000000000 |
cfa_mi/textual_x3/epc |
-0.27163800620144091 | -0.27163796166546739 | -0.27163800620144046 | 6.8 | 4.4535973520165584e-8 | 0.00020000000000000001 |
cfa_mi/textual_x3/sepc |
-0.23809967934719048 | -0.23809964045977816 | -0.23809967934718998 | 6.8 | 3.8887412318056036e-8 | 0.00020000000000000001 |
cfa_wlsmv/pvalue |
5.0513700217001149e-7 | 5.0513697426506437e-7 | 5.0513705365208724e-7 | 6.8 | 7.9387022875062295e-14 | 0.000010000000000000001 |
cfa/se_x9 |
0.15116740663940953 | 0.15116738527322804 | 0.15116740663940953 | 6.8 | 2.1366181490733993e-8 | 0.00050000000000000001 |
cfa_mi/x7_x8/sepc |
0.85915053971163624 | 0.85915064860832924 | 0.85915053971163668 | 6.9 | 1.0889669299629645e-7 | 0.00020000000000000001 |
cfa/loading_x2 |
0.55350018387989697 | 0.55350024425832622 | 0.55350018387989663 | 7.0 | 6.0378429589391658e-8 | 0.00010000000000000000 |
irt_lsat7_2pl/se_a4_ltm |
0.13412180377537150 | 0.13412181832283920 | 0.13412180852691874 | 7.0 | 1.4547467702685779e-8 | 0.0020000000000000000 |
irt_lsat7_2pl/se_a4_mirt |
0.13412180377537150 | 0.13412181832283920 | 0.13412180852691874 | 7.0 | 1.4547467702685779e-8 | 0.0020000000000000000 |
cfa_mi/visual_x7/sepc |
-0.34890933073881969 | -0.34890929333236415 | -0.34890933073881908 | 7.0 | 3.7406455544175543e-8 | 0.00020000000000000001 |
cfa_mi/visual_x7/epc |
-0.42186322303520352 | -0.42186318346767709 | -0.42186322303520279 | 7.0 | 3.9567526421979693e-8 | 0.00020000000000000001 |
cfa_mi/visual_x9/epc |
0.57702198573054364 | 0.57702193561215742 | 0.57702198573054364 | 7.1 | 5.0118386218933608e-8 | 0.00020000000000000001 |
cfa_mi/visual_x7/mi |
18.630678864592728 | 18.630677341450930 | 18.630678864592696 | 7.1 | 0.0000015231417975769546 | 0.0020000000000000000 |
cfa_mi/visual_x9/sepc |
0.51524928705783413 | 0.51524924566101549 | 0.51524928705783424 | 7.1 | 4.1396818750705222e-8 | 0.00020000000000000001 |