Statistics-package file formats

Part of the numerical-validation suite. How the SPSS, Stata and SAS readers are proven against the packages that wrote the files.

Added 2026-08-06 (release 0.14.0 item 2); the .sas7bdat leg 2026-08-21 (0.21.0 item 7). Not a numerical gate — no statistic is computed here — but the same discipline applied to a different kind of correctness: does a .sav, .dta, .xpt or .sas7bdat arrive as the file actually says?

The readers (rust/src/foreign.rs) are ambers (SPSS), dta (Stata), sas_xport (SAS transport) and sas7bdat (SAS data files and .sas7bcat catalogs) — young, single-author, pure-Rust crates. That is a deliberate trade against vendoring the mature ReadStat C library, and the evidence below is what makes it defensible.

The corpus rule (standing policy, restated 0.21.0 item 8a). A real-world foreign file that misbehaves in one of these readers lands in test/fixtures/foreign/corpus/ with its fix, and expected.json is regenerated (generate_foreign_corpus.py), so every field quirk becomes a permanent per-cell differential test. Fix the reader, grow the corpus, one commit. The rule is also at the top of foreign.rs (where the fixer is) and in the corpus README (where the files are). Growth so far: the first sweep (0.14.0, 11 files), the strL file (0.18.0), and the seven .sas7bdat files with two catalogs (0.21.0 item 7).

Two artifacts, the same shape as the numerical suite:

Both generators corroborate before committing: the fixture writer reads its own output back and asserts the writer stored what was intended, and the corpus generator refuses to write ground truth it cannot verify (see the .xpt note below). Fixtures are create-if-absent, so a regeneration cannot silently churn committed bytes.

Why pyreadstat is a valid oracle here — and was not going to be

This is the one piece of reasoning worth preserving, because it inverted when the implementation route changed:

• Under the original plan (vendor ReadStat), pyreadstat, R's haven and pandas.read_spss would all have been the same implementation as the thing under test — they are ReadStat wrappers. Comparing against them would have proven only that the library was called correctly, and could not have caught a ReadStat bug. The plan therefore specified GNU PSPP and IBM's SPSS Statistics Data File Driver as the oracles carrying independent evidence.
• Under the shipped route (pure-Rust readers), pyreadstat shares no code with the readers under test. It is now a genuinely independent second implementation, and every corpus row is the cross-implementation check the plan wanted PSPP for.

So PSPP and the IBM driver are optional corroboration, not a gate. Standing them up would add a third opinion rather than the first independent one. What would change this: adopting a ReadStat-based reader after all, at which point the original argument returns in full and an independent oracle becomes mandatory again.

.xpt and .sas7bdat go one step further. Their corpus entries are cross-checked against pandas.read_sas — a native Python implementation independent of both ReadStat and our reader — and the generator fails rather than writing ground truth the two oracles disagree on. pandas.read_sas does not interpret SAS display formats for .xpt, and for .sas7bdat interprets a different set of them than ReadStat (it turns the partial-date formats into dates too), so any column either oracle rendered temporally is excluded from that check and pinned against our own reader instead.

.sas7bdat has no open-source writer (pyreadstat, pandas and haven all read it only), so there is no pyreadstat-written demo.sas7bdat: the Rust unit tests (ffi.rs) pin the committed corpus files directly, and the stress slot reads one committed real file unscaled (recorded in its report as such). The .sas7bdat conventions, decided before the pins were written:

• One SAS table for both SAS readers. .sas7bdat renders dates, datetimes and times through the same format-stem table as .xpt (sas_render in foreign.rs), not the crate's own classification — the crate would type the partial-date families (MMYY, YEAR, MONYY, …) as dates, where pyreadstat (and the .xpt leg) leave them as the stored number with a warning. The reader overrides every numeric column to "stored number" and renders itself, so one value renders identically from either file.
• Fractional seconds truncate toward the earlier second (sas_whole): 02:16:02.88 is in second 02:16:02, which is how pyreadstat renders it; a value within 1e-6 of a whole number snaps to it first (XPORT's IBM hexadecimal floats can land a hair off). Shared by both SAS readers.
• Every SAS missing is blank — . and the tagged .A–.Z, ._ — matching pyreadstat's default read; a catalog label attached to a tagged missing is not carried and is named in a warning.
• Value labels come only from a catalog beside the file: name.sas7bcat first, else formats.sas7bcat (SAS's default library catalog name, what a folder exported whole from SAS contains). A column gains labels when its format names one of the catalog's label sets, keys rendered as the cells render. A user-defined format with no label set is named in a warning — with the file name to place when there is no catalog, or the catalog that does not define it when there is. A catalog that fails to parse is a warning, not a failed import.

Recorded divergences

Where our reader and pyreadstat legitimately disagree, the app's behaviour is pinned in test/foreign_corpus_test.dart with its reason, exactly as a convention choice is recorded above — never loosened away:

• Truncated multibyte strings (tegulu.sav): the file's Telugu text is cut mid-codepoint at the SPSS string width. pyreadstat drops the partial character; we surface U+FFFD. Neither invents data; ours shows something was cut.
• SPSS QYR/MOYR/WKYR display formats (simple_alltypes.sav): pyreadstat leaves raw epoch seconds, we render real dates. Ours is the more useful behaviour, so it is kept and asserted date-shaped rather than exact-pinned (the exact rendering is upstream-formatting territory).

Known limits, surfaced not silent

Each of these is reported in the import's warnings — to stderr headless, to the import summary in the GUI — so a user never has to infer them (production_readness.md gate 1):