The ChakataStat dataset format (.ckd)

.ckd is ChakataStat's native document format — what File → Open and File → Save read and write. This document specifies version 2 of the format and records the engineering decisions behind it.

Reference implementation: lib/io/document_codec.dart, exercised by test/document_format_test.dart.

At a glance

A .ckd file is a ZIP archive — the same container approach as .xlsx and .ods — with two members:

example.ckd  (ZIP)
├── meta.json      — variables + provenance (UTF-8 JSON)
└── data.parquet   — the dataset's columns, typed and columnar (Apache Parquet)

• meta.json holds everything ChakataStat knows about the dataset except the cell values: each variable's full metadata, plus document provenance (format/version, timestamps, the writing app and engine). It is human-readable — unzip the file and open it.
• data.parquet holds the cell values as typed, columnar Parquet, written and read by the native engine (over the Apache Arrow / Parquet crates). It is omitted when the dataset has no variables (an empty document is just meta.json).

The file holds data only. Session and application state — the Select / Weight / Split case state, the output log, window layout, the AI conversation — is deliberately not in the document; it is owned by the session store. The document owns what belongs to the dataset; the session owns the rest.

meta.json

{
  "format": "ChakataStat-dataset",
  "version": 2,
  "created": "2026-01-02T03:04:05.000Z",
  "modified": "2026-06-14T09:00:00.000Z",
  "writer": { "app": "0.5.0+6", "engine": "ChakataStat engine 0.5.0 (native)" },
  "rows": 3,
  "variables": [
    {
      "name": "Age",
      "type": "numeric",
      "width": 5,
      "decimals": 0,
      "label": "Age in years",
      "valueLabels": { "99": "Unknown" },
      "missingValues": ["99"],
      "alignment": "right",
      "measure": "scale"
    }
  ]
}

The variables array is the source of truth for the variable list. The Parquet part's column names are ignored on read — display names come from meta.json — which is how duplicate variable names survive (the data part deduplicates them only to satisfy Parquet's unique-name requirement).

data.parquet

One Parquet column per variable, in the same order as meta.json's variables, with the engine's two storage types:

• a numeric variable → an f64 column;
• a string or date variable → a UTF-8 string column.

The data is stored faithfully — and this is the key design point:

A value a variable declares user-missing (e.g. 99 flagged as a missing code) is a real value, so the file stores it as 99, not as null. The missing-value definition lives in meta.json and is reapplied when the file is opened, so analyses still exclude it. Missing-ness is metadata, not a hole in the data.

This is why saving does not reuse the engine's analysis frame (which nulls user-missing values): a dedicated, frame-independent write path (ig_write_parquet_staged) persists a faithful, typed view of the sheet. It is the mirror image of ig_read_parquet, which also leaves the active frame untouched.

What is and isn't preserved

Versioning

The format is versioned and the reader validates it. This build reads version 2 only.

• Version 1 was a single pretty-printed JSON object ({ "format": …, "version": 1, "variables": […], "cells": [[row, col, value], …] }). It is not read by this build: the format was reworked while the project is still pre-1.0, so backward compatibility was traded for a clean design. A v1 file is detected (it begins with {) and rejected with a clear message rather than a cryptic ZIP error.
• A future version 3 would add a migration path here: the reader dispatches on version, so an upgrade reads the old shape and produces the new model.

Compatibility commitment

This is the promise users can rely on for their saved work — what production_readness.md gate 3 (file-format stability) asks for. It applies from 1.0 onward; pre-1.0 is explicitly exempt (which is why version 1 was dropped, above).

For .ckd documents:

1. Forward-readable. Every 1.x release reads every .ckd version ≥ 2 — the version this build writes. A file you save today keeps opening in later 1.x releases.
2. Additive evolution. A new format version is introduced only when the model genuinely grows, and the reader gains a migration path for it (it dispatches on version and upgrades the old shape). A newer version is never made to silently fail on an older file: an unreadable file is rejected with a clear message, never a cryptic error or silent data loss.
3. Retiring a version is a major event. An old version's read support is removed only at a major release (e.g. 2.0), only after it has been readable across the whole prior major line, and the release notes say so. The pre-1.0 → v2 break will not recur within 1.x.
4. Faithful round-trip. Within these rules the documented round-trip guarantees (the What is and isn't preserved table) hold across versions.

For the session / recovery file (session_state.dart): best-effort and forward-tolerant, not a hard guarantee. It holds recoverable application state, not user documents, so an unreadable or newer-than-expected session falls back to a clean start and never blocks launch. Older sessions load with newer sections defaulted (the reader is deliberately lenient).

How this is enforced. A committed golden file — test/fixtures/golden_v2.ckd, a real v2 file written by an earlier build — is opened and checked on every run by test/format_compat_test.dart, alongside an old-version session fixture. A change that breaks reading a prior file fails that test. (The in-build round-trip tests in test/document_format_test.dart prove self-consistency; the golden file proves cross-build compatibility.) Regenerate the fixture only on a deliberate format change:

flutter test --dart-define=GENERATE_FIXTURES=true test/format_compat_test.dart

Determinism

The format avoids gratuitous nondeterminism so a content-identical re-save is as reproducible as the toolchain allows:

• ZIP entries use a fixed modification time (the real timestamps live in meta.json), so the archive structure does not churn.
• created is stable across saves; only modified advances.

meta.json is fully deterministic for identical content. The Parquet part's exact bytes are the engine's (the Parquet writer's) to define and are not guaranteed byte-stable — a property intentionally not relied upon (the app tracks unsaved changes in memory, not by hashing files).

Why these choices

• ZIP container, not single-file. A data-analysis document wants typed, columnar, compressed storage that loads straight into the engine. Parquet is exactly that, and the engine already reads and writes it. Variable metadata doesn't fit in Parquet columns; rather than depend on Parquet footer key-value metadata (which the engine's Parquet write path deliberately does not use) and the fragile low-level code needed to write it, the metadata rides in a sibling meta.json. The container reuses stable, tested engine endpoints with no changes to the Parquet writer — the robust choice for a format we want to get right once.
• Opening requires the native engine. Reading the Parquet part goes through the engine (FFI). That is acceptable: the engine is already a hard requirement for the app to run at all.
• Data only; state lives elsewhere. Keeping the document to data (and data-about-the-data: timestamps, writer) keeps a clean boundary with the session store, which already owns application and session state.

Engine endpoints