Working with Data

Everything in ChakataStat starts with a dataset. This page covers getting data in and out, the two ways you look at a dataset (Data View and Variable View), the native .ckd document, and the tools for tidying a dataset up.

The shape of a dataset

A dataset is a rectangle: rows are cases, columns are variables. Behind the scenes a variable holds one of three storage types:

• Numeric: numbers; a blank or non-number is missing.
• String: text (imported timestamps stay here, as ISO text).
• Date: a calendar date, stored as ISO YYYY-MM-DD text. Typing into a date column is validated: 2024-02-29 and the compact 20240229 are accepted (and stored in the dashed form), while 1/2/2024, an impossible date like 2023-02-29, or a time-of-day is refused. Dates are never guessed from a regional format. Because the stored form is fixed-width ISO text, sorting and Select Cases comparisons on a date variable are chronological. Cells that don't conform (for example after re-typing a text column to Date) are never rewritten: the Variable View marks the variable with a warning badge and analyses over it carry a one-line note.

On top of storage sits each variable's definition (label, value labels, measurement level, missing rules). That lives in the Variable View and travels with the data inside an .ckd file.

Importing data

File → Import Data… reads the formats below. The Carries metadata column is the one to notice: those formats bring variable labels, value labels and more across with the data, so you do not re-type in the Variable View what the file already knew.

• CSV / TSV: the delimiter is sniffed from the content: comma, semicolon, tab or pipe, with quoted fields respected (so a ;-file full of quoted commas still reads correctly; .tsv always reads as tab). The header row is detected, and each column's type is inferred (all-numeric → numeric, otherwise string). An import-options dialog lets you confirm before loading. A gzipped text file (.csv.gz, .tsv.gz, .json.gz, …) is decompressed and read the same way.
• Fixed-width text (.fwf, .prn): column-aligned text with no delimiter (mainframe extracts, printed reports). The one thing no sniffer can know is where the columns break, so a small dialog asks you: the first lines of the file are shown in a monospaced preview under a clickable ruler; click to place or remove a column break (▼), with initial breaks suggested from the runs of spaces every sampled line shares. The same spec is editable as text beside the preview (start-end ranges, 1-based and inclusive, e.g. 1-8,9-12,13-20), and the two stay in step; ranges may skip over junk columns. Once split, columns type-infer exactly as CSV's do. Headless (batch or MCP) passes the same spec with --fwf-spec.
• Excel (.xlsx), legacy .xls and OpenDocument .ods: the first sheet, read by the native engine, with the same header prompt and type inference. A cell holding a formula imports as the value the workbook last calculated for it (never the formula text), dates keep the workbook's own date system (including the 1904 system of older Mac workbooks) and import as YYYY-MM-DD, time-only cells as HH:MM:SS, and an error cell (#DIV/0!, #N/A) imports blank; the import tells you how many there were.
• Parquet / Arrow: typed columns read directly by the native engine.

Big integer identifiers stay exact. An integer column holding values beyond what double precision can represent exactly (IDs beyond about 9 × 10¹⁵; adjacent identifiers there can collapse into the same number) imports as a String variable holding the exact digits, and the import summary says so. That keeps every ID distinct for grouping, joins and saving; the trade is that such a column sorts as text and is refused by numeric analyses (which is what you want for an identifier). The same rule applies to CSV/JSON type inference, and a .ckd save preserves such values exactly. If a number really was a measurement, not an ID, retype the variable to Numeric in the Variable View; analyses will then note that the values were rounded to double precision.

• JSON: either an array of records ([{"age": 30, …}, …], one object per case) or a columnar object ({"age": [30, 31, …], …}); newline-delimited .ndjson/.jsonl has one object per line. The column names come from the JSON, so there is no header prompt; each column's type is inferred like CSV.
• SPSS (.sav, .zsav). Typed columns plus the file's metadata: variable labels, value labels (1 → Male arrives as a ChakataStat value label), measurement levels, and discrete user-missing values, on a date or time variable too, where the code arrives as a date (1900-01-01) matching the cells it marks. SPSS ranges of missing values have no equivalent here; the import says so in its summary rather than dropping them silently. Legacy portable .por files are not read. Save them as .sav from SPSS first.
• Stata (.dta): typed columns plus variable labels and value labels (every .dta release is read). Daily dates (%td) and datetimes (%tc) arrive as readable dates; the rarer week/month/quarter display formats arrive as their stored numbers, and the import summary notes it. Stata's long strings (strL) are carried like any other text, including files written by third-party tools whose strL layout differs from Stata's own. A rare binary strL payload is decoded as text and the import summary names the variable.
• SAS transport (.xpt): the open format SAS writes for exchange (PROC CPORT/COPY), typed columns plus variable labels. SAS date, datetime and time formats arrive as readable dates and times. Two limits, both noted in the import summary: value labels are not in the file at all (SAS keeps them in a separate .sas7bcat catalog), and if the file holds more than one dataset, the first is imported.
• SAS data (.sas7bdat): SAS's own data files open directly, typed columns plus variable labels and the file label, with dates, datetimes and times rendered exactly as from .xpt (fractional seconds are truncated to the second; the month-year / quarter / year display formats arrive as their stored numbers, noted in the summary). Value labels live in a separate SAS format catalog, so put the .sas7bcat next to the data file, named like it (study.sas7bcat beside study.sas7bdat) or as formats.sas7bcat, SAS's default catalog name, and every variable whose format the catalog defines gets its value labels. With no catalog beside the file, a variable with a user-defined format is named in the import summary, with that advice. SAS's missing values (. and the special .A–.Z) arrive blank, and a .sas7bcat that cannot be read is reported, not fatal. Both compressed (COMPRESS=CHAR/BINARY) and uncompressed files read, in any of SAS's session encodings.

Text encodings

Every text import (delimited, fixed-width, JSON, gzipped included) detects its encoding rather than assuming UTF-8: a byte-order mark wins when present (UTF-8, UTF-16LE, UTF-16BE); without one, UTF-16 is recognised heuristically, then strict UTF-8 is tried, and Windows-1252 (what "Latin-1" files in the wild almost always are) is the final fallback, so the decode itself never fails. UTF-8 files open silently (that is the norm, not news); anything else imports and says so: the import summary carries a note such as decoded as UTF-16LE, telling you a conversion happened and which encoding was assumed. If text still looks wrong despite the note, the file is in some other single-byte code page; re-save it as UTF-8 and it opens exactly. Exported files are always UTF-8.

A progress dialog shows while large files load, and the import runs without freezing the window.

Tip. After importing a CSV, visit the Variable View and set measurement levels and labels. Type inference gets storage right but cannot know that a numeric 1/2 column means Male/Female; that is a value-label job (below). An SPSS or Stata import does this for you: the labels arrive with the data.

Exporting data

File → Export Data… writes the current dataset to CSV, TSV, Excel, Parquet, JSON (an array of records) or NDJSON (.ndjson / .jsonl, one record per line). The format is chosen from the file extension you give, and a name with no extension is written as CSV. Every case is written: an active Select Cases filter decides what the analyses see, it does not remove rows from an export.

What each format keeps:

For a file that keeps everything, including the variable metadata, save the dataset as .ckd instead.

CSV/TSV exports are protected against spreadsheet formula injection by default: a text cell that starts with =, +, -, @, a tab or a carriage return and is not a number is written with a leading apostrophe ('), which Excel and Calc read as "this is text" and hide, so a value that arrived in your data can never run as a formula on someone else's machine. Numbers such as -5 are never touched. If you need a byte-faithful file (for a tool that would show the apostrophe), turn the guard off under Settings → Files. A CSV/TSV export that holds non-ASCII text (é, ü, …) starts with a UTF-8 byte-order mark so Excel reads it correctly; that too can be turned off under Settings → Files.

To export results rather than data, see the output log export.

The Data View

The Data View (Ctrl+1) is the spreadsheet you type into:

• Click a cell and type to edit; Tab/Enter move the active cell.
• Paste (Ctrl+V) a block from another spreadsheet; rows and columns grow to fit. Typing into the empty grid creates the first variables automatically. Copy and paste use the spreadsheet clipboard format, quoting included, so a cell holding a line break (a multi-line Excel cell, an address) stays one cell both ways. One copy or paste handles up to 200,000 cells; a larger one does nothing and the status line says why (a cut that large leaves the cells in place).
• Find & Replace (Ctrl+F) opens a search bar with match case, entire cell and wrap-around options; Replace All is a single undoable edit.
• Hover any column header to see that variable's label, type and measurement level as a tooltip. The editor documents itself.
• When the dataset is empty, the view shows an empty state with Open and Import buttons instead of a blank grid.
• Hide a column you don't need right now: right-click its header and choose Hide column (selecting several headers first hides them all together). A thin double line marks where columns are hidden. Click it, or right-click the header on either side and choose Unhide, to bring them back. View → Unhide All Columns is the escape hatch if you lose track. Hiding is a display preference, not a data change: the variable still exists, still appears in every dialog's variable list, and stays valid in .cks syntax; it just isn't shown in the grid.

With a screen reader, the grid reads like a grid: each visible cell announces its variable, row number, value and selection state, moving the selection (arrow keys, Tab, Enter, or the reader's own navigation) announces the newly active cell, and every cell offers an Edit cell action alongside its activate action. Only what is on screen is exposed (scrolling brings the next window of cells into the reader's view), so large datasets stay responsive.

Edits are undoable (Ctrl+Z / Ctrl+Y), and undo works on the whole dataset, not just cells: deleting a variable and undoing it brings back its data and its full definition (name, labels, value labels, missing values, type, measure) at its place, with any Select Cases / Weight Cases / Split File reference that pointed at it restored too. Transforms (Compute, Recode, Rank, a merge, a sort) undo as one step; metadata edits in the Variable View are undoable as well, with consecutive keystrokes on one field counting as a single step. Undoing a transform also removes its command from the Syntax console's journal (and redo puts it back), so the journal keeps describing what was actually done. Transpose is the one exception: it replaces the whole dataset and cannot be undone; the dialog says so.

The Variable View

The Variable View (Ctrl+2) shows one row per variable. The columns are the variable's metadata; click a cell to edit it:

Why the measurement level matters

The Measure tells procedures how to treat a variable:

• Scale. A true number (age, blood pressure): means, correlations, the numeric axis of a histogram.
• Ordinal. Ordered categories (education level): ranks and ordered tests.
• Nominal. Unordered categories (region, sex): counts, crosstabs, grouping.

Charts and dialogs use the level to offer sensible defaults and to label output, so setting it correctly is worth the minute it takes.

Value labels and missing values

Both are edited in a small dialog from the relevant Variable View cell:

• Value labels make output readable: a crosstab shows Male/Female instead of 1/2, and the underlying codes stay numeric for analysis.
• User-missing values mark specific codes as "not a real value". They reach the engine as nulls and are dropped from every statistic, while still being visible (and editable) in the Data View.

Saving: the .ckd document

File → Save (Ctrl+S) writes an .ckd file, ChakataStat's native document. Unlike a CSV, it stores both the data and every variable definition (labels, value labels, missing rules, measurement levels), plus document provenance. Reopening an .ckd restores the dataset exactly.

Internally .ckd is a versioned ZIP of a meta.json (all metadata) and a typed data.parquet (the columns). You never need to look inside, but the format is documented for the curious in docs/file_format.md.

Your wider session (the open document, the results log, the AI conversation and the syntax) is also remembered between launches, and the unsaved working dataset is autosaved for crash recovery.

Tidying a dataset

A few tools under the Data menu help with messy or wide datasets:

• Define Variable Properties…: scan the data and set value labels, measurement levels and missing rules for several variables at once, guided by the values actually present.
• Bulk Rename Variables…: rename many variables with a regular expression (with $1 group references), a live preview and collision-safe de-duplication. Ideal for stripping a common prefix or renaming q1, q2, ….
• Define Variable Sets…: group related variables under a name, then pick the set in the Variable View's filter bar to focus on just that group. Since 0.14.0 the active set reaches everywhere you pick variables: the Data View shows only the set's columns, and analysis dialogs offer only the set's variables in their source lists. This is the way to keep a 400-column survey workable. Sets are a view convenience: they never change the data, a hidden column stays in every analysis, and a variable you already selected in a dialog stays listed even if the set excludes it. Sets persist with the session.

The data dictionary (.ckdict)

File → Export Data Dictionary… writes all variable metadata to a standalone, versioned JSON .ckdict file: names, labels, types, value labels and missing rules, without the data. Import Data Dictionary… applies one to the current dataset: it matches by variable name, writes only the fields each (even hand-edited, partial) entry carries, leaves your data untouched, and reports the fit (applied / unmatched / not in dictionary).

This is how you reuse one carefully-defined codebook across many data files.

Where to go next

• Case Semantics & Transforms: filter, weight and split cases; compute, recode and rank.
• Analyzing Data: run statistics on your dataset.