jsonstat-io 0.1.0

package/docs/formats/polars.md

@@ -0,0 +1,111 @@
+# Polars Format
+
+Convert a Polars `DataFrame` (from [`nodejs-polars`](https://pola-rs.github.io/nodejs-polars/)) to JSON-stat via the [Arrow hub](./arrow.md). Polars DataFrames expose `toArrow()` / `toIPC()`, which produce Apache Arrow tables. On export, the Arrow table is converted back to a Polars DataFrame via `fromArrow()`.
+
+## Install
+
+```sh
+npm install jsonstat-io nodejs-polars
+```
+
+`nodejs-polars` is an **optional peer dependency**. It is a native Node module — **Node only**. It cannot run in the browser.
+
+## Import path
+
+```ts
+import { polarsToCube, polarsToDataset, loadPolars } from "jsonstat-io/polars";
+```
+
+The Polars adapter is **not** available via the CLI for import — it requires a live DataFrame object.
+
+## Import example
+
+```ts
+import { polarsToDataset, loadPolars } from "jsonstat-io/polars";
+
+const pl = await loadPolars();
+const df = pl.readCSV("sales.csv");
+
+const dataset = await polarsToDataset(df, {
+  measure: "amount",
+  roles: { time: ["year"], geo: ["country"] },
+});
+```
+
+`loadPolars()` lazily imports `nodejs-polars` so it never enters a bundle that doesn't use it. You can also import `nodejs-polars` directly and pass the DataFrame.
+
+## Reading Parquet via Polars
+
+Polars can read Parquet natively (often faster than `parquet-wasm` in Node):
+
+```ts
+import { polarsToDataset, loadPolars } from "jsonstat-io/polars";
+
+const pl = await loadPolars();
+const df = pl.readParquet("sales.parquet");
+const dataset = await polarsToDataset(df);
+```
+
+## Export path
+
+[`cubeToPolars`](../../src/sources/polars.ts) converts the `Observations` IR to a Polars `DataFrame`: it calls `cubeToArrow` to build the Arrow table, then `pl.fromArrow(table)` to materialize the DataFrame.
+
+```ts
+import { exportDataset } from "jsonstat-io";
+
+// JSON-stat → Polars DataFrame
+const df = await exportDataset(dataset, { to: "polars" });
+```
+
+Or use `cubeToPolars` directly on the IR:
+
+```ts
+import { cubeToPolars, loadPolars } from "jsonstat-io/polars";
+
+await loadPolars();  // ensure nodejs-polars is imported
+const df = await cubeToPolars(obs);
+```
+
+## Options
+
+### `PolarsToCubeOptions` (import)
+
+Extends [`ArrowToCubeOptions`](./arrow.md):
+
+| Field        | Type                              | Description |
+|--------------|-----------------------------------|-------------|
+| `arrowMethod`| `"toArrow" \| "toIPC" \| "auto"` | Force a specific Arrow conversion method. Default `"auto"` tries `toArrow()` then `toIPC()`. |
+| `measure`    | `string`                          | Measure column name |
+| `dimensions` | `string[]`                        | Dimension column names |
+| `roles`      | `RoleMap`                         | Role assignments |
+
+### `CubeToPolarsOptions` (export)
+
+| Field   | Type                                  | Default | Description |
+|---------|---------------------------------------|---------|-------------|
+| `method`| `"fromArrow" \| "ipc" \| "auto"`     | `"auto"`| Conversion method: `pl.fromArrow(table)` or round-trip through IPC. |
+
+## How it works
+
+**Import:**
+1. `polarsToCube` extracts an Arrow Table from the DataFrame, trying `toArrow()` first, then `toIPC()` (decoded via `tableFromIPC`).
+2. Passes the table to `arrowToCube` → `Observations` IR.
+
+**Export:**
+1. `cubeToPolars` calls `cubeToArrow(obs)` → Apache Arrow `Table`.
+2. Converts via `pl.fromArrow(table)` (or round-trips through IPC if `fromArrow` is unavailable).
+
+If `nodejs-polars` is not installed or the DataFrame has no `toArrow()`/`toIPC()` method, a `PolarsSourceError` is thrown.
+
+## Browser alternative
+
+For browser Polars data, convert to Arrow IPC first (Polars can serialize to IPC), then use the Arrow hub directly:
+
+```ts
+import { arrowToDataset } from "jsonstat-io";
+import { tableFromIPC } from "apache-arrow";
+
+const ipcBytes = /* obtain Arrow IPC bytes */;
+const table = tableFromIPC(ipcBytes);
+const dataset = arrowToDataset(table);
+```

package/docs/mapping.md

@@ -0,0 +1,136 @@
+# Spec-Fidelity Mapping
+
+This is the **correctness contract**: how every JSON-stat 2.0 concept maps to/from the columnar world and the internal Observations IR. If a round-trip loses information, it is a bug.
+
+## JSON-stat ↔ Observations IR
+
+| JSON-stat concept          | Observations IR field         | Notes |
+|----------------------------|-------------------------------|-------|
+| `dataset.id`               | `meta.label` / `dimensions`   | Built from dimension IDs in `id[]` order |
+| `dataset.size`             | *(computed)*                  | Product of resolved category counts; not stored in IR |
+| `dataset.dimension[id]`    | `dimensions[id]`              | One `DimensionColumn` per dimension |
+| `dimension.category.index` | *(resolved position map)*     | Built by `resolveDimension` — explicit order or first-seen |
+| `dimension.category.label` | `DimensionColumn.categoryLabels` | `{id: label}` |
+| `dimension.label`          | `DimensionColumn.label`       | |
+| `dataset.value` (array)    | `measure.values` scattered     | Dense form: full row-major array |
+| `dataset.value` (object)   | `measure.values` scattered     | Sparse form: `{position: value}` |
+| `dataset.status`           | `status.values`               | Per-row status, deduplicated on emit |
+| `dataset.role`             | `roles`                       | `{time?: [], geo?: [], metric?: []}` |
+| `dimension.category.unit`  | `DimensionColumn.categoryUnits` | `{id: {decimals, label}}` |
+| `dimension.category.coordinates` | `DimensionColumn.categoryCoords` | `{id: [lon, lat]}` |
+| `dimension.category.child` | `DimensionColumn.categoryChild` | `{parent: [child, ...]}` |
+| `dimension.category.order` | `DimensionColumn.categoryOrder` | Explicit `[id, ...]` ordering |
+
+## Arrow schema metadata contract
+
+Producers annotate Arrow `Field` and `Schema` metadata with `jsonstat.*` keys so `arrowToCube` can reconstruct the full model losslessly. Without metadata, the converter uses heuristics (dictionary = dimension, first numeric = measure).
+
+### Field-level keys (`Field.metadata`)
+
+| Key                       | Value                              | Meaning |
+|---------------------------|------------------------------------|---------|
+| `jsonstat.role`           | `"time"` / `"geo"` / `"metric"`    | Dimension role |
+| `jsonstat.measure`        | `"true"`                           | Marks this column as the measure (overrides type heuristics) |
+| `jsonstat.status`         | `"true"`                           | Marks this column as the per-row status column |
+| `jsonstat.label`          | string                             | Dimension label |
+| `jsonstat.categoryLabels` | JSON object: `{id: label}`         | Category labels |
+| `jsonstat.categoryUnits`  | JSON object: `{id: {decimals,...}}`| Per-category unit metadata (metric role) |
+| `jsonstat.categoryCoords` | JSON object: `{id: [lon,lat]}`     | Per-category coordinates (geo role) |
+| `jsonstat.categoryChild`  | JSON object: `{parent: [child,...]}`| Hierarchy |
+| `jsonstat.categoryOrder`  | JSON array: `[id,...]`             | Explicit category ordering (e.g. chronological) |
+
+### Schema-level keys (`Schema.metadata`)
+
+| Key                  | Value                          | Meaning |
+|----------------------|--------------------------------|---------|
+| `jsonstat.label`     | string                         | Dataset label |
+| `jsonstat.source`    | string                         | Dataset source |
+| `jsonstat.updated`   | ISO 8601 string                | Dataset update time |
+| `jsonstat.extension` | JSON object                    | Provider extension data |
+| `jsonstat.valueForm` | `"auto"`/`"dense"`/`"sparse"`  | Value emission hint |
+
+## Arrow type → JSON-stat mapping
+
+| Arrow type                  | JSON-stat role | Notes |
+|-----------------------------|----------------|-------|
+| `Dictionary<Utf8, Int*>`    | dimension      | Natural fit — dictionary-encoded strings |
+| `Utf8` (non-dictionary)     | dimension      | Stringified |
+| `Bool`                      | dimension      | → `"true"`/`"false"` |
+| `Int32`, `Int64`            | dimension or measure | Dimension if not the detected/declared measure |
+| `Float32`, `Float64`        | measure (default) | First numeric column is the measure heuristic |
+| `DateDay`, `DateMillisecond`| dimension (time) | ISO date string |
+| `TimestampSecond/Millisecond`| dimension (time)| ISO 8601 string |
+| Other types                 | dimension      | Stringified via `String(vec.get(i))` |
+
+**Detection order** (when metadata is absent):
+1. If `options.measure` is set, that column is the measure.
+2. Otherwise, the first `Float64`/`Float32`/`Int64`/`Int32` column is the measure.
+3. A column named `status` (case-insensitive) is the status column.
+4. All remaining columns are dimensions, in schema order.
+
+**Roles** (when metadata is absent):
+- A dimension named `year`, `date`, `time`, or `period` (case-insensitive) gets the `time` role.
+- A dimension named `country`, `region`, `geo`, or `area` gets the `geo` role.
+- The measure column gets the `metric` role.
+- Explicit `options.roles` merge with and take precedence over both metadata and heuristics.
+
+## Row-major value ordering
+
+JSON-stat stores values in **row-major** order: the first dimension varies slowest, the last dimension varies fastest. For dimensions `[sex, age, year]` with sizes `[2, 2, 2]`:
+
+```
+Position  0: sex[0], age[0], year[0]
+Position  1: sex[0], age[0], year[1]
+Position  2: sex[0], age[1], year[0]
+Position  3: sex[0], age[1], year[1]
+Position  4: sex[1], age[0], year[0]
+Position  5: sex[1], age[0], year[1]
+Position  6: sex[1], age[1], year[0]
+Position  7: sex[1], age[1], year[1]
+```
+
+The stride of dimension `d` is the product of all subsequent sizes:
+
+```
+strides[sex]  = size[age] × size[year] = 2 × 2 = 4
+strides[age]  = size[year]             = 2
+strides[year] = 1
+```
+
+`flatPosition = Σ indices[d] × strides[d]`. The builder scatters each IR row's measure value into its computed position. `null` values are preserved in position (dense) or omitted (sparse).
+
+## Dense vs sparse value form
+
+| Form   | JSON shape               | When used |
+|--------|--------------------------|-----------|
+| Dense  | `"value": [10,20,...]`   | Full array, including `null` holes |
+| Sparse | `"value": {"0":10,...}`  | Object with only present positions |
+
+The `decideDensity` heuristic: if the null ratio exceeds the threshold (default `0.5`), emit sparse; otherwise dense. Override with `valueForm: "dense"` or `"sparse"`.
+
+## Status forms
+
+JSON-stat allows three `status` representations:
+
+| Form   | JSON shape                  | When emitted |
+|--------|-----------------------------|--------------|
+| String | `"status": "p"`            | Every row has the same status |
+| Array  | `"status": ["p","e",...]`  | Per-cell status, mixed |
+| (none) | *(omitted)*                 | No status column present |
+
+`statusForm: "object"` is also available to force the explicit object form `{position: code}` regardless of uniformity.
+
+## CSVW column mapping
+
+CSVW metadata declares column types and roles explicitly:
+
+| CSVW property                    | Maps to                          |
+|----------------------------------|----------------------------------|
+| `tableSchema.columns[].titles`   | Column name (first title)        |
+| `tableSchema.columns[].datatype` | Measure if `decimal`/`integer`/`double` |
+| `tableSchema.columns[].propertyUrl` containing `#time` / `#geo` | Role |
+| `tableSchema.primaryKey`         | Dimension columns (if no explicit dims) |
+
+## Round-trip guarantee
+
+The test suite verifies that `readDataset` (JSON-stat → IR) followed by `buildDataset` (IR → JSON-stat) reproduces the original dataset's values, dimensions, roles, and status. The same guarantee holds for `arrowToCube` → `cubeToArrow` on the Arrow side, and for the full export round-trip: `export → bytes → import` returns the original dataset's values, dimensions, roles, and status. Any information loss in these round-trips is considered a bug.

package/package.json

@@ -0,0 +1,119 @@
+{
+  "name": "jsonstat-io",
+  "version": "0.1.0",
+  "description": "Bidirectional bridge between the columnar stack (Apache Arrow, Parquet, DuckDB, Polars, CSVW, CSV) and JSON-stat 2.0 cubes. An Arrow-hubbed interop layer for the lakehouse ecosystem.",
+  "license": "Apache-2.0",
+  "author": "Xavier Badosa",
+  "homepage": "https://github.com/jsonstat/io#readme",
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/jsonstat/io.git"
+  },
+  "bugs": {
+    "url": "https://github.com/jsonstat/io/issues"
+  },
+  "keywords": [
+    "json-stat",
+    "jsonstat",
+    "arrow",
+    "apache-arrow",
+    "parquet",
+    "duckdb",
+    "polars",
+    "csvw",
+    "statistical-data",
+    "cube",
+    "lakehouse",
+    "interop"
+  ],
+  "type": "module",
+  "engines": {
+    "node": ">=18"
+  },
+  "exports": {
+    ".": {
+      "types": "./dist/index.d.ts",
+      "import": "./dist/index.js",
+      "require": "./dist/index.cjs"
+    },
+    "./arrow": {
+      "types": "./dist/arrow/index.d.ts",
+      "import": "./dist/arrow/index.js",
+      "require": "./dist/arrow/index.cjs"
+    },
+    "./parquet": {
+      "types": "./dist/sources/parquet.d.ts",
+      "import": "./dist/sources/parquet.js",
+      "require": "./dist/sources/parquet.cjs"
+    },
+    "./duckdb": {
+      "types": "./dist/sources/duckdb.d.ts",
+      "import": "./dist/sources/duckdb.js",
+      "require": "./dist/sources/duckdb.cjs"
+    },
+    "./polars": {
+      "types": "./dist/sources/polars.d.ts",
+      "import": "./dist/sources/polars.js",
+      "require": "./dist/sources/polars.cjs"
+    },
+    "./csvw": {
+      "types": "./dist/sources/csvw.d.ts",
+      "import": "./dist/sources/csvw.js",
+      "require": "./dist/sources/csvw.cjs"
+    },
+    "./csv": {
+      "types": "./dist/sources/csv.d.ts",
+      "import": "./dist/sources/csv.js",
+      "require": "./dist/sources/csv.cjs"
+    },
+    "./package.json": "./package.json"
+  },
+  "main": "./dist/index.cjs",
+  "module": "./dist/index.js",
+  "types": "./dist/index.d.ts",
+  "bin": {
+    "jsonstat-io": "./dist/cli/index.js"
+  },
+  "files": [
+    "dist",
+    "README.md",
+    "LICENSE",
+    "docs"
+  ],
+  "sideEffects": false,
+  "scripts": {
+    "build": "tsup",
+    "dev": "tsup --watch",
+    "typecheck": "tsc --noEmit",
+    "test": "vitest run",
+    "test:watch": "vitest",
+    "lint": "biome check .",
+    "format": "biome format --write .",
+    "prepublishOnly": "npm run typecheck && npm run test && npm run build"
+  },
+  "dependencies": {
+    "apache-arrow": "^17.0.0",
+    "commander": "^12.1.0"
+  },
+  "optionalDependencies": {
+    "@duckdb/duckdb-wasm": "^1.29.0",
+    "duckdb-async": "^1.2.1",
+    "jsonstat-validator": "^2.5.0",
+    "nodejs-polars": "^0.15.0",
+    "parquet-wasm": "^0.6.1"
+  },
+  "peerDependenciesMeta": {
+    "@duckdb/duckdb-wasm": { "optional": true },
+    "duckdb-async": { "optional": true },
+    "jsonstat-validator": { "optional": true },
+    "nodejs-polars": { "optional": true },
+    "parquet-wasm": { "optional": true }
+  },
+  "devDependencies": {
+    "@biomejs/biome": "^1.9.4",
+    "@types/node": "^22.10.0",
+    "tsup": "^8.3.5",
+    "typescript": "^5.7.2",
+    "vitest": "^2.1.8"
+  }
+}