jsonstat-io 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (99) hide show
  1. package/LICENSE +201 -0
  2. package/README.md +200 -0
  3. package/dist/arrow/index.cjs +60 -0
  4. package/dist/arrow/index.cjs.map +1 -0
  5. package/dist/arrow/index.d.cts +125 -0
  6. package/dist/arrow/index.d.ts +125 -0
  7. package/dist/arrow/index.js +3 -0
  8. package/dist/arrow/index.js.map +1 -0
  9. package/dist/arrowToCube-BdRWZ5oG.d.cts +29 -0
  10. package/dist/arrowToCube-Mjr79ZHM.d.ts +29 -0
  11. package/dist/chunk-2HY7NRKX.js +259 -0
  12. package/dist/chunk-2HY7NRKX.js.map +1 -0
  13. package/dist/chunk-62NIY7ML.js +191 -0
  14. package/dist/chunk-62NIY7ML.js.map +1 -0
  15. package/dist/chunk-AXUZF6O5.cjs +352 -0
  16. package/dist/chunk-AXUZF6O5.cjs.map +1 -0
  17. package/dist/chunk-EFRDFQOT.js +335 -0
  18. package/dist/chunk-EFRDFQOT.js.map +1 -0
  19. package/dist/chunk-GDKDYLZ4.cjs +263 -0
  20. package/dist/chunk-GDKDYLZ4.cjs.map +1 -0
  21. package/dist/chunk-IDTQZG4B.js +432 -0
  22. package/dist/chunk-IDTQZG4B.js.map +1 -0
  23. package/dist/chunk-KFIKGBIJ.cjs +200 -0
  24. package/dist/chunk-KFIKGBIJ.cjs.map +1 -0
  25. package/dist/chunk-L65CRXTV.js +64 -0
  26. package/dist/chunk-L65CRXTV.js.map +1 -0
  27. package/dist/chunk-PHLCWTHB.cjs +446 -0
  28. package/dist/chunk-PHLCWTHB.cjs.map +1 -0
  29. package/dist/chunk-VQCS4LVD.cjs +166 -0
  30. package/dist/chunk-VQCS4LVD.cjs.map +1 -0
  31. package/dist/chunk-VYS7DPB3.js +162 -0
  32. package/dist/chunk-VYS7DPB3.js.map +1 -0
  33. package/dist/chunk-ZEDTWRDN.cjs +70 -0
  34. package/dist/chunk-ZEDTWRDN.cjs.map +1 -0
  35. package/dist/cli/index.cjs +337 -0
  36. package/dist/cli/index.cjs.map +1 -0
  37. package/dist/cli/index.d.cts +90 -0
  38. package/dist/cli/index.d.ts +90 -0
  39. package/dist/cli/index.js +333 -0
  40. package/dist/cli/index.js.map +1 -0
  41. package/dist/cubeBuilder-BEGA4C5Y.js +4 -0
  42. package/dist/cubeBuilder-BEGA4C5Y.js.map +1 -0
  43. package/dist/cubeBuilder-ZU76VVEX.cjs +25 -0
  44. package/dist/cubeBuilder-ZU76VVEX.cjs.map +1 -0
  45. package/dist/cubeReader-7QVUF3TI.js +4 -0
  46. package/dist/cubeReader-7QVUF3TI.js.map +1 -0
  47. package/dist/cubeReader-UN2BHQWK.cjs +21 -0
  48. package/dist/cubeReader-UN2BHQWK.cjs.map +1 -0
  49. package/dist/index.cjs +172 -0
  50. package/dist/index.cjs.map +1 -0
  51. package/dist/index.d.cts +445 -0
  52. package/dist/index.d.ts +445 -0
  53. package/dist/index.js +7 -0
  54. package/dist/index.js.map +1 -0
  55. package/dist/ir-BArfNxo-.d.cts +143 -0
  56. package/dist/ir-Jae2ymov.d.ts +143 -0
  57. package/dist/jsonstat-BX4mBLci.d.cts +179 -0
  58. package/dist/jsonstat-BX4mBLci.d.ts +179 -0
  59. package/dist/sources/csv.cjs +40 -0
  60. package/dist/sources/csv.cjs.map +1 -0
  61. package/dist/sources/csv.d.cts +112 -0
  62. package/dist/sources/csv.d.ts +112 -0
  63. package/dist/sources/csv.js +3 -0
  64. package/dist/sources/csv.js.map +1 -0
  65. package/dist/sources/csvw.cjs +236 -0
  66. package/dist/sources/csvw.cjs.map +1 -0
  67. package/dist/sources/csvw.d.cts +143 -0
  68. package/dist/sources/csvw.d.ts +143 -0
  69. package/dist/sources/csvw.js +230 -0
  70. package/dist/sources/csvw.js.map +1 -0
  71. package/dist/sources/duckdb.cjs +121 -0
  72. package/dist/sources/duckdb.cjs.map +1 -0
  73. package/dist/sources/duckdb.d.cts +111 -0
  74. package/dist/sources/duckdb.d.ts +111 -0
  75. package/dist/sources/duckdb.js +115 -0
  76. package/dist/sources/duckdb.js.map +1 -0
  77. package/dist/sources/parquet.cjs +117 -0
  78. package/dist/sources/parquet.cjs.map +1 -0
  79. package/dist/sources/parquet.d.cts +83 -0
  80. package/dist/sources/parquet.d.ts +83 -0
  81. package/dist/sources/parquet.js +111 -0
  82. package/dist/sources/parquet.js.map +1 -0
  83. package/dist/sources/polars.cjs +94 -0
  84. package/dist/sources/polars.cjs.map +1 -0
  85. package/dist/sources/polars.d.cts +91 -0
  86. package/dist/sources/polars.d.ts +91 -0
  87. package/dist/sources/polars.js +87 -0
  88. package/dist/sources/polars.js.map +1 -0
  89. package/docs/api.md +322 -0
  90. package/docs/architecture.md +140 -0
  91. package/docs/cli.md +159 -0
  92. package/docs/formats/arrow.md +111 -0
  93. package/docs/formats/csv.md +115 -0
  94. package/docs/formats/csvw.md +152 -0
  95. package/docs/formats/duckdb.md +151 -0
  96. package/docs/formats/parquet.md +132 -0
  97. package/docs/formats/polars.md +111 -0
  98. package/docs/mapping.md +136 -0
  99. package/package.json +119 -0
@@ -0,0 +1,111 @@
1
+ # Arrow Format
2
+
3
+ Apache Arrow is the **hub** of `jsonstat-io`. Parquet, DuckDB, and Polars all produce Arrow tables, and a single converter pair ([`arrowToCube`](../../src/arrow/arrowToCube.ts) / [`cubeToArrow`](../../src/arrow/arrowFromCube.ts)) handles them all in both directions.
4
+
5
+ ## Import path
6
+
7
+ ```ts
8
+ import { arrowToCube, arrowToDataset } from "jsonstat-io";
9
+ // or from the dedicated subpath:
10
+ import { arrowToCube } from "jsonstat-io/arrow";
11
+ ```
12
+
13
+ `apache-arrow` is a **hard dependency** — no peer install needed.
14
+
15
+ ## Reading an Arrow IPC file
16
+
17
+ ```ts
18
+ import { tableFromIPC } from "apache-arrow";
19
+ import { arrowToDataset } from "jsonstat-io";
20
+ import { readFileSync } from "node:fs";
21
+
22
+ const bytes = readFileSync("./data.arrow");
23
+ const table = tableFromIPC(bytes);
24
+ const dataset = arrowToDataset(table, { measure: "amount" });
25
+ ```
26
+
27
+ In the browser, fetch the bytes and pass them directly:
28
+
29
+ ```ts
30
+ const res = await fetch("/data.arrow");
31
+ const bytes = new Uint8Array(await res.arrayBuffer());
32
+ const table = tableFromIPC(bytes);
33
+ const dataset = arrowToDataset(table);
34
+ ```
35
+
36
+ ## How Arrow columns map
37
+
38
+ | Arrow column type | JSON-stat role | Detection |
39
+ |------------------------------|------------------|-----------|
40
+ | `Dictionary<Utf8, Int*>` | dimension | Always a dimension |
41
+ | `Utf8`, `Bool`, temporal | dimension | Stringified |
42
+ | `Float64`, `Float32`, `Int*` | measure (first) | First numeric column, unless overridden |
43
+ | Any column with `jsonstat.status: "true"` | status | Metadata marker |
44
+
45
+ ### Without metadata (heuristic mode)
46
+
47
+ `arrowToCube` infers:
48
+ 1. The first `Float64`/`Float32`/`Int64`/`Int32` column is the **measure**.
49
+ 2. A column named `status` (case-insensitive) is the **status** column.
50
+ 3. All other columns are **dimensions**, in schema order.
51
+ 4. Roles: `year`/`date`/`time`/`period` → `time`; `country`/`region`/`geo`/`area` → `geo`; the measure → `metric`.
52
+
53
+ ### With metadata (lossless mode)
54
+
55
+ Annotate Arrow `Field` and `Schema` metadata with `jsonstat.*` keys for exact, lossless mapping. See the [metadata key reference](../mapping.md#arrow-schema-metadata-contract).
56
+
57
+ ```ts
58
+ import { Field, Schema, Dictionary, Utf8, Int32, Float64 } from "apache-arrow";
59
+ import { buildFieldMeta, buildSchemaMeta } from "jsonstat-io";
60
+
61
+ const yearField = new Field("year", new Dictionary(new Utf8(), new Int32()), false,
62
+ new Map(Object.entries(buildFieldMeta({
63
+ role: "time",
64
+ label: "Year",
65
+ categoryOrder: ["2020", "2021", "2022"],
66
+ categoryLabels: { "2020": "FY 2020", "2021": "FY 2021" },
67
+ })))
68
+ );
69
+
70
+ const schema = new Schema(
71
+ [yearField, /* ... */],
72
+ new Map(Object.entries(buildSchemaMeta({ label: "Annual Sales", source: "Finance Dept" })))
73
+ );
74
+ ```
75
+
76
+ ## Annotating for round-trip fidelity
77
+
78
+ To guarantee that `arrowToCube` → `cubeToArrow` round-trips losslessly, annotate:
79
+
80
+ - **Schema**: `jsonstat.label`, `jsonstat.source`, `jsonstat.updated`, `jsonstat.valueForm`.
81
+ - **Each dimension field**: `jsonstat.role`, `jsonstat.label`, `jsonstat.categoryLabels`, `jsonstat.categoryUnits`, `jsonstat.categoryCoords`, `jsonstat.categoryChild`, `jsonstat.categoryOrder`.
82
+ - **Measure field**: `jsonstat.measure: "true"` (or `jsonstat.role: "metric"`).
83
+ - **Status field**: `jsonstat.status: "true"`.
84
+
85
+ ## Export path
86
+
87
+ [`cubeToArrow`](../../src/arrow/arrowFromCube.ts) converts the `Observations` IR back to an Arrow `Table`, building dictionary-encoded dimensions and a Float64 measure with full `jsonstat.*` metadata annotation.
88
+
89
+ ```ts
90
+ import { exportDataset } from "jsonstat-io";
91
+
92
+ // JSON-stat → Arrow Table
93
+ const table = await exportDataset(dataset, { to: "arrow" });
94
+ ```
95
+
96
+ Or use `cubeToArrow` directly on the IR:
97
+
98
+ ```ts
99
+ import { cubeToArrow } from "jsonstat-io";
100
+ // obs = Observations IR
101
+ const table = cubeToArrow(obs);
102
+ ```
103
+
104
+ ### CLI
105
+
106
+ ```sh
107
+ # JSON-stat → Arrow IPC file
108
+ npx jsonstat-io ./data.jsonstat.json --to arrow -o data.arrow
109
+ ```
110
+
111
+ The exporter writes Arrow IPC stream format. `cubeToArrow` is also the foundation for the Parquet, DuckDB, and Polars exporters — they all call `cubeToArrow` first, then convert the resulting Arrow Table to their target format.
@@ -0,0 +1,115 @@
1
+ # CSV Format
2
+
3
+ Plain CSV (no metadata) is the simplest path. The [`csvToCube`](../../src/sources/csv.ts) adapter infers the measure and dimensions heuristically on import. On export, [`cubeToCsv`](../../src/sources/csv.ts) serializes the IR to CSV text. It is **dependency-free** and works in both Node and the browser.
4
+
5
+ ## Import path
6
+
7
+ ```ts
8
+ import { csvToCube, parseCsv } from "jsonstat-io/csv";
9
+ ```
10
+
11
+ Or use the high-level dispatcher:
12
+
13
+ ```ts
14
+ import { importToDataset } from "jsonstat-io";
15
+ const dataset = await importToDataset("./data.csv", { from: "csv", measure: "amount" });
16
+ ```
17
+
18
+ ## Inference heuristics (no metadata)
19
+
20
+ Without a schema, the adapter infers:
21
+
22
+ 1. The **measure** is the first column that parses as a number for every non-empty row — or the one named by `options.measure`.
23
+ 2. A column named `status` (case-insensitive) is the **status** column — unless `options.status` overrides it.
24
+ 3. Every other column is a **dimension** (string-typed), in file order.
25
+
26
+ Empty cells become `null` in the measure and `""` in dimensions. For richer, lossless mapping with declared types/labels/roles, use [CSVW](./csvw.md) instead.
27
+
28
+ ## Import example
29
+
30
+ ```ts
31
+ import { csvToCube } from "jsonstat-io/csv";
32
+ import { buildDataset } from "jsonstat-io";
33
+
34
+ const csvText = `year,country,amount
35
+ 2020,Spain,100
36
+ 2020,France,200
37
+ 2021,Spain,150
38
+ 2021,France,250`;
39
+
40
+ const obs = csvToCube(csvText, {
41
+ measure: "amount",
42
+ roles: { time: ["year"], geo: ["country"] },
43
+ });
44
+
45
+ const { dataset } = buildDataset(obs, { valueForm: "dense" });
46
+ ```
47
+
48
+ ## Export path
49
+
50
+ [`cubeToCsv`](../../src/sources/csv.ts) serializes the `Observations` IR to CSV text. The output has one column per dimension (in `id[]` order), followed by the measure column, and an optional status column.
51
+
52
+ ```ts
53
+ import { exportDataset } from "jsonstat-io";
54
+
55
+ // JSON-stat → CSV text
56
+ const csv = await exportDataset(dataset, { to: "csv" });
57
+ ```
58
+
59
+ Or use `cubeToCsv` directly on the IR:
60
+
61
+ ```ts
62
+ import { cubeToCsv } from "jsonstat-io/csv";
63
+
64
+ const csv = cubeToCsv(obs, { delimiter: ";" });
65
+ ```
66
+
67
+ ### CLI
68
+
69
+ ```sh
70
+ # JSON-stat → CSV
71
+ npx jsonstat-io ./data.jsonstat.json --to csv -o data.csv
72
+
73
+ # With custom delimiter via CSVW metadata (CSV itself has no delimiter flag)
74
+ npx jsonstat-io ./data.jsonstat.json --to csvw -o data.csvw --delimiter ";"
75
+ ```
76
+
77
+ ## Import CLI
78
+
79
+ ```sh
80
+ # From a file
81
+ npx jsonstat-io ./data.csv --from csv --measure amount --role time=year,geo=country
82
+
83
+ # From stdin
84
+ cat data.csv | npx jsonstat-io - --measure amount
85
+
86
+ # Custom delimiter
87
+ npx jsonstat-io ./data.csv --delimiter ";"
88
+ ```
89
+
90
+ When `--from` is omitted, `.csv` extension triggers CSV detection. The CLI checks for a sibling `*-metadata.json` first; if found, it uses CSVW, otherwise plain CSV.
91
+
92
+ ## Options
93
+
94
+ ### `CsvToCubeOptions` (import)
95
+
96
+ | Field | Type | Default | Description |
97
+ |--------------|-----------------------------------|----------|-------------|
98
+ | `measure` | `string` | *(inferred)* | Explicit measure column name |
99
+ | `dimensions` | `string[]` | *(inferred)* | Explicit dimension column names |
100
+ | `status` | `string` | *(inferred)* | Status column name |
101
+ | `roles` | `RoleMap` | — | Role assignments |
102
+ | `valueForm` | `"auto" \| "dense" \| "sparse"` | `"auto"` | Value form hint |
103
+ | `delimiter` | `string` | `","` | CSV delimiter |
104
+ | `header` | `boolean` | `true` | Treat first row as header |
105
+
106
+ ### `CubeToCsvOptions` (export)
107
+
108
+ | Field | Type | Default | Description |
109
+ |-----------------|----------|---------|-------------|
110
+ | `delimiter` | `string` | `","` | CSV delimiter. |
111
+ | `lineTerminator`| `string` | `"\n"` | Row separator. |
112
+
113
+ ## Parser
114
+
115
+ The bundled `parseCsv(input, delimiter?)` is a tiny, dependency-free RFC-4180-ish CSV parser. It handles quoted fields, embedded quotes (doubled), and embedded newlines. For heavy CSV work (large files, complex quoting), prefer piping through DuckDB or Polars.
@@ -0,0 +1,152 @@
1
+ # CSVW Format
2
+
3
+ [CSV on the Web (CSVW)](https://www.w3.org/TR/tabular-metadata/) provides a metadata layer over plain CSV, declaring column types, names, and roles. This enables **lossless** CSV → JSON-stat mapping without heuristics. On export, the IR is serialized to CSV text plus a CSVW metadata JSON document.
4
+
5
+ CSVW is a **dependency-free** path — no optional peers needed. It works in both Node and the browser.
6
+
7
+ ## Import path
8
+
9
+ ```ts
10
+ import { csvwToCube, csvwToDataset, parseCsvwMetadata } from "jsonstat-io/csvw";
11
+ ```
12
+
13
+ Or use the high-level dispatcher:
14
+
15
+ ```ts
16
+ import { importToDataset } from "jsonstat-io";
17
+
18
+ // If a sibling *-metadata.json exists next to the CSV (Node):
19
+ const dataset = await importToDataset("./data.csv", { from: "csvw" });
20
+
21
+ // Or pass metadata inline:
22
+ const dataset = await importToDataset("./data.csv", {
23
+ from: "csvw",
24
+ csvwMetadata: metadataObject,
25
+ });
26
+ ```
27
+
28
+ ## Metadata structure
29
+
30
+ `csvwToCube` expects a parsed CSVW metadata object with this shape:
31
+
32
+ ```json
33
+ {
34
+ "tableSchema": {
35
+ "columns": [
36
+ { "titles": "year", "datatype": "string", "propertyUrl": "http://example.org/#time" },
37
+ { "titles": "country", "datatype": "string" },
38
+ { "titles": "amount", "datatype": "decimal" }
39
+ ],
40
+ "primaryKey": ["year", "country"]
41
+ }
42
+ }
43
+ ```
44
+
45
+ ### Column mapping
46
+
47
+ | CSVW property | Maps to |
48
+ |---------------------------------------|---------|
49
+ | `titles` (first) | Column name |
50
+ | `datatype` = `decimal`/`integer`/`double` | Measure column |
51
+ | Other `datatype` | Dimension column |
52
+ | `propertyUrl` containing `#time` | `time` role |
53
+ | `propertyUrl` containing `#geo` | `geo` role |
54
+ | `primaryKey` | Dimension columns (if no explicit `options.dimensions`) |
55
+
56
+ ### Role detection from `propertyUrl`
57
+
58
+ The adapter checks if the `propertyUrl` contains a fragment hint:
59
+
60
+ - `...#time` or `...#temporal` → `time` role
61
+ - `...#geo` or `...#spatial` → `geo` role
62
+
63
+ ## Import example
64
+
65
+ ```ts
66
+ import { csvwToDataset, parseCsvwMetadata } from "jsonstat-io/csvw";
67
+ import { readFileSync } from "node:fs";
68
+
69
+ const csvText = readFileSync("./sales.csv", "utf8");
70
+ const metadataJson = JSON.parse(readFileSync("./sales-metadata.json", "utf8"));
71
+ const metadata = parseCsvwMetadata(metadataJson);
72
+
73
+ const dataset = csvwToDataset(csvText, metadata, {
74
+ dimensions: ["year", "country"], // optional: overrides primaryKey
75
+ roles: { time: ["year"] }, // optional: overrides propertyUrl
76
+ });
77
+ ```
78
+
79
+ ## Export path
80
+
81
+ [`cubeToCsvw`](../../src/sources/csvw.ts) serializes the `Observations` IR to a CSV text string **plus** a CSVW metadata JSON object, so the pair round-trips losslessly back through `csvwToCube`.
82
+
83
+ ```ts
84
+ import { exportDataset } from "jsonstat-io";
85
+
86
+ // JSON-stat → CSV text + CSVW metadata
87
+ const { csv, metadata } = await exportDataset(dataset, { to: "csvw" });
88
+ ```
89
+
90
+ Or use `cubeToCsvw` directly on the IR:
91
+
92
+ ```ts
93
+ import { cubeToCsvw } from "jsonstat-io/csvw";
94
+
95
+ const { csv, metadata } = cubeToCsvw(obs);
96
+ // csv: string — the CSV text
97
+ // metadata: CsvwMetadata — the CSVW metadata object (JSON-serializable)
98
+ ```
99
+
100
+ The metadata object includes `@context`, `url`, `tableSchema` (columns with `titles`, `datatype`, `propertyUrl`), `primaryKey` (dimension IDs), and `dc:title`/`dc:source` from the dataset label/source.
101
+
102
+ ### CLI
103
+
104
+ ```sh
105
+ # JSON-stat → CSV + sibling CSVW metadata file
106
+ npx jsonstat-io ./sales.jsonstat.json --to csvw -o sales.csvw
107
+
108
+ # With -o, writes sales.csvw and sales-metadata.json
109
+ # Without -o, prints CSV + a separator + metadata JSON to stdout
110
+ ```
111
+
112
+ ## Import CLI
113
+
114
+ ```sh
115
+ # Auto-loads sibling *-metadata.json
116
+ npx jsonstat-io ./sales.csv --from csvw
117
+
118
+ # Inline metadata
119
+ npx jsonstat-io ./sales.csv --from csvw \
120
+ --csvw-metadata '{"tableSchema":{"columns":[{"titles":"year"},{"titles":"amount","datatype":"decimal"}],"primaryKey":["year"]}}'
121
+ ```
122
+
123
+ When `--from csvw` is set and no metadata is supplied, the CLI tries to load `sales-metadata.json` next to `sales.csv`. If not found, it throws with a helpful message suggesting `--from csv` instead.
124
+
125
+ ## Plain CSV (no metadata)
126
+
127
+ For plain CSV without a metadata file, use [`jsonstat-io/csv`](./csv.md) which infers the measure and dimensions heuristically:
128
+
129
+ ```ts
130
+ import { csvToCube } from "jsonstat-io/csv";
131
+
132
+ const obs = csvToCube(csvText, { measure: "amount" });
133
+ ```
134
+
135
+ See the [CSV adapter](../../src/sources/csv.ts) for inference rules: first all-numeric column = measure, `status`-named column = status, everything else = dimension.
136
+
137
+ ## Options
138
+
139
+ ### Import (`csvwToCube`)
140
+
141
+ | Field | Type | Description |
142
+ |--------------|------------|-------------|
143
+ | `measure` | `string` | Override the detected measure column |
144
+ | `dimensions` | `string[]` | Override dimension columns (overrides primaryKey) |
145
+ | `status` | `string` | Status column name |
146
+ | `roles` | `RoleMap` | Role assignments (overrides propertyUrl) |
147
+
148
+ ### Export (`CubeToCsvwOptions`)
149
+
150
+ | Field | Type | Default | Description |
151
+ |--------------|----------|---------|-------------|
152
+ | `delimiter` | `string` | `","` | CSV delimiter. |
@@ -0,0 +1,151 @@
1
+ # DuckDB Format
2
+
3
+ Run a SQL query against DuckDB and convert the result (an Arrow table) to JSON-stat via the [Arrow hub](./arrow.md). On export, register an Arrow-backed view or table into a DuckDB connection. Two optional peer backends are supported:
4
+
5
+ - **Node**: [`duckdb-async`](https://github.com/mortonreddy/duckdb-async) — native binding.
6
+ - **Browser**: [`@duckdb/duckdb-wasm`](https://github.com/duckdb/duckdb-wasm) — WASM.
7
+
8
+ ## Install
9
+
10
+ ```sh
11
+ # Node
12
+ npm install jsonstat-io duckdb-async
13
+
14
+ # Browser
15
+ npm install jsonstat-io @duckdb/duckdb-wasm
16
+ ```
17
+
18
+ Both are **optional peer dependencies**. The DuckDB adapter is **not** available via the CLI for import — it requires a live connection, so use the programmatic API.
19
+
20
+ ## Import path
21
+
22
+ ```ts
23
+ import { duckdbToCube, duckdbToDataset, openDuckdbNode } from "jsonstat-io/duckdb";
24
+ ```
25
+
26
+ ## Node example
27
+
28
+ ```ts
29
+ import { duckdbToDataset, openDuckdbNode } from "jsonstat-io/duckdb";
30
+
31
+ const conn = await openDuckdbNode("./warehouse.duckdb");
32
+ const dataset = await duckdbToDataset(conn,
33
+ "SELECT year, country, amount FROM sales WHERE year >= 2020",
34
+ { roles: { time: ["year"], geo: ["country"] } }
35
+ );
36
+ ```
37
+
38
+ `openDuckdbNode(path?)` is a convenience helper that lazily imports `duckdb-async` and returns a `DuckdbConnection`. Pass `":memory:"` (the default) for an in-memory database.
39
+
40
+ ## Browser example
41
+
42
+ Set up `@duckdb/duckdb-wasm` yourself, then pass the connection:
43
+
44
+ ```ts
45
+ import * as duckdb from "@duckdb/duckdb-wasm";
46
+ import { duckdbToDataset } from "jsonstat-io/duckdb";
47
+
48
+ // Standard duckdb-wasm bootstrap (see their docs for full setup)
49
+ const bundle = await duckdb.selectBundle(duckdb.getBundle());
50
+ const worker = new Worker(bundle.mainWorker);
51
+ const db = new duckdb.AsyncDuckDB(logger, worker);
52
+ await db.instantiate(bundle.mainModule, bundle.pthreadWorker);
53
+ const conn = await db.connect();
54
+
55
+ const dataset = await duckdbToDataset(conn, "SELECT * FROM sales");
56
+ ```
57
+
58
+ The adapter normalizes both `duckdb-async` and `duckdb-wasm` connection shapes — it looks for `.arrow()` or `.arrowResult()` methods.
59
+
60
+ ## Connection factory
61
+
62
+ For lazy browser setups, pass a `connect` factory instead of a live connection. The adapter opens and closes it automatically:
63
+
64
+ ```ts
65
+ const obs = await duckdbToCube(null, query, {
66
+ connect: async () => await createConnection(),
67
+ });
68
+ ```
69
+
70
+ ## Export path
71
+
72
+ [`cubeToDuckdb`](../../src/sources/duckdb.ts) converts the `Observations` IR to an Arrow table (via `cubeToArrow`) and registers it as a named view or table in the DuckDB connection.
73
+
74
+ ```ts
75
+ import { exportDataset } from "jsonstat-io";
76
+
77
+ // JSON-stat → Arrow-backed view in DuckDB
78
+ const table = await exportDataset(dataset, {
79
+ to: "duckdb",
80
+ connection: conn,
81
+ tableName: "sales",
82
+ mode: "view", // or "table" to materialize
83
+ });
84
+ ```
85
+
86
+ Or use `cubeToDuckdb` directly on the IR:
87
+
88
+ ```ts
89
+ import { cubeToDuckdb } from "jsonstat-io/duckdb";
90
+
91
+ await cubeToDuckdb(conn, obs, { tableName: "sales", mode: "view" });
92
+
93
+ // Now queryable in DuckDB:
94
+ const result = await conn.arrow("SELECT * FROM sales WHERE amount > 100");
95
+ ```
96
+
97
+ ### Export modes
98
+
99
+ | Mode | Behavior |
100
+ |--------|----------|
101
+ | `"view"` (default) | Registers a temporary Arrow-backed view — fast, zero-copy, lives for the connection's lifetime. |
102
+ | `"table"` | Materializes the Arrow data into a persistent DuckDB table via `CREATE TABLE ... AS SELECT * FROM <view>`. |
103
+
104
+ ## Options
105
+
106
+ ### `DuckdbToCubeOptions` (import)
107
+
108
+ Extends [`ArrowToCubeOptions`](./arrow.md):
109
+
110
+ | Field | Type | Description |
111
+ |----------|---------------------------------|-------------|
112
+ | `connect`| `() => Promise<DuckdbConnection>` | Factory that returns a fresh connection (browser). Takes precedence over `connection`, closed after use. |
113
+ | `measure`| `string` | Measure column name |
114
+ | `dimensions` | `string[]` | Dimension column names |
115
+ | `roles` | `RoleMap` | Role assignments |
116
+
117
+ ### `CubeToDuckdbOptions` (export)
118
+
119
+ | Field | Type | Default | Description |
120
+ |-------------|---------------------------|-----------|-------------|
121
+ | `tableName` | `string` | `"data"` | Name of the view/table to create. |
122
+ | `mode` | `"view" \| "table"` | `"view"` | Register a zero-copy view or materialize a table. |
123
+
124
+ ## DuckdbConnection interface
125
+
126
+ ```ts
127
+ interface DuckdbConnection {
128
+ arrow(query: string): Promise<any>; // Returns an Apache Arrow Table
129
+ close?(): Promise<void>;
130
+ // Export-side (registered by cubeToDuckdb):
131
+ register?(name: string, table: any): Promise<void>;
132
+ insert_arrow_table?(table: any, name: string): Promise<void>;
133
+ run?(sql: string): Promise<void>;
134
+ }
135
+ ```
136
+
137
+ Both `duckdb-async` and `@duckdb/duckdb-wasm` connections satisfy this interface. If your connection uses `arrowResult()` instead of `arrow()`, the adapter handles that too.
138
+
139
+ ## How it works
140
+
141
+ **Import:**
142
+ 1. `duckdbToCube` runs the SQL query via `conn.arrow(query)` (or `arrowResult()`).
143
+ 2. Gets back an Apache Arrow `Table`.
144
+ 3. Passes it to `arrowToCube` → `Observations` IR.
145
+
146
+ **Export:**
147
+ 1. `cubeToDuckdb` calls `cubeToArrow(obs)` → Apache Arrow `Table`.
148
+ 2. Registers the table into the connection via `conn.register(name, table)` (or `insert_arrow_table`).
149
+ 3. For `mode: "table"`, runs `CREATE TABLE <name> AS SELECT * FROM <tmpview>`.
150
+
151
+ If neither DuckDB package is installed, a `DuckdbSourceError` is thrown.
@@ -0,0 +1,132 @@
1
+ # Parquet Format
2
+
3
+ Parquet files are read via [`parquet-wasm`](https://github.com/kylebarron/parquet-wasm), which decodes Parquet into an Apache Arrow table — then the [Arrow hub](./arrow.md) takes over. On export, the same Arrow table is written back to Parquet bytes via `parquet-wasm`'s `writeParquet`.
4
+
5
+ ## Install
6
+
7
+ ```sh
8
+ npm install jsonstat-io parquet-wasm
9
+ ```
10
+
11
+ `parquet-wasm` is an **optional peer dependency**. It works in both Node and the browser.
12
+
13
+ ## Import path
14
+
15
+ ```ts
16
+ import { parquetToCube, parquetToDataset } from "jsonstat-io/parquet";
17
+ ```
18
+
19
+ Or use the high-level dispatcher (auto-detects Parquet from `PAR1` magic bytes):
20
+
21
+ ```ts
22
+ import { importToDataset } from "jsonstat-io";
23
+ const dataset = await importToDataset("./sales.parquet");
24
+ ```
25
+
26
+ ## Node example
27
+
28
+ ```ts
29
+ import { readFileSync } from "node:fs";
30
+ import { parquetToDataset } from "jsonstat-io/parquet";
31
+
32
+ const bytes = readFileSync("./sales.parquet");
33
+ const dataset = await parquetToDataset(bytes, {
34
+ measure: "amount",
35
+ roles: { time: ["year"], geo: ["country"] },
36
+ });
37
+ ```
38
+
39
+ ## Browser example
40
+
41
+ `parquet-wasm` needs its WASM binary loaded. Pass an `init` function to set the base URL or call `parquetWasm.init()`:
42
+
43
+ ```ts
44
+ import { parquetToDataset } from "jsonstat-io/parquet";
45
+
46
+ async function initParquet() {
47
+ const wasm = await import("parquet-wasm");
48
+ if (typeof wasm.init === "function") await wasm.init();
49
+ }
50
+
51
+ const bytes = new Uint8Array(await (await fetch("/sales.parquet")).arrayBuffer());
52
+ const dataset = await parquetToDataset(bytes, { init: initParquet });
53
+ ```
54
+
55
+ ## Export path
56
+
57
+ [`cubeToParquet`](../../src/sources/parquet.ts) converts the `Observations` IR to Parquet bytes: it calls `cubeToArrow` to build the Arrow table, then `parquet-wasm`'s `writeParquet` to serialize.
58
+
59
+ ```ts
60
+ import { exportDataset } from "jsonstat-io";
61
+
62
+ // JSON-stat → Parquet bytes
63
+ const bytes = await exportDataset(dataset, {
64
+ to: "parquet",
65
+ init: () => parquetWasm.init(),
66
+ });
67
+ ```
68
+
69
+ Or use `cubeToParquet` directly on the IR:
70
+
71
+ ```ts
72
+ import { cubeToParquet } from "jsonstat-io/parquet";
73
+
74
+ const bytes = await cubeToParquet(obs, {
75
+ init: () => parquetWasm.init(),
76
+ compression: "snappy",
77
+ });
78
+ ```
79
+
80
+ ### CLI
81
+
82
+ ```sh
83
+ # JSON-stat → Parquet
84
+ npx jsonstat-io ./sales.jsonstat.json --to parquet -o sales.parquet
85
+ ```
86
+
87
+ ## Import CLI
88
+
89
+ ```sh
90
+ npx jsonstat-io ./sales.parquet -o sales.jsonstat.json
91
+ npx jsonstat-io ./sales.parquet --from parquet --sparse
92
+ ```
93
+
94
+ The CLI auto-detects Parquet from the `PAR1` magic bytes, so `--from parquet` is optional.
95
+
96
+ ## Options
97
+
98
+ ### `ParquetToCubeOptions` (import)
99
+
100
+ Extends [`ArrowToCubeOptions`](./arrow.md):
101
+
102
+ | Field | Type | Description |
103
+ |-----------|----------------|-------------|
104
+ | `init` | `() => Promise<void>` | Async initializer for parquet-wasm (e.g. to set the WASM base URL in the browser). Awaited before reading. |
105
+ | `measure` | `string` | Measure column name |
106
+ | `dimensions` | `string[]` | Dimension column names |
107
+ | `roles` | `RoleMap` | Role assignments |
108
+
109
+ ### `CubeToParquetOptions` (export)
110
+
111
+ | Field | Type | Description |
112
+ |--------------|----------------|-------------|
113
+ | `init` | `() => Promise<void>` | Async initializer for parquet-wasm. |
114
+ | `compression`| `string` | Parquet compression codec (e.g. `"snappy"`, `"gzip"`). |
115
+
116
+ ## How it works
117
+
118
+ > **Dual-Arrow IPC bridge:** `parquet-wasm` bundles its own copy of `apache-arrow`, so an `instanceof Table` check against its internal `Table` class fails for our caller's `Table` (and vice versa). The adapter bridges this with IPC serialization — a format both copies understand:
119
+
120
+ **Import:**
121
+ 1. `parquetToCube` lazily imports `parquet-wasm`.
122
+ 2. Calls `wasm.readParquet(bytes)` → parquet-wasm `Table`.
123
+ 3. Serializes via `wasmTable.intoIPCStream()` → reconstructs with `tableFromIPC()` (our apache-arrow).
124
+ 4. Passes the reconstructed table to `arrowToCube` → `Observations` IR.
125
+
126
+ **Export:**
127
+ 1. `cubeToParquet` lazily imports `parquet-wasm`.
128
+ 2. Calls `cubeToArrow(obs)` → Apache Arrow `Table`.
129
+ 3. Serializes via `tableToIPC(table, "stream")` → reconstructs with `wasm.Table.fromIPCStream(ipc)` (parquet-wasm's `Table`).
130
+ 4. Calls `wasm.writeParquet(wasmTable, writerProps)` → `Uint8Array`.
131
+
132
+ If `parquet-wasm` is not installed, a `ParquetSourceError` is thrown with install instructions.