jsonstat-io 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +201 -0
- package/README.md +200 -0
- package/dist/arrow/index.cjs +60 -0
- package/dist/arrow/index.cjs.map +1 -0
- package/dist/arrow/index.d.cts +125 -0
- package/dist/arrow/index.d.ts +125 -0
- package/dist/arrow/index.js +3 -0
- package/dist/arrow/index.js.map +1 -0
- package/dist/arrowToCube-BdRWZ5oG.d.cts +29 -0
- package/dist/arrowToCube-Mjr79ZHM.d.ts +29 -0
- package/dist/chunk-2HY7NRKX.js +259 -0
- package/dist/chunk-2HY7NRKX.js.map +1 -0
- package/dist/chunk-62NIY7ML.js +191 -0
- package/dist/chunk-62NIY7ML.js.map +1 -0
- package/dist/chunk-AXUZF6O5.cjs +352 -0
- package/dist/chunk-AXUZF6O5.cjs.map +1 -0
- package/dist/chunk-EFRDFQOT.js +335 -0
- package/dist/chunk-EFRDFQOT.js.map +1 -0
- package/dist/chunk-GDKDYLZ4.cjs +263 -0
- package/dist/chunk-GDKDYLZ4.cjs.map +1 -0
- package/dist/chunk-IDTQZG4B.js +432 -0
- package/dist/chunk-IDTQZG4B.js.map +1 -0
- package/dist/chunk-KFIKGBIJ.cjs +200 -0
- package/dist/chunk-KFIKGBIJ.cjs.map +1 -0
- package/dist/chunk-L65CRXTV.js +64 -0
- package/dist/chunk-L65CRXTV.js.map +1 -0
- package/dist/chunk-PHLCWTHB.cjs +446 -0
- package/dist/chunk-PHLCWTHB.cjs.map +1 -0
- package/dist/chunk-VQCS4LVD.cjs +166 -0
- package/dist/chunk-VQCS4LVD.cjs.map +1 -0
- package/dist/chunk-VYS7DPB3.js +162 -0
- package/dist/chunk-VYS7DPB3.js.map +1 -0
- package/dist/chunk-ZEDTWRDN.cjs +70 -0
- package/dist/chunk-ZEDTWRDN.cjs.map +1 -0
- package/dist/cli/index.cjs +337 -0
- package/dist/cli/index.cjs.map +1 -0
- package/dist/cli/index.d.cts +90 -0
- package/dist/cli/index.d.ts +90 -0
- package/dist/cli/index.js +333 -0
- package/dist/cli/index.js.map +1 -0
- package/dist/cubeBuilder-BEGA4C5Y.js +4 -0
- package/dist/cubeBuilder-BEGA4C5Y.js.map +1 -0
- package/dist/cubeBuilder-ZU76VVEX.cjs +25 -0
- package/dist/cubeBuilder-ZU76VVEX.cjs.map +1 -0
- package/dist/cubeReader-7QVUF3TI.js +4 -0
- package/dist/cubeReader-7QVUF3TI.js.map +1 -0
- package/dist/cubeReader-UN2BHQWK.cjs +21 -0
- package/dist/cubeReader-UN2BHQWK.cjs.map +1 -0
- package/dist/index.cjs +172 -0
- package/dist/index.cjs.map +1 -0
- package/dist/index.d.cts +445 -0
- package/dist/index.d.ts +445 -0
- package/dist/index.js +7 -0
- package/dist/index.js.map +1 -0
- package/dist/ir-BArfNxo-.d.cts +143 -0
- package/dist/ir-Jae2ymov.d.ts +143 -0
- package/dist/jsonstat-BX4mBLci.d.cts +179 -0
- package/dist/jsonstat-BX4mBLci.d.ts +179 -0
- package/dist/sources/csv.cjs +40 -0
- package/dist/sources/csv.cjs.map +1 -0
- package/dist/sources/csv.d.cts +112 -0
- package/dist/sources/csv.d.ts +112 -0
- package/dist/sources/csv.js +3 -0
- package/dist/sources/csv.js.map +1 -0
- package/dist/sources/csvw.cjs +236 -0
- package/dist/sources/csvw.cjs.map +1 -0
- package/dist/sources/csvw.d.cts +143 -0
- package/dist/sources/csvw.d.ts +143 -0
- package/dist/sources/csvw.js +230 -0
- package/dist/sources/csvw.js.map +1 -0
- package/dist/sources/duckdb.cjs +121 -0
- package/dist/sources/duckdb.cjs.map +1 -0
- package/dist/sources/duckdb.d.cts +111 -0
- package/dist/sources/duckdb.d.ts +111 -0
- package/dist/sources/duckdb.js +115 -0
- package/dist/sources/duckdb.js.map +1 -0
- package/dist/sources/parquet.cjs +117 -0
- package/dist/sources/parquet.cjs.map +1 -0
- package/dist/sources/parquet.d.cts +83 -0
- package/dist/sources/parquet.d.ts +83 -0
- package/dist/sources/parquet.js +111 -0
- package/dist/sources/parquet.js.map +1 -0
- package/dist/sources/polars.cjs +94 -0
- package/dist/sources/polars.cjs.map +1 -0
- package/dist/sources/polars.d.cts +91 -0
- package/dist/sources/polars.d.ts +91 -0
- package/dist/sources/polars.js +87 -0
- package/dist/sources/polars.js.map +1 -0
- package/docs/api.md +322 -0
- package/docs/architecture.md +140 -0
- package/docs/cli.md +159 -0
- package/docs/formats/arrow.md +111 -0
- package/docs/formats/csv.md +115 -0
- package/docs/formats/csvw.md +152 -0
- package/docs/formats/duckdb.md +151 -0
- package/docs/formats/parquet.md +132 -0
- package/docs/formats/polars.md +111 -0
- package/docs/mapping.md +136 -0
- package/package.json +119 -0
|
@@ -0,0 +1,111 @@
|
|
|
1
|
+
# Arrow Format
|
|
2
|
+
|
|
3
|
+
Apache Arrow is the **hub** of `jsonstat-io`. Parquet, DuckDB, and Polars all produce Arrow tables, and a single converter pair ([`arrowToCube`](../../src/arrow/arrowToCube.ts) / [`cubeToArrow`](../../src/arrow/arrowFromCube.ts)) handles them all in both directions.
|
|
4
|
+
|
|
5
|
+
## Import path
|
|
6
|
+
|
|
7
|
+
```ts
|
|
8
|
+
import { arrowToCube, arrowToDataset } from "jsonstat-io";
|
|
9
|
+
// or from the dedicated subpath:
|
|
10
|
+
import { arrowToCube } from "jsonstat-io/arrow";
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
`apache-arrow` is a **hard dependency** — no peer install needed.
|
|
14
|
+
|
|
15
|
+
## Reading an Arrow IPC file
|
|
16
|
+
|
|
17
|
+
```ts
|
|
18
|
+
import { tableFromIPC } from "apache-arrow";
|
|
19
|
+
import { arrowToDataset } from "jsonstat-io";
|
|
20
|
+
import { readFileSync } from "node:fs";
|
|
21
|
+
|
|
22
|
+
const bytes = readFileSync("./data.arrow");
|
|
23
|
+
const table = tableFromIPC(bytes);
|
|
24
|
+
const dataset = arrowToDataset(table, { measure: "amount" });
|
|
25
|
+
```
|
|
26
|
+
|
|
27
|
+
In the browser, fetch the bytes and pass them directly:
|
|
28
|
+
|
|
29
|
+
```ts
|
|
30
|
+
const res = await fetch("/data.arrow");
|
|
31
|
+
const bytes = new Uint8Array(await res.arrayBuffer());
|
|
32
|
+
const table = tableFromIPC(bytes);
|
|
33
|
+
const dataset = arrowToDataset(table);
|
|
34
|
+
```
|
|
35
|
+
|
|
36
|
+
## How Arrow columns map
|
|
37
|
+
|
|
38
|
+
| Arrow column type | JSON-stat role | Detection |
|
|
39
|
+
|------------------------------|------------------|-----------|
|
|
40
|
+
| `Dictionary<Utf8, Int*>` | dimension | Always a dimension |
|
|
41
|
+
| `Utf8`, `Bool`, temporal | dimension | Stringified |
|
|
42
|
+
| `Float64`, `Float32`, `Int*` | measure (first) | First numeric column, unless overridden |
|
|
43
|
+
| Any column with `jsonstat.status: "true"` | status | Metadata marker |
|
|
44
|
+
|
|
45
|
+
### Without metadata (heuristic mode)
|
|
46
|
+
|
|
47
|
+
`arrowToCube` infers:
|
|
48
|
+
1. The first `Float64`/`Float32`/`Int64`/`Int32` column is the **measure**.
|
|
49
|
+
2. A column named `status` (case-insensitive) is the **status** column.
|
|
50
|
+
3. All other columns are **dimensions**, in schema order.
|
|
51
|
+
4. Roles: `year`/`date`/`time`/`period` → `time`; `country`/`region`/`geo`/`area` → `geo`; the measure → `metric`.
|
|
52
|
+
|
|
53
|
+
### With metadata (lossless mode)
|
|
54
|
+
|
|
55
|
+
Annotate Arrow `Field` and `Schema` metadata with `jsonstat.*` keys for exact, lossless mapping. See the [metadata key reference](../mapping.md#arrow-schema-metadata-contract).
|
|
56
|
+
|
|
57
|
+
```ts
|
|
58
|
+
import { Field, Schema, Dictionary, Utf8, Int32, Float64 } from "apache-arrow";
|
|
59
|
+
import { buildFieldMeta, buildSchemaMeta } from "jsonstat-io";
|
|
60
|
+
|
|
61
|
+
const yearField = new Field("year", new Dictionary(new Utf8(), new Int32()), false,
|
|
62
|
+
new Map(Object.entries(buildFieldMeta({
|
|
63
|
+
role: "time",
|
|
64
|
+
label: "Year",
|
|
65
|
+
categoryOrder: ["2020", "2021", "2022"],
|
|
66
|
+
categoryLabels: { "2020": "FY 2020", "2021": "FY 2021" },
|
|
67
|
+
})))
|
|
68
|
+
);
|
|
69
|
+
|
|
70
|
+
const schema = new Schema(
|
|
71
|
+
[yearField, /* ... */],
|
|
72
|
+
new Map(Object.entries(buildSchemaMeta({ label: "Annual Sales", source: "Finance Dept" })))
|
|
73
|
+
);
|
|
74
|
+
```
|
|
75
|
+
|
|
76
|
+
## Annotating for round-trip fidelity
|
|
77
|
+
|
|
78
|
+
To guarantee that `arrowToCube` → `cubeToArrow` round-trips losslessly, annotate:
|
|
79
|
+
|
|
80
|
+
- **Schema**: `jsonstat.label`, `jsonstat.source`, `jsonstat.updated`, `jsonstat.valueForm`.
|
|
81
|
+
- **Each dimension field**: `jsonstat.role`, `jsonstat.label`, `jsonstat.categoryLabels`, `jsonstat.categoryUnits`, `jsonstat.categoryCoords`, `jsonstat.categoryChild`, `jsonstat.categoryOrder`.
|
|
82
|
+
- **Measure field**: `jsonstat.measure: "true"` (or `jsonstat.role: "metric"`).
|
|
83
|
+
- **Status field**: `jsonstat.status: "true"`.
|
|
84
|
+
|
|
85
|
+
## Export path
|
|
86
|
+
|
|
87
|
+
[`cubeToArrow`](../../src/arrow/arrowFromCube.ts) converts the `Observations` IR back to an Arrow `Table`, building dictionary-encoded dimensions and a Float64 measure with full `jsonstat.*` metadata annotation.
|
|
88
|
+
|
|
89
|
+
```ts
|
|
90
|
+
import { exportDataset } from "jsonstat-io";
|
|
91
|
+
|
|
92
|
+
// JSON-stat → Arrow Table
|
|
93
|
+
const table = await exportDataset(dataset, { to: "arrow" });
|
|
94
|
+
```
|
|
95
|
+
|
|
96
|
+
Or use `cubeToArrow` directly on the IR:
|
|
97
|
+
|
|
98
|
+
```ts
|
|
99
|
+
import { cubeToArrow } from "jsonstat-io";
|
|
100
|
+
// obs = Observations IR
|
|
101
|
+
const table = cubeToArrow(obs);
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
### CLI
|
|
105
|
+
|
|
106
|
+
```sh
|
|
107
|
+
# JSON-stat → Arrow IPC file
|
|
108
|
+
npx jsonstat-io ./data.jsonstat.json --to arrow -o data.arrow
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
The exporter writes Arrow IPC stream format. `cubeToArrow` is also the foundation for the Parquet, DuckDB, and Polars exporters — they all call `cubeToArrow` first, then convert the resulting Arrow Table to their target format.
|
|
@@ -0,0 +1,115 @@
|
|
|
1
|
+
# CSV Format
|
|
2
|
+
|
|
3
|
+
Plain CSV (no metadata) is the simplest path. The [`csvToCube`](../../src/sources/csv.ts) adapter infers the measure and dimensions heuristically on import. On export, [`cubeToCsv`](../../src/sources/csv.ts) serializes the IR to CSV text. It is **dependency-free** and works in both Node and the browser.
|
|
4
|
+
|
|
5
|
+
## Import path
|
|
6
|
+
|
|
7
|
+
```ts
|
|
8
|
+
import { csvToCube, parseCsv } from "jsonstat-io/csv";
|
|
9
|
+
```
|
|
10
|
+
|
|
11
|
+
Or use the high-level dispatcher:
|
|
12
|
+
|
|
13
|
+
```ts
|
|
14
|
+
import { importToDataset } from "jsonstat-io";
|
|
15
|
+
const dataset = await importToDataset("./data.csv", { from: "csv", measure: "amount" });
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
## Inference heuristics (no metadata)
|
|
19
|
+
|
|
20
|
+
Without a schema, the adapter infers:
|
|
21
|
+
|
|
22
|
+
1. The **measure** is the first column that parses as a number for every non-empty row — or the one named by `options.measure`.
|
|
23
|
+
2. A column named `status` (case-insensitive) is the **status** column — unless `options.status` overrides it.
|
|
24
|
+
3. Every other column is a **dimension** (string-typed), in file order.
|
|
25
|
+
|
|
26
|
+
Empty cells become `null` in the measure and `""` in dimensions. For richer, lossless mapping with declared types/labels/roles, use [CSVW](./csvw.md) instead.
|
|
27
|
+
|
|
28
|
+
## Import example
|
|
29
|
+
|
|
30
|
+
```ts
|
|
31
|
+
import { csvToCube } from "jsonstat-io/csv";
|
|
32
|
+
import { buildDataset } from "jsonstat-io";
|
|
33
|
+
|
|
34
|
+
const csvText = `year,country,amount
|
|
35
|
+
2020,Spain,100
|
|
36
|
+
2020,France,200
|
|
37
|
+
2021,Spain,150
|
|
38
|
+
2021,France,250`;
|
|
39
|
+
|
|
40
|
+
const obs = csvToCube(csvText, {
|
|
41
|
+
measure: "amount",
|
|
42
|
+
roles: { time: ["year"], geo: ["country"] },
|
|
43
|
+
});
|
|
44
|
+
|
|
45
|
+
const { dataset } = buildDataset(obs, { valueForm: "dense" });
|
|
46
|
+
```
|
|
47
|
+
|
|
48
|
+
## Export path
|
|
49
|
+
|
|
50
|
+
[`cubeToCsv`](../../src/sources/csv.ts) serializes the `Observations` IR to CSV text. The output has one column per dimension (in `id[]` order), followed by the measure column, and an optional status column.
|
|
51
|
+
|
|
52
|
+
```ts
|
|
53
|
+
import { exportDataset } from "jsonstat-io";
|
|
54
|
+
|
|
55
|
+
// JSON-stat → CSV text
|
|
56
|
+
const csv = await exportDataset(dataset, { to: "csv" });
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
Or use `cubeToCsv` directly on the IR:
|
|
60
|
+
|
|
61
|
+
```ts
|
|
62
|
+
import { cubeToCsv } from "jsonstat-io/csv";
|
|
63
|
+
|
|
64
|
+
const csv = cubeToCsv(obs, { delimiter: ";" });
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
### CLI
|
|
68
|
+
|
|
69
|
+
```sh
|
|
70
|
+
# JSON-stat → CSV
|
|
71
|
+
npx jsonstat-io ./data.jsonstat.json --to csv -o data.csv
|
|
72
|
+
|
|
73
|
+
# With custom delimiter via CSVW metadata (CSV itself has no delimiter flag)
|
|
74
|
+
npx jsonstat-io ./data.jsonstat.json --to csvw -o data.csvw --delimiter ";"
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
## Import CLI
|
|
78
|
+
|
|
79
|
+
```sh
|
|
80
|
+
# From a file
|
|
81
|
+
npx jsonstat-io ./data.csv --from csv --measure amount --role time=year,geo=country
|
|
82
|
+
|
|
83
|
+
# From stdin
|
|
84
|
+
cat data.csv | npx jsonstat-io - --measure amount
|
|
85
|
+
|
|
86
|
+
# Custom delimiter
|
|
87
|
+
npx jsonstat-io ./data.csv --delimiter ";"
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
When `--from` is omitted, `.csv` extension triggers CSV detection. The CLI checks for a sibling `*-metadata.json` first; if found, it uses CSVW, otherwise plain CSV.
|
|
91
|
+
|
|
92
|
+
## Options
|
|
93
|
+
|
|
94
|
+
### `CsvToCubeOptions` (import)
|
|
95
|
+
|
|
96
|
+
| Field | Type | Default | Description |
|
|
97
|
+
|--------------|-----------------------------------|----------|-------------|
|
|
98
|
+
| `measure` | `string` | *(inferred)* | Explicit measure column name |
|
|
99
|
+
| `dimensions` | `string[]` | *(inferred)* | Explicit dimension column names |
|
|
100
|
+
| `status` | `string` | *(inferred)* | Status column name |
|
|
101
|
+
| `roles` | `RoleMap` | — | Role assignments |
|
|
102
|
+
| `valueForm` | `"auto" \| "dense" \| "sparse"` | `"auto"` | Value form hint |
|
|
103
|
+
| `delimiter` | `string` | `","` | CSV delimiter |
|
|
104
|
+
| `header` | `boolean` | `true` | Treat first row as header |
|
|
105
|
+
|
|
106
|
+
### `CubeToCsvOptions` (export)
|
|
107
|
+
|
|
108
|
+
| Field | Type | Default | Description |
|
|
109
|
+
|-----------------|----------|---------|-------------|
|
|
110
|
+
| `delimiter` | `string` | `","` | CSV delimiter. |
|
|
111
|
+
| `lineTerminator`| `string` | `"\n"` | Row separator. |
|
|
112
|
+
|
|
113
|
+
## Parser
|
|
114
|
+
|
|
115
|
+
The bundled `parseCsv(input, delimiter?)` is a tiny, dependency-free RFC-4180-ish CSV parser. It handles quoted fields, embedded quotes (doubled), and embedded newlines. For heavy CSV work (large files, complex quoting), prefer piping through DuckDB or Polars.
|
|
@@ -0,0 +1,152 @@
|
|
|
1
|
+
# CSVW Format
|
|
2
|
+
|
|
3
|
+
[CSV on the Web (CSVW)](https://www.w3.org/TR/tabular-metadata/) provides a metadata layer over plain CSV, declaring column types, names, and roles. This enables **lossless** CSV → JSON-stat mapping without heuristics. On export, the IR is serialized to CSV text plus a CSVW metadata JSON document.
|
|
4
|
+
|
|
5
|
+
CSVW is a **dependency-free** path — no optional peers needed. It works in both Node and the browser.
|
|
6
|
+
|
|
7
|
+
## Import path
|
|
8
|
+
|
|
9
|
+
```ts
|
|
10
|
+
import { csvwToCube, csvwToDataset, parseCsvwMetadata } from "jsonstat-io/csvw";
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
Or use the high-level dispatcher:
|
|
14
|
+
|
|
15
|
+
```ts
|
|
16
|
+
import { importToDataset } from "jsonstat-io";
|
|
17
|
+
|
|
18
|
+
// If a sibling *-metadata.json exists next to the CSV (Node):
|
|
19
|
+
const dataset = await importToDataset("./data.csv", { from: "csvw" });
|
|
20
|
+
|
|
21
|
+
// Or pass metadata inline:
|
|
22
|
+
const dataset = await importToDataset("./data.csv", {
|
|
23
|
+
from: "csvw",
|
|
24
|
+
csvwMetadata: metadataObject,
|
|
25
|
+
});
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
## Metadata structure
|
|
29
|
+
|
|
30
|
+
`csvwToCube` expects a parsed CSVW metadata object with this shape:
|
|
31
|
+
|
|
32
|
+
```json
|
|
33
|
+
{
|
|
34
|
+
"tableSchema": {
|
|
35
|
+
"columns": [
|
|
36
|
+
{ "titles": "year", "datatype": "string", "propertyUrl": "http://example.org/#time" },
|
|
37
|
+
{ "titles": "country", "datatype": "string" },
|
|
38
|
+
{ "titles": "amount", "datatype": "decimal" }
|
|
39
|
+
],
|
|
40
|
+
"primaryKey": ["year", "country"]
|
|
41
|
+
}
|
|
42
|
+
}
|
|
43
|
+
```
|
|
44
|
+
|
|
45
|
+
### Column mapping
|
|
46
|
+
|
|
47
|
+
| CSVW property | Maps to |
|
|
48
|
+
|---------------------------------------|---------|
|
|
49
|
+
| `titles` (first) | Column name |
|
|
50
|
+
| `datatype` = `decimal`/`integer`/`double` | Measure column |
|
|
51
|
+
| Other `datatype` | Dimension column |
|
|
52
|
+
| `propertyUrl` containing `#time` | `time` role |
|
|
53
|
+
| `propertyUrl` containing `#geo` | `geo` role |
|
|
54
|
+
| `primaryKey` | Dimension columns (if no explicit `options.dimensions`) |
|
|
55
|
+
|
|
56
|
+
### Role detection from `propertyUrl`
|
|
57
|
+
|
|
58
|
+
The adapter checks if the `propertyUrl` contains a fragment hint:
|
|
59
|
+
|
|
60
|
+
- `...#time` or `...#temporal` → `time` role
|
|
61
|
+
- `...#geo` or `...#spatial` → `geo` role
|
|
62
|
+
|
|
63
|
+
## Import example
|
|
64
|
+
|
|
65
|
+
```ts
|
|
66
|
+
import { csvwToDataset, parseCsvwMetadata } from "jsonstat-io/csvw";
|
|
67
|
+
import { readFileSync } from "node:fs";
|
|
68
|
+
|
|
69
|
+
const csvText = readFileSync("./sales.csv", "utf8");
|
|
70
|
+
const metadataJson = JSON.parse(readFileSync("./sales-metadata.json", "utf8"));
|
|
71
|
+
const metadata = parseCsvwMetadata(metadataJson);
|
|
72
|
+
|
|
73
|
+
const dataset = csvwToDataset(csvText, metadata, {
|
|
74
|
+
dimensions: ["year", "country"], // optional: overrides primaryKey
|
|
75
|
+
roles: { time: ["year"] }, // optional: overrides propertyUrl
|
|
76
|
+
});
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
## Export path
|
|
80
|
+
|
|
81
|
+
[`cubeToCsvw`](../../src/sources/csvw.ts) serializes the `Observations` IR to a CSV text string **plus** a CSVW metadata JSON object, so the pair round-trips losslessly back through `csvwToCube`.
|
|
82
|
+
|
|
83
|
+
```ts
|
|
84
|
+
import { exportDataset } from "jsonstat-io";
|
|
85
|
+
|
|
86
|
+
// JSON-stat → CSV text + CSVW metadata
|
|
87
|
+
const { csv, metadata } = await exportDataset(dataset, { to: "csvw" });
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
Or use `cubeToCsvw` directly on the IR:
|
|
91
|
+
|
|
92
|
+
```ts
|
|
93
|
+
import { cubeToCsvw } from "jsonstat-io/csvw";
|
|
94
|
+
|
|
95
|
+
const { csv, metadata } = cubeToCsvw(obs);
|
|
96
|
+
// csv: string — the CSV text
|
|
97
|
+
// metadata: CsvwMetadata — the CSVW metadata object (JSON-serializable)
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
The metadata object includes `@context`, `url`, `tableSchema` (columns with `titles`, `datatype`, `propertyUrl`), `primaryKey` (dimension IDs), and `dc:title`/`dc:source` from the dataset label/source.
|
|
101
|
+
|
|
102
|
+
### CLI
|
|
103
|
+
|
|
104
|
+
```sh
|
|
105
|
+
# JSON-stat → CSV + sibling CSVW metadata file
|
|
106
|
+
npx jsonstat-io ./sales.jsonstat.json --to csvw -o sales.csvw
|
|
107
|
+
|
|
108
|
+
# With -o, writes sales.csvw and sales-metadata.json
|
|
109
|
+
# Without -o, prints CSV + a separator + metadata JSON to stdout
|
|
110
|
+
```
|
|
111
|
+
|
|
112
|
+
## Import CLI
|
|
113
|
+
|
|
114
|
+
```sh
|
|
115
|
+
# Auto-loads sibling *-metadata.json
|
|
116
|
+
npx jsonstat-io ./sales.csv --from csvw
|
|
117
|
+
|
|
118
|
+
# Inline metadata
|
|
119
|
+
npx jsonstat-io ./sales.csv --from csvw \
|
|
120
|
+
--csvw-metadata '{"tableSchema":{"columns":[{"titles":"year"},{"titles":"amount","datatype":"decimal"}],"primaryKey":["year"]}}'
|
|
121
|
+
```
|
|
122
|
+
|
|
123
|
+
When `--from csvw` is set and no metadata is supplied, the CLI tries to load `sales-metadata.json` next to `sales.csv`. If not found, it throws with a helpful message suggesting `--from csv` instead.
|
|
124
|
+
|
|
125
|
+
## Plain CSV (no metadata)
|
|
126
|
+
|
|
127
|
+
For plain CSV without a metadata file, use [`jsonstat-io/csv`](./csv.md) which infers the measure and dimensions heuristically:
|
|
128
|
+
|
|
129
|
+
```ts
|
|
130
|
+
import { csvToCube } from "jsonstat-io/csv";
|
|
131
|
+
|
|
132
|
+
const obs = csvToCube(csvText, { measure: "amount" });
|
|
133
|
+
```
|
|
134
|
+
|
|
135
|
+
See the [CSV adapter](../../src/sources/csv.ts) for inference rules: first all-numeric column = measure, `status`-named column = status, everything else = dimension.
|
|
136
|
+
|
|
137
|
+
## Options
|
|
138
|
+
|
|
139
|
+
### Import (`csvwToCube`)
|
|
140
|
+
|
|
141
|
+
| Field | Type | Description |
|
|
142
|
+
|--------------|------------|-------------|
|
|
143
|
+
| `measure` | `string` | Override the detected measure column |
|
|
144
|
+
| `dimensions` | `string[]` | Override dimension columns (overrides primaryKey) |
|
|
145
|
+
| `status` | `string` | Status column name |
|
|
146
|
+
| `roles` | `RoleMap` | Role assignments (overrides propertyUrl) |
|
|
147
|
+
|
|
148
|
+
### Export (`CubeToCsvwOptions`)
|
|
149
|
+
|
|
150
|
+
| Field | Type | Default | Description |
|
|
151
|
+
|--------------|----------|---------|-------------|
|
|
152
|
+
| `delimiter` | `string` | `","` | CSV delimiter. |
|
|
@@ -0,0 +1,151 @@
|
|
|
1
|
+
# DuckDB Format
|
|
2
|
+
|
|
3
|
+
Run a SQL query against DuckDB and convert the result (an Arrow table) to JSON-stat via the [Arrow hub](./arrow.md). On export, register an Arrow-backed view or table into a DuckDB connection. Two optional peer backends are supported:
|
|
4
|
+
|
|
5
|
+
- **Node**: [`duckdb-async`](https://github.com/mortonreddy/duckdb-async) — native binding.
|
|
6
|
+
- **Browser**: [`@duckdb/duckdb-wasm`](https://github.com/duckdb/duckdb-wasm) — WASM.
|
|
7
|
+
|
|
8
|
+
## Install
|
|
9
|
+
|
|
10
|
+
```sh
|
|
11
|
+
# Node
|
|
12
|
+
npm install jsonstat-io duckdb-async
|
|
13
|
+
|
|
14
|
+
# Browser
|
|
15
|
+
npm install jsonstat-io @duckdb/duckdb-wasm
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
Both are **optional peer dependencies**. The DuckDB adapter is **not** available via the CLI for import — it requires a live connection, so use the programmatic API.
|
|
19
|
+
|
|
20
|
+
## Import path
|
|
21
|
+
|
|
22
|
+
```ts
|
|
23
|
+
import { duckdbToCube, duckdbToDataset, openDuckdbNode } from "jsonstat-io/duckdb";
|
|
24
|
+
```
|
|
25
|
+
|
|
26
|
+
## Node example
|
|
27
|
+
|
|
28
|
+
```ts
|
|
29
|
+
import { duckdbToDataset, openDuckdbNode } from "jsonstat-io/duckdb";
|
|
30
|
+
|
|
31
|
+
const conn = await openDuckdbNode("./warehouse.duckdb");
|
|
32
|
+
const dataset = await duckdbToDataset(conn,
|
|
33
|
+
"SELECT year, country, amount FROM sales WHERE year >= 2020",
|
|
34
|
+
{ roles: { time: ["year"], geo: ["country"] } }
|
|
35
|
+
);
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
`openDuckdbNode(path?)` is a convenience helper that lazily imports `duckdb-async` and returns a `DuckdbConnection`. Pass `":memory:"` (the default) for an in-memory database.
|
|
39
|
+
|
|
40
|
+
## Browser example
|
|
41
|
+
|
|
42
|
+
Set up `@duckdb/duckdb-wasm` yourself, then pass the connection:
|
|
43
|
+
|
|
44
|
+
```ts
|
|
45
|
+
import * as duckdb from "@duckdb/duckdb-wasm";
|
|
46
|
+
import { duckdbToDataset } from "jsonstat-io/duckdb";
|
|
47
|
+
|
|
48
|
+
// Standard duckdb-wasm bootstrap (see their docs for full setup)
|
|
49
|
+
const bundle = await duckdb.selectBundle(duckdb.getBundle());
|
|
50
|
+
const worker = new Worker(bundle.mainWorker);
|
|
51
|
+
const db = new duckdb.AsyncDuckDB(logger, worker);
|
|
52
|
+
await db.instantiate(bundle.mainModule, bundle.pthreadWorker);
|
|
53
|
+
const conn = await db.connect();
|
|
54
|
+
|
|
55
|
+
const dataset = await duckdbToDataset(conn, "SELECT * FROM sales");
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
The adapter normalizes both `duckdb-async` and `duckdb-wasm` connection shapes — it looks for `.arrow()` or `.arrowResult()` methods.
|
|
59
|
+
|
|
60
|
+
## Connection factory
|
|
61
|
+
|
|
62
|
+
For lazy browser setups, pass a `connect` factory instead of a live connection. The adapter opens and closes it automatically:
|
|
63
|
+
|
|
64
|
+
```ts
|
|
65
|
+
const obs = await duckdbToCube(null, query, {
|
|
66
|
+
connect: async () => await createConnection(),
|
|
67
|
+
});
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
## Export path
|
|
71
|
+
|
|
72
|
+
[`cubeToDuckdb`](../../src/sources/duckdb.ts) converts the `Observations` IR to an Arrow table (via `cubeToArrow`) and registers it as a named view or table in the DuckDB connection.
|
|
73
|
+
|
|
74
|
+
```ts
|
|
75
|
+
import { exportDataset } from "jsonstat-io";
|
|
76
|
+
|
|
77
|
+
// JSON-stat → Arrow-backed view in DuckDB
|
|
78
|
+
const table = await exportDataset(dataset, {
|
|
79
|
+
to: "duckdb",
|
|
80
|
+
connection: conn,
|
|
81
|
+
tableName: "sales",
|
|
82
|
+
mode: "view", // or "table" to materialize
|
|
83
|
+
});
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
Or use `cubeToDuckdb` directly on the IR:
|
|
87
|
+
|
|
88
|
+
```ts
|
|
89
|
+
import { cubeToDuckdb } from "jsonstat-io/duckdb";
|
|
90
|
+
|
|
91
|
+
await cubeToDuckdb(conn, obs, { tableName: "sales", mode: "view" });
|
|
92
|
+
|
|
93
|
+
// Now queryable in DuckDB:
|
|
94
|
+
const result = await conn.arrow("SELECT * FROM sales WHERE amount > 100");
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
### Export modes
|
|
98
|
+
|
|
99
|
+
| Mode | Behavior |
|
|
100
|
+
|--------|----------|
|
|
101
|
+
| `"view"` (default) | Registers a temporary Arrow-backed view — fast, zero-copy, lives for the connection's lifetime. |
|
|
102
|
+
| `"table"` | Materializes the Arrow data into a persistent DuckDB table via `CREATE TABLE ... AS SELECT * FROM <view>`. |
|
|
103
|
+
|
|
104
|
+
## Options
|
|
105
|
+
|
|
106
|
+
### `DuckdbToCubeOptions` (import)
|
|
107
|
+
|
|
108
|
+
Extends [`ArrowToCubeOptions`](./arrow.md):
|
|
109
|
+
|
|
110
|
+
| Field | Type | Description |
|
|
111
|
+
|----------|---------------------------------|-------------|
|
|
112
|
+
| `connect`| `() => Promise<DuckdbConnection>` | Factory that returns a fresh connection (browser). Takes precedence over `connection`, closed after use. |
|
|
113
|
+
| `measure`| `string` | Measure column name |
|
|
114
|
+
| `dimensions` | `string[]` | Dimension column names |
|
|
115
|
+
| `roles` | `RoleMap` | Role assignments |
|
|
116
|
+
|
|
117
|
+
### `CubeToDuckdbOptions` (export)
|
|
118
|
+
|
|
119
|
+
| Field | Type | Default | Description |
|
|
120
|
+
|-------------|---------------------------|-----------|-------------|
|
|
121
|
+
| `tableName` | `string` | `"data"` | Name of the view/table to create. |
|
|
122
|
+
| `mode` | `"view" \| "table"` | `"view"` | Register a zero-copy view or materialize a table. |
|
|
123
|
+
|
|
124
|
+
## DuckdbConnection interface
|
|
125
|
+
|
|
126
|
+
```ts
|
|
127
|
+
interface DuckdbConnection {
|
|
128
|
+
arrow(query: string): Promise<any>; // Returns an Apache Arrow Table
|
|
129
|
+
close?(): Promise<void>;
|
|
130
|
+
// Export-side (registered by cubeToDuckdb):
|
|
131
|
+
register?(name: string, table: any): Promise<void>;
|
|
132
|
+
insert_arrow_table?(table: any, name: string): Promise<void>;
|
|
133
|
+
run?(sql: string): Promise<void>;
|
|
134
|
+
}
|
|
135
|
+
```
|
|
136
|
+
|
|
137
|
+
Both `duckdb-async` and `@duckdb/duckdb-wasm` connections satisfy this interface. If your connection uses `arrowResult()` instead of `arrow()`, the adapter handles that too.
|
|
138
|
+
|
|
139
|
+
## How it works
|
|
140
|
+
|
|
141
|
+
**Import:**
|
|
142
|
+
1. `duckdbToCube` runs the SQL query via `conn.arrow(query)` (or `arrowResult()`).
|
|
143
|
+
2. Gets back an Apache Arrow `Table`.
|
|
144
|
+
3. Passes it to `arrowToCube` → `Observations` IR.
|
|
145
|
+
|
|
146
|
+
**Export:**
|
|
147
|
+
1. `cubeToDuckdb` calls `cubeToArrow(obs)` → Apache Arrow `Table`.
|
|
148
|
+
2. Registers the table into the connection via `conn.register(name, table)` (or `insert_arrow_table`).
|
|
149
|
+
3. For `mode: "table"`, runs `CREATE TABLE <name> AS SELECT * FROM <tmpview>`.
|
|
150
|
+
|
|
151
|
+
If neither DuckDB package is installed, a `DuckdbSourceError` is thrown.
|
|
@@ -0,0 +1,132 @@
|
|
|
1
|
+
# Parquet Format
|
|
2
|
+
|
|
3
|
+
Parquet files are read via [`parquet-wasm`](https://github.com/kylebarron/parquet-wasm), which decodes Parquet into an Apache Arrow table — then the [Arrow hub](./arrow.md) takes over. On export, the same Arrow table is written back to Parquet bytes via `parquet-wasm`'s `writeParquet`.
|
|
4
|
+
|
|
5
|
+
## Install
|
|
6
|
+
|
|
7
|
+
```sh
|
|
8
|
+
npm install jsonstat-io parquet-wasm
|
|
9
|
+
```
|
|
10
|
+
|
|
11
|
+
`parquet-wasm` is an **optional peer dependency**. It works in both Node and the browser.
|
|
12
|
+
|
|
13
|
+
## Import path
|
|
14
|
+
|
|
15
|
+
```ts
|
|
16
|
+
import { parquetToCube, parquetToDataset } from "jsonstat-io/parquet";
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
Or use the high-level dispatcher (auto-detects Parquet from `PAR1` magic bytes):
|
|
20
|
+
|
|
21
|
+
```ts
|
|
22
|
+
import { importToDataset } from "jsonstat-io";
|
|
23
|
+
const dataset = await importToDataset("./sales.parquet");
|
|
24
|
+
```
|
|
25
|
+
|
|
26
|
+
## Node example
|
|
27
|
+
|
|
28
|
+
```ts
|
|
29
|
+
import { readFileSync } from "node:fs";
|
|
30
|
+
import { parquetToDataset } from "jsonstat-io/parquet";
|
|
31
|
+
|
|
32
|
+
const bytes = readFileSync("./sales.parquet");
|
|
33
|
+
const dataset = await parquetToDataset(bytes, {
|
|
34
|
+
measure: "amount",
|
|
35
|
+
roles: { time: ["year"], geo: ["country"] },
|
|
36
|
+
});
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
## Browser example
|
|
40
|
+
|
|
41
|
+
`parquet-wasm` needs its WASM binary loaded. Pass an `init` function to set the base URL or call `parquetWasm.init()`:
|
|
42
|
+
|
|
43
|
+
```ts
|
|
44
|
+
import { parquetToDataset } from "jsonstat-io/parquet";
|
|
45
|
+
|
|
46
|
+
async function initParquet() {
|
|
47
|
+
const wasm = await import("parquet-wasm");
|
|
48
|
+
if (typeof wasm.init === "function") await wasm.init();
|
|
49
|
+
}
|
|
50
|
+
|
|
51
|
+
const bytes = new Uint8Array(await (await fetch("/sales.parquet")).arrayBuffer());
|
|
52
|
+
const dataset = await parquetToDataset(bytes, { init: initParquet });
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
## Export path
|
|
56
|
+
|
|
57
|
+
[`cubeToParquet`](../../src/sources/parquet.ts) converts the `Observations` IR to Parquet bytes: it calls `cubeToArrow` to build the Arrow table, then `parquet-wasm`'s `writeParquet` to serialize.
|
|
58
|
+
|
|
59
|
+
```ts
|
|
60
|
+
import { exportDataset } from "jsonstat-io";
|
|
61
|
+
|
|
62
|
+
// JSON-stat → Parquet bytes
|
|
63
|
+
const bytes = await exportDataset(dataset, {
|
|
64
|
+
to: "parquet",
|
|
65
|
+
init: () => parquetWasm.init(),
|
|
66
|
+
});
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
Or use `cubeToParquet` directly on the IR:
|
|
70
|
+
|
|
71
|
+
```ts
|
|
72
|
+
import { cubeToParquet } from "jsonstat-io/parquet";
|
|
73
|
+
|
|
74
|
+
const bytes = await cubeToParquet(obs, {
|
|
75
|
+
init: () => parquetWasm.init(),
|
|
76
|
+
compression: "snappy",
|
|
77
|
+
});
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
### CLI
|
|
81
|
+
|
|
82
|
+
```sh
|
|
83
|
+
# JSON-stat → Parquet
|
|
84
|
+
npx jsonstat-io ./sales.jsonstat.json --to parquet -o sales.parquet
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
## Import CLI
|
|
88
|
+
|
|
89
|
+
```sh
|
|
90
|
+
npx jsonstat-io ./sales.parquet -o sales.jsonstat.json
|
|
91
|
+
npx jsonstat-io ./sales.parquet --from parquet --sparse
|
|
92
|
+
```
|
|
93
|
+
|
|
94
|
+
The CLI auto-detects Parquet from the `PAR1` magic bytes, so `--from parquet` is optional.
|
|
95
|
+
|
|
96
|
+
## Options
|
|
97
|
+
|
|
98
|
+
### `ParquetToCubeOptions` (import)
|
|
99
|
+
|
|
100
|
+
Extends [`ArrowToCubeOptions`](./arrow.md):
|
|
101
|
+
|
|
102
|
+
| Field | Type | Description |
|
|
103
|
+
|-----------|----------------|-------------|
|
|
104
|
+
| `init` | `() => Promise<void>` | Async initializer for parquet-wasm (e.g. to set the WASM base URL in the browser). Awaited before reading. |
|
|
105
|
+
| `measure` | `string` | Measure column name |
|
|
106
|
+
| `dimensions` | `string[]` | Dimension column names |
|
|
107
|
+
| `roles` | `RoleMap` | Role assignments |
|
|
108
|
+
|
|
109
|
+
### `CubeToParquetOptions` (export)
|
|
110
|
+
|
|
111
|
+
| Field | Type | Description |
|
|
112
|
+
|--------------|----------------|-------------|
|
|
113
|
+
| `init` | `() => Promise<void>` | Async initializer for parquet-wasm. |
|
|
114
|
+
| `compression`| `string` | Parquet compression codec (e.g. `"snappy"`, `"gzip"`). |
|
|
115
|
+
|
|
116
|
+
## How it works
|
|
117
|
+
|
|
118
|
+
> **Dual-Arrow IPC bridge:** `parquet-wasm` bundles its own copy of `apache-arrow`, so an `instanceof Table` check against its internal `Table` class fails for our caller's `Table` (and vice versa). The adapter bridges this with IPC serialization — a format both copies understand:
|
|
119
|
+
|
|
120
|
+
**Import:**
|
|
121
|
+
1. `parquetToCube` lazily imports `parquet-wasm`.
|
|
122
|
+
2. Calls `wasm.readParquet(bytes)` → parquet-wasm `Table`.
|
|
123
|
+
3. Serializes via `wasmTable.intoIPCStream()` → reconstructs with `tableFromIPC()` (our apache-arrow).
|
|
124
|
+
4. Passes the reconstructed table to `arrowToCube` → `Observations` IR.
|
|
125
|
+
|
|
126
|
+
**Export:**
|
|
127
|
+
1. `cubeToParquet` lazily imports `parquet-wasm`.
|
|
128
|
+
2. Calls `cubeToArrow(obs)` → Apache Arrow `Table`.
|
|
129
|
+
3. Serializes via `tableToIPC(table, "stream")` → reconstructs with `wasm.Table.fromIPCStream(ipc)` (parquet-wasm's `Table`).
|
|
130
|
+
4. Calls `wasm.writeParquet(wasmTable, writerProps)` → `Uint8Array`.
|
|
131
|
+
|
|
132
|
+
If `parquet-wasm` is not installed, a `ParquetSourceError` is thrown with install instructions.
|