@nshiab/simple-data-analysis-core 2.1.12 → 2.1.14

package/README.md

@@ -1,8 +1,8 @@
 # Simple data analysis core (SDA-core)
 
 SDA-core is a lightweight DuckDB-powered TypeScript library for tabular, SQL,
-CSV, Parquet, and geospatial data analysis on Deno, Node.js, and Bun. It has one
-runtime dependency: DuckDB.
+CSV, Parquet, and geospatial data analysis on Node.js. It has one runtime dependency: DuckDB.
+It requires Node.js 24.21.0 or newer and uses ECMAScript modules (ESM).
 
 Choose this package for core data loading, cleaning, joining, statistics, and
 geospatial operations. For AI, vector search, Google Sheets, and data
@@ -10,8 +10,8 @@ visualization features, use the full
 [simple-data-analysis library](https://github.com/nshiab/simple-data-analysis).
 
 The library is available on
-[JSR](https://jsr.io/@nshiab/simple-data-analysis-core) with its
-[documentation](https://jsr.io/@nshiab/simple-data-analysis-core/doc).
+[npm](https://www.npmjs.com/package/@nshiab/simple-data-analysis-core), with its
+[API documentation](https://nshiab.github.io/simple-data-analysis-core/).
 
 AI coding assistants and agents can start with the concise
 [llms.txt](https://github.com/nshiab/simple-data-analysis-core/blob/main/llms.txt)
@@ -28,21 +28,14 @@ journalist and senior data producer for [CBC News](https://www.cbc.ca/news).
 
 ## Installation
 
-The library is available on
-[JSR](https://jsr.io/@nshiab/simple-data-analysis-core) and
-[NPM](https://www.npmjs.com/package/@nshiab/simple-data-analysis-core).
-
 ```bash
-# Deno
-deno add jsr:@nshiab/simple-data-analysis-core
-
-# Node.js
 npm i @nshiab/simple-data-analysis-core
-
-# Bun
-bun add @nshiab/simple-data-analysis-core
 ```
 
+Use `"type": "module"` in your project’s `package.json` and import the public
+classes from `@nshiab/simple-data-analysis-core`. Extension utilities are
+available from `@nshiab/simple-data-analysis-core/helpers`.
+
 ## Quick setup
 
 To quickly set up a data project with essential folders, configurations, and
@@ -50,14 +43,7 @@ documentation for AI agents, you can use
 [@nshiab/setup-data-project](https://github.com/nshiab/setup-data-project).
 
 ```bash
-# Deno
-deno run -A jsr:@nshiab/setup-data-project
-
-# Node
 npx @nshiab/setup-data-project
-
-# Bun
-bunx @nshiab/setup-data-project
 ```
 
 ## Running a data pipeline
@@ -68,10 +54,7 @@ such as `getData()`, `log()`, and `writeData()` execute pending work before
 returning results, so only the final method in a chain needs to be awaited.
 
 ```ts
-await table
-  .filter("salary > 50000")
-  .selectColumns(["name", "salary"])
-  .log();
+await table.filter("salary > 50000").selectColumns(["name", "salary"]).log();
 ```
 
 Use `await table.run()` or `await sdb.run()` to execute pending work without
@@ -86,16 +69,22 @@ These benchmarks compare SDA-core with raw DuckDB and popular Python and R
 libraries, measuring duration and peak memory.
 
 They were run on a MacBook Pro with an Apple M4 Max and 64 GB of memory.
+JavaScript implementations run on Node.js. Run `npm run benchmark` to regenerate
+these workflow tables on macOS with Node.js, Python, and R installed and the input files
+available locally. The AHCCD file may also be placed in `test/data/files/`.
+
+Each implementation runs once to warm up, then three times in fresh processes.
+Duration includes process startup, loading, computation, and file output; memory
+is whole-process peak RSS. Warm-up outputs are checked against raw DuckDB,
+including every cleaned tabular row, and every measured result is checked again.
+Tables show mean duration with population standard deviation and mean peak
+memory. Raw measurements are retained in `benchmarks/.work/observations.csv`.
 
 <!-- benchmark-results:start -->
 
-**Tabular versions:** @duckdb/node-api 1.5.5-r.4; DuckDB v1.5.5 (Deno 2.9.6);
-SDA-core 2.0.5 (Deno 2.9.6); pandas 3.0.5 (Python 3.14.7); tidyverse 2.0.0 (R
-4.6.1).
+**Tabular versions:** @duckdb/node-api 1.5.6-r.1; DuckDB v1.5.6 (Node.js 24.21.0); SDA-core 2.1.13 (Node.js 24.21.0); pandas 3.0.5 (Python 3.14.7); tidyverse 2.0.0 (R 4.6.1).
 
-**Spatial versions:** @duckdb/node-api 1.5.5-r.4; DuckDB v1.5.5 (Deno 2.9.6);
-SDA-core 2.0.5 (Deno 2.9.6); GeoPandas 1.1.4 (Python 3.14.7); sf 1.1.2 (R
-4.6.1).
+**Spatial versions:** @duckdb/node-api 1.5.6-r.1; DuckDB v1.5.6 (Node.js 24.21.0); SDA-core 2.1.13 (Node.js 24.21.0); GeoPandas 1.1.4 (Python 3.14.7); sf 1.1.2 (R 4.6.1).
 
 ### Tabular workload
 
@@ -106,10 +95,10 @@ and export the sorted results.
 
 | Library   |  Mean duration | Duration difference | Mean peak memory | Memory difference |
 | --------- | -------------: | ------------------: | ---------------: | ----------------: |
-| DuckDB    |  1.30 ± 0.03 s |               -9.7% |         2,457 MB |             -2.7% |
-| SDA-core  |  1.44 ± 0.03 s |            baseline |         2,524 MB |          baseline |
-| pandas    | 30.08 ± 0.07 s |            +1987.9% |         4,700 MB |            +86.2% |
-| tidyverse | 82.62 ± 0.22 s |            +5634.4% |         8,178 MB |           +224.0% |
+| DuckDB    |  1.16 ± 0.02 s |              -12.8% |        2,430 MiB |             -1.1% |
+| SDA-core  |  1.33 ± 0.01 s |            baseline |        2,457 MiB |          baseline |
+| pandas    | 27.98 ± 0.04 s |            +2002.1% |        4,699 MiB |            +91.2% |
+| tidyverse | 78.60 ± 0.46 s |            +5804.3% |        8,178 MiB |           +232.8% |
 
 ### Spatial workload
 
@@ -121,10 +110,10 @@ results.
 
 | Library   | Mean duration | Duration difference | Mean peak memory | Memory difference |
 | --------- | ------------: | ------------------: | ---------------: | ----------------: |
-| DuckDB    | 0.78 ± 0.00 s |               -5.1% |           255 MB |             -5.8% |
-| SDA-core  | 0.82 ± 0.01 s |            baseline |           271 MB |          baseline |
-| GeoPandas | 1.25 ± 0.01 s |              +51.4% |           294 MB |             +8.5% |
-| sf        | 1.80 ± 0.01 s |             +118.9% |           490 MB |            +81.1% |
+| DuckDB    | 0.72 ± 0.01 s |              -11.8% |          285 MiB |             -9.9% |
+| SDA-core  | 0.81 ± 0.00 s |            baseline |          316 MiB |          baseline |
+| GeoPandas | 1.11 ± 0.00 s |              +36.3% |          292 MiB |             -7.7% |
+| sf        | 1.58 ± 0.01 s |              +94.4% |          489 MiB |            +54.8% |
 
 <!-- benchmark-results:end -->
 
@@ -132,8 +121,8 @@ results.
 
 These additional benchmarks compare Core and DuckDB on data transfer, joins,
 aggregations, and JavaScript geometry updates to help track and improve
-performance over time. Run `deno task benchmark-core` to regenerate these
-tables. See the [benchmark methodology](benchmarks/operations/README.md) for
+performance over time. Run `npm run benchmark-core` to regenerate these
+tables. See the [benchmark methodology](https://github.com/nshiab/simple-data-analysis-core/blob/main/benchmarks/operations/README.md) for
 fixture sizes, validation, and measurement details. Peak process memory includes
 setup, warm-up, the measured operation, and validation.
 
@@ -143,8 +132,8 @@ setup, warm-up, the measured operation, and validation.
 
 | Operation        |      Rows | Batch size | Implementation | Mean duration ± SD | Mean peak process memory |
 | ---------------- | --------: | ---------: | -------------- | -----------------: | -----------------------: |
-| Join → aggregate | 1,000,000 |          — | Core           |    12.11 ± 0.25 ms |                123.1 MiB |
-| Join → aggregate | 1,000,000 |          — | DuckDB         |    11.29 ± 0.33 ms |                134.2 MiB |
+| Join → aggregate | 1,000,000 |          — | Core           |    11.54 ± 0.03 ms |                174.4 MiB |
+| Join → aggregate | 1,000,000 |          — | DuckDB         |    10.59 ± 0.06 ms |                168.0 MiB |
 
 #### JavaScript data transfer
 
@@ -154,34 +143,34 @@ increment, including Core's staging and table replacement.
 
 | Operation      |    Rows | Batch size | Implementation | Mean duration ± SD | Mean peak process memory |
 | -------------- | ------: | ---------: | -------------- | -----------------: | -----------------------: |
-| loadArray()    | 100,000 |          — | Core           |     9.40 ± 0.68 ms |                202.3 MiB |
-| loadArray()    | 100,000 |          — | DuckDB         |    19.80 ± 0.94 ms |                230.8 MiB |
-| getData()      | 100,000 |          — | Core           |     9.29 ± 0.26 ms |                191.2 MiB |
-| getData()      | 100,000 |          — | DuckDB         |    12.54 ± 0.27 ms |                209.0 MiB |
-| updateWithJS() | 100,000 |       1000 | Core           |    92.45 ± 3.19 ms |                208.2 MiB |
-| updateWithJS() | 100,000 |       1000 | DuckDB         |    65.89 ± 1.13 ms |                239.6 MiB |
-| updateWithJS() | 100,000 |      10000 | Core           |    42.93 ± 1.05 ms |                218.6 MiB |
-| updateWithJS() | 100,000 |      10000 | DuckDB         |    41.76 ± 0.81 ms |                233.5 MiB |
+| loadArray()    | 100,000 |          — | Core           |     8.63 ± 0.04 ms |                355.8 MiB |
+| loadArray()    | 100,000 |          — | DuckDB         |    16.28 ± 0.16 ms |                282.4 MiB |
+| getData()      | 100,000 |          — | Core           |    10.14 ± 0.04 ms |                302.2 MiB |
+| getData()      | 100,000 |          — | DuckDB         |    12.05 ± 0.16 ms |                289.2 MiB |
+| updateWithJS() | 100,000 |       1000 | Core           |    60.85 ± 0.19 ms |                306.5 MiB |
+| updateWithJS() | 100,000 |       1000 | DuckDB         |    52.77 ± 0.37 ms |                314.1 MiB |
+| updateWithJS() | 100,000 |      10000 | Core           |    35.22 ± 1.31 ms |                323.8 MiB |
+| updateWithJS() | 100,000 |      10000 | DuckDB         |    34.60 ± 1.68 ms |                334.7 MiB |
 
 #### JavaScript geometry updates
 
 Points have one position per geometry; polygons have one ring with 1,001
 positions. Both implementations transfer GeoJSON through JavaScript and stage
-writes. These updates add a label and shift every longitude by 0.01 degrees. A
-batch size of — means all input rows at once. Attribute-only updates are also
-measured and retained in the raw results; their similar timings are omitted here
-to keep the table concise.
+writes. These updates add a label and shift every longitude by 0.01 degrees.
+A batch size of — means all input rows at once. Attribute-only updates are
+also measured and retained in the raw results; their similar timings are
+omitted here to keep the table concise.
 
 | Operation                 |    Rows | Batch size | Implementation | Mean duration ± SD | Mean peak process memory |
 | ------------------------- | ------: | ---------: | -------------- | -----------------: | -----------------------: |
-| Points: geometry update   | 100,000 |          — | Core           |   182.59 ± 1.02 ms |                402.4 MiB |
-| Points: geometry update   | 100,000 |          — | DuckDB         |   180.20 ± 1.98 ms |                402.2 MiB |
-| Points: geometry update   | 100,000 |       1000 | Core           |   252.07 ± 6.59 ms |                267.9 MiB |
-| Points: geometry update   | 100,000 |       1000 | DuckDB         |   217.69 ± 2.43 ms |                245.4 MiB |
-| Polygons: geometry update |   1,000 |          — | Core           |   403.35 ± 1.92 ms |                938.3 MiB |
-| Polygons: geometry update |   1,000 |          — | DuckDB         |   408.62 ± 2.57 ms |               1104.8 MiB |
-| Polygons: geometry update |   1,000 |        100 | Core           |   418.45 ± 2.45 ms |                747.0 MiB |
-| Polygons: geometry update |   1,000 |        100 | DuckDB         |   416.90 ± 2.54 ms |                714.6 MiB |
+| Points: geometry update   | 100,000 |          — | Core           |   174.87 ± 0.50 ms |                469.9 MiB |
+| Points: geometry update   | 100,000 |          — | DuckDB         |   172.13 ± 1.32 ms |                537.3 MiB |
+| Points: geometry update   | 100,000 |       1000 | Core           |   245.27 ± 4.96 ms |                298.3 MiB |
+| Points: geometry update   | 100,000 |       1000 | DuckDB         |   206.80 ± 1.60 ms |                272.4 MiB |
+| Polygons: geometry update |   1,000 |          — | Core           |   549.70 ± 0.67 ms |               1067.7 MiB |
+| Polygons: geometry update |   1,000 |          — | DuckDB         |   555.25 ± 4.17 ms |               1103.9 MiB |
+| Polygons: geometry update |   1,000 |        100 | Core           |   549.62 ± 2.49 ms |                786.0 MiB |
+| Polygons: geometry update |   1,000 |        100 | DuckDB         |   552.29 ± 0.43 ms |                881.3 MiB |
 
 <!-- benchmark-operations:end -->
 
@@ -205,10 +194,7 @@ such as `getData()`, `log()`, or `run()` is called. Observers execute pending
 work across the database in program order.
 
 ```ts
-import {
-  SimpleDB as CoreDB,
-  SimpleTable as CoreTable,
-} from "@nshiab/simple-data-analysis-core";
+import { SimpleDB as CoreDB, SimpleTable as CoreTable } from "@nshiab/simple-data-analysis-core";
 
 class MyTable extends CoreTable {
   selectForPublication(columns: string[]): this {
@@ -246,7 +232,7 @@ class RemoteTable extends CoreTable {
       method: "loadRemote()",
       parameters: { url },
       execute: async () => {
-        const rows = await fetch(url).then((response) => response.json()) as {
+        const rows = (await fetch(url).then((response) => response.json())) as {
           [key: string]: unknown;
         }[];
         this.loadArray(rows);
@@ -276,5 +262,5 @@ Other useful helpers, imported from `@nshiab/simple-data-analysis-core/helpers`:
 - `quoteIdentifier()`: quote table and column names safely when constructing
   SQL.
 
-See the [exported helpers](src/helpers/index.ts) for the complete list and links
+See the [exported helpers](https://nshiab.github.io/simple-data-analysis-core/) for the complete list and links
 to their documentation and examples. Helpers outside that list are internal.