@nshiab/simple-data-analysis 5.21.1 → 6.0.0

package/README.md

@@ -1,16 +1,30 @@
 # Simple data analysis (SDA)
 
 SDA is an easy-to-use and high-performance TypeScript library for data analysis.
-You can use it with tabular and geospatial data.
+You can use it with tabular, geospatial, and vector data. It can also
+automatically pull data from public sources such as Statistics Canada and
+OpenStreetMap when your analysis runs. See the
+[public data source examples](#public-data-sources).
 
 The library is available on [JSR](https://jsr.io/@nshiab/simple-data-analysis)
 with its [documentation](https://jsr.io/@nshiab/simple-data-analysis/doc) and on
 [NPM](https://www.npmjs.com/package/@nshiab/simple-data-analysis).
 
-The documentation is also available as the markdown file
+> [!NOTE]
+> SDA's `SimpleDB` and `SimpleTable` inherit all core methods, but JSR currently
+> omits inherited methods from their documentation because of an
+> [upstream limitation](https://github.com/jsr-io/jsr/issues/747). For those
+> methods, see the core
+> [`SimpleDB` reference](https://jsr.io/@nshiab/simple-data-analysis-core/doc/~/SimpleDB)
+> and
+> [`SimpleTable` reference](https://jsr.io/@nshiab/simple-data-analysis-core/doc/~/SimpleTable).
+> You can call these methods directly on SDA instances.
+
+AI coding assistants and agents can start with the concise
+[llms.txt](https://github.com/nshiab/simple-data-analysis/blob/main/llms.txt)
+index. The complete generated API reference is available in
 [llm.md](https://github.com/nshiab/simple-data-analysis/blob/main/llm.md), which
-can be passed as context to improve the use of the library by AI coding
-assistants or agents.
+combines core and SDA documentation.
 
 The library is maintained by [Nael Shiab](http://naelshiab.com/), computational
 journalist and senior data producer for [CBC News](https://www.cbc.ca/news).
@@ -38,11 +52,17 @@ SDA is split into two packages:
 - **`simple-data-analysis`** (this package) extends the core with additional
   features: AI methods (row-by-row processing, embeddings, vector similarity,
   hybrid search, RAG, natural language queries), Google Sheets integration, and
-  charting/dataviz methods.
+  charting/dataviz methods. These features are built respectively on the
+  [`journalism-ai`](https://jsr.io/@nshiab/journalism-ai),
+  [`journalism-google`](https://jsr.io/@nshiab/journalism-google), and
+  [`journalism-dataviz`](https://jsr.io/@nshiab/journalism-dataviz) libraries.
 
 Most users will want this package (`simple-data-analysis`), which includes all
 of the core functionality plus the extended features.
 
+AI, Google Sheets, and charting dependencies load only when those operations
+run. If you only use core methods, you don't pay their startup cost.
+
 ## Installation
 
 The library is available on [JSR](https://jsr.io/@nshiab/simple-data-analysis)
@@ -76,6 +96,52 @@ npx @nshiab/setup-data-project
 bunx @nshiab/setup-data-project
 ```
 
+## Performance
+
+SDA uses DuckDB to handle large tabular and geospatial analyses efficiently,
+often outperforming traditional dataframe tools while keeping the code simple
+and readable.
+
+These benchmarks were run on a MacBook Pro with an Apple M4 Max and 64 GB of
+memory.
+
+### Tabular data
+
+Using 22,051,025 temperature records (`ahccd.csv`, 1.77 GB, in
+`benchmarks/data/`), we remove missing temperatures, convert dates and numbers,
+save the cleaned data, then calculate average temperatures by station and decade
+and export the sorted results.
+
+We ran the same analysis with SDA, pandas, the tidyverse, and raw DuckDB. SDA
+stays close to raw DuckDB while keeping the code simple and readable.
+
+<!-- benchmark-tabular:start -->
+
+![Horizontal bars comparing mean duration for the tabular workload; lower is better](./assets/benchmark-tabular-duration.png)
+
+![Horizontal bars comparing mean peak memory for the tabular workload; lower is better](./assets/benchmark-tabular-memory.png)
+
+<!-- benchmark-tabular:end -->
+
+### Geospatial data
+
+Using 335,024 Montreal public trees (`arbres-publics.csv`, 135.5 MB) and 91
+neighbourhood boundaries (`quartierreferencehabitation.geojson`, 1.14 MB), both
+in `benchmarks/data/`, we remove missing coordinates, create points, join trees
+to neighbourhoods, then count trees per neighbourhood and export the sorted
+results.
+
+We ran the same analysis with SDA, GeoPandas, sf, and raw DuckDB. SDA brings
+DuckDB's geospatial tools to the same simple, chainable API.
+
+<!-- benchmark-spatial:start -->
+
+![Horizontal bars comparing mean duration for the spatial workload; lower is better](./assets/benchmark-spatial-duration.png)
+
+![Horizontal bars comparing mean peak memory for the spatial workload; lower is better](./assets/benchmark-spatial-memory.png)
+
+<!-- benchmark-spatial:end -->
+
 ## Core principles
 
 SDA is born out of the frustration of switching between Python, R, and
@@ -88,214 +154,189 @@ The missing piece in the JavaScript/TypeScript ecosystem was an easy-to-use and
 performant library for data analysis. This is why SDA was created.
 
 The library is based on [DuckDB](https://duckdb.org/), a fast in-process
-analytical database. Under the hood, SDA sends SQL queries to be executed by
-DuckDB. We use [duckdb-node-neo](https://github.com/duckdb/duckdb-node-neo). For
-geospatial computations, we rely on the
-[duckdb_spatial](https://github.com/duckdb/duckdb_spatial) extension.
+analytical database. Under the hood, SDA executes SQL queries in DuckDB through
+[duckdb-node-neo](https://github.com/duckdb/duckdb-node-neo). SDA also uses
+[DuckDB extensions](https://duckdb.org/docs/current/extensions/overview) for
+additional capabilities, including
+[duckdb_spatial](https://github.com/duckdb/duckdb-spatial) for geospatial
+computations and other extensions for fuzzy matching, full-text search, vector
+indexes, and more. These extensions are loaded lazily, only when the
+corresponding features are used.
+
+To keep data pipelines concise and fast, SDA queues core transformations instead
+of executing each one immediately. Consecutive compatible operations are fused
+into a single DuckDB statement, reducing database round trips. Transformation
+methods are synchronous and chainable; async observer methods such as
+`getData()`, `log()`, and `writeData()` flush the queue before producing their
+result. This means only the final observer needs to be awaited.
+
+Methods that load or transform data—including `aiRowByRow()`, `aiEmbeddings()`,
+`aiVectorSimilarity()`, `hybridSearch()`, `aiQuery()`, `loadSheet()`,
+`loadDatawrapper()`, and `loadGeoDatawrapper()`—are also synchronous builders.
+Their external work runs when an observer executes the queue. Await a final
+observer or call `.run()` to execute a chain without reading or exporting its
+result.
+
+Methods that return an answer or export data, such as `aiRAG()`, `toSheet()`,
+`toDatawrapper()`, `toGeoDatawrapper()`, `writeChart()`, and `writeMap()`,
+remain asynchronous and must be awaited.
 
 The syntax and the available methods were inspired by
 [Pandas](https://github.com/pandas-dev/pandas) (Python) and the
-[Tidyverse](https://www.tidyverse.org/) (R).
-
-You can also write your own SQL queries if you want to (check the
-[customQuery method](https://jsr.io/@nshiab/simple-data-analysis/doc/~/SimpleDB.prototype.customQuery))
-or use JavaScript to process your data (check the
-[updateWithJS method](https://jsr.io/@nshiab/simple-data-analysis/doc/~/SimpleTable.prototype.updateWithJS)).
-
-Several methods can also leverage LLMs (large language models). See
-[aiRowByRow](https://jsr.io/@nshiab/simple-data-analysis/doc/~/SimpleTable.prototype.aiRowByRow)
-for cleaning, extracting, or categorizing data, and
-[aiQuery](https://jsr.io/@nshiab/simple-data-analysis/doc/~/SimpleTable.prototype.aiQuery)
-for interacting with your data using natural language. For embeddings and
-semantic search, have a look at
-[aiEmbeddings](https://jsr.io/@nshiab/simple-data-analysis/doc/~/SimpleTable.prototype.aiEmbeddings)
-and
-[aiVectorSimilarity](https://jsr.io/@nshiab/simple-data-analysis/doc/~/SimpleTable.prototype.aiVectorSimilarity).
-
-Feel free to start a conversation or open an issue. Check how you can
-[contribute](https://github.com/nshiab/simple-data-analysis/blob/main/CONTRIBUTING.md).
+[Tidyverse](https://www.tidyverse.org/) (R). Method and option names are kept
+simple and descriptive, so anyone can read an SDA pipeline and understand what
+is happening step by step.
 
-## Performance
+## Examples
 
 ### Tabular data
 
-To test and compare the library's performance, we calculated the average
-temperature per decade and city with the daily temperatures from the
-[Adjusted and Homogenized Canadian Climate Data](https://api.weather.gc.ca/collections/ahccd-annual).
-See [this repository](https://github.com/nshiab/simple-data-analysis-benchmarks)
-for the code.
-
-We ran the same calculations with **simple-data-analysis** (Node.js, Bun, and
-Deno), **Pandas (Python)**, and the **tidyverse (R)**.
+In this example, we load daily temperatures for three Canadian weather stations,
+remove missing values, and compute the average temperature for each station. We
+then log the results and write them to a CSV file.
 
-In each script, we:
+```ts
+import { SimpleDB } from "@nshiab/simple-data-analysis";
 
-1. Loaded a CSV file (_Importing_)
-2. Selected four columns, removed rows with missing temperature, converted date
-   strings to date and temperature strings to float (_Cleaning_)
-3. Added a new column _decade_ and calculated the decade (_Modifying_)
-4. Calculated the average temperature per decade and city (_Summarizing_)
-5. Wrote the cleaned-up data that we computed the averages from in a new CSV
-   file (_Writing_)
+const sdb = new SimpleDB();
 
-Each script has been run ten times on a MacBook Pro (Apple M4 Max / 64 GB).
+const data = await sdb
+  .newTable("temperatures")
+  .loadData(
+    "https://raw.githubusercontent.com/nshiab/simple-data-analysis/main/test/data/files/dailyTemperatures.csv",
+  )
+  .renameColumns({ t: "temperature", id: "station" })
+  .removeMissing({ columns: "temperature" })
+  .summarize({
+    columns: "temperature",
+    by: "station",
+    stats: "mean",
+    decimals: 2,
+  })
+  .sort({ mean: "desc" })
+  .log();
+
+await data.writeData("sda/output/averageTemperatures.csv");
+await sdb.close();
+```
 
-With _ahccd.csv_:
+### Geospatial data
 
-- 1.7 GB
-- 773 cities
-- 20 columns
-- 22,051,025 rows
+In this example, we load a CSV file with the latitude and longitude of 2023
+wildfires in Canada, create point geometries from it, do a spatial join with
+provinces' boundaries, write the joined data to a GeoJSON file, and then compute
+the number of fires and the total area burnt per province.
 
-Thanks to DuckDB, **simple-data-analysis** is the fastest option.
+```ts
+import { SimpleDB } from "@nshiab/simple-data-analysis";
 
-![A chart showing the processing duration of multiple scripts in various languages](./assets/big-file.png)
+const sdb = new SimpleDB();
 
-### Geospatial data
+const fires = await sdb
+  .newTable("fires")
+  .loadData(
+    "https://raw.githubusercontent.com/nshiab/simple-data-analysis/main/test/geodata/files/firesCanada2023.csv",
+  )
+  .createPoints("lat", "lon", "geom")
+  .log();
+
+const provinces = await sdb
+  .newTable("provinces")
+  .loadGeoData(
+    "https://raw.githubusercontent.com/nshiab/simple-data-analysis/main/test/geodata/files/CanadianProvincesAndTerritories.json",
+  )
+  .log();
+
+const firesInsideProvinces = await fires
+  .joinGeo(provinces, "inside", {
+    outputTable: "firesInsideProvinces",
+  })
+  .removeMissing()
+  .removeColumns("geomProvinces")
+  .log();
+
+// Each fire now has a province value.
+await firesInsideProvinces.writeGeoData(
+  "sda/output/firesInsideProvinces.geojson",
+);
 
-To test the geospatial computation speed, we performed a spatial join to match
-each public tree in Montreal to its neighbourhood. We then counted the number of
-trees in each neighbourhood. For more information, check this
-[repository](https://github.com/nshiab/simple-data-analysis-spatial-benchmarks).
+// We can use any other method, such as summarize.
+await firesInsideProvinces
+  .summarize({
+    columns: "hectares",
+    by: "nameEnglish",
+    stats: { nbFires: "count", burntArea: "sum" },
+    decimals: 0,
+  })
+  .sort({ burntArea: "desc" })
+  .log();
+
+await sdb.close();
+```
 
-With _trees.csv_:
+### Public data sources
 
-- 128 MB
-- 316,321 trees
-- 33 columns
+SDA can download data directly from established public sources. Retrieved data
+is cached locally by default, making it easy to build reproducible workflows
+without repeatedly downloading the same datasets.
 
-And _neighbourhoods.geojson_:
+#### Statistics Canada
 
-- 991 KB
-- 91 neighbourhoods
-- 6 columns
+Use `loadStatCanData` with a Statistics Canada table identifier:
 
-Each script has been run ten times on a MacBook Pro (Apple M4 Max / 64 GB).
+```ts
+import { SimpleDB } from "@nshiab/simple-data-analysis";
 
-As we can see, **simple-data-analysis** is also the fastest option here.
+const sdb = new SimpleDB();
 
-![A chart showing the processing duration of multiple scripts in various languages, for geospatial computations](./assets/spatial.png)
+await sdb
+  .newTable("population")
+  .loadStatCanData("17-10-0005-01")
+  .filter("GEO = 'Canada'")
+  .log();
 
-DuckDB, which powers SDA, can also be used with
-[Python](https://duckdb.org/docs/api/python/overview.html) and
-[R](https://duckdb.org/docs/api/r).
+await sdb.close();
+```
 
-## Examples
+#### OpenStreetMap
 
-In this example, we load a CSV file with the latitude and longitude of 2023
-wildfires in Canada, create point geometries from it, do a spatial join with
-provinces' boundaries, and then compute the number of fires and the total area
-burnt per province. We create charts and write the results to a file.
+Use `loadOSM` with a bounding box and one or more OpenStreetMap tags:
 
-If you are using Deno, make sure to install and enable the
-[Deno extension](https://docs.deno.com/runtime/getting_started/setup_your_environment/).
+The default endpoint is a shared public service. Follow the
+[Overpass public-instance usage guidelines](https://dev.overpass-api.de/overpass-doc/en/preface/commons.html),
+and configure another endpoint or run your own instance for high-volume usage.
 
 ```ts
 import { SimpleDB } from "@nshiab/simple-data-analysis";
-import { barX, plot } from "@observablehq/plot";
 
-// We start a SimpleDB instance.
 const sdb = new SimpleDB();
 
-// We create a new table
-const fires = sdb.newTable("fires");
-// We fetch the wildfires data. It's a csv.
-await fires.loadData(
-  "https://raw.githubusercontent.com/nshiab/simple-data-analysis/main/test/geodata/files/firesCanada2023.csv",
-);
-// We create point geometries from the lat and lon columns
-// and we store the points in the new column geom
-await fires.points("lat", "lon", "geom");
-// We log the fires
-await fires.logTable();
-
-// We create a new table
-const provinces = sdb.newTable("provinces");
-// We fetch the provinces' boundaries. It's a geojson.
-await provinces.loadGeoData(
-  "https://raw.githubusercontent.com/nshiab/simple-data-analysis/main/test/geodata/files/CanadianProvincesAndTerritories.json",
-);
-// We log the provinces
-await provinces.logTable();
-
-// We match fires with provinces
-// and we output the results into a new table.
-// By default, joinGeo will automatically look
-// for columns storing geometries in the tables,
-// do a left join, and put the results
-// in the left table. For non-spatial data,
-// you can use the method join.
-const firesInsideProvinces = await fires.joinGeo(provinces, "inside", {
-  outputTable: "firesInsideProvinces",
-});
-
-// We summarize to count the number of fires
-// and sum up the area burnt in each province.
-await firesInsideProvinces.summarize({
-  values: "hectares",
-  categories: "nameEnglish",
-  summaries: ["count", "sum"],
-  decimals: 0,
-});
-// We rename columns.
-await firesInsideProvinces.renameColumns({
-  count: "nbFires",
-  sum: "burntArea",
-});
-// We want the province with
-// the greatest burnt area first.
-await firesInsideProvinces.sort({ burntArea: "desc" });
-
-// We log the results. By default, the method
-// logs the first 10 rows, but there is 13
-// rows in our data. We also log the data types.
-await firesInsideProvinces.logTable({ nbRowsToLog: 13, types: true });
-
-// We can also log a bar chart directly in the terminal...
-await firesInsideProvinces.logBarChart("nameEnglish", "burntArea");
+await sdb
+  .newTable("schools")
+  .loadOSM(
+    {
+      west: -73.587799,
+      south: 45.445078,
+      east: -73.552265,
+      north: 45.471086,
+    },
+    { filters: ["amenity", "school"] },
+  )
+  .log();
 
-// ... or make a fancier chart or map
-// with Observable Plot (don't forget to install it)
-// and save it to a file.
-const chart = (data: unknown[]) =>
-  plot({
-    marginLeft: 170,
-    grid: true,
-    x: { tickFormat: (d) => `${d / 1_000_000}M`, label: "Burnt area (ha)" },
-    y: { label: null },
-    color: { scheme: "Reds" },
-    marks: [
-      barX(data, {
-        x: "burntArea",
-        y: "nameEnglish",
-        fill: "burntArea",
-        sort: { y: "-x" },
-      }),
-    ],
-  });
-await firesInsideProvinces.writeChart(chart, "sda/output/chart.png");
-
-// And we can write the data to a parquet, json or csv file.
-// For geospatial data, you can use writeGeoData to
-// write geojson or geoparquet files.
-await firesInsideProvinces.writeData("sda/output/firesInsideProvinces.parquet");
-
-// We close everything.
-await sdb.done();
+await sdb.close();
 ```
 
-Here's what you should see in your console if your run this script.
-
-![The console tab in VS Code showing the result of simple-data-analysis computations.](./assets/nodejs-console-with-chart.png)
-
-You'll also find a `chart.png` file and a `firesInsideProvinces.parquet` file in
-your folder.
+Is there another reliable and broadly useful public data source you would like
+SDA to support directly?
+[Open an issue](https://github.com/nshiab/simple-data-analysis/issues/new) with
+a link to the source and an example of the data you would like to retrieve.
 
-![A chart showing the burnt area of wildfires in Canadian provinces.](./assets/chart.png)
+### Data visualisations
 
-## More on charts and maps
+#### Charts
 
-You can easily display charts and maps directly in the terminal with the
+You can easily display charts directly in the terminal with the
 [`logBarChart`](https://jsr.io/@nshiab/simple-data-analysis/doc/~/SimpleTable.prototype.logBarChart),
 [`logDotChart`](https://jsr.io/@nshiab/simple-data-analysis/doc/~/SimpleTable.prototype.logDotChart),
 [`logLineChart`](https://jsr.io/@nshiab/simple-data-analysis/doc/~/SimpleTable.prototype.logLineChart)
@@ -314,48 +355,46 @@ import { SimpleDB } from "@nshiab/simple-data-analysis";
 import { dodgeX, dot, plot } from "@observablehq/plot";
 
 const sdb = new SimpleDB();
-const table = sdb.newTable();
-
-await table.loadData(
-  "https://raw.githubusercontent.com/nshiab/simple-data-analysis/main/test/geodata/files/firesCanada2023.csv",
+const table = await sdb
+  .newTable()
+  .loadData(
+    "https://raw.githubusercontent.com/nshiab/simple-data-analysis/main/test/geodata/files/firesCanada2023.csv",
+  )
+  .filter(`hectares > 1`)
+  .replace("cause", { "H": "Human", "N": "Natural", "U": "Unknown" })
+  .log();
+
+// We create a beeswarm chart with a log scale, faceted by cause.
+await table.writeChart(
+  (data) =>
+    plot({
+      height: 600,
+      width: 800,
+      color: { legend: true },
+      y: { type: "log", label: "Hectares" },
+      r: { range: [1, 20] },
+      marks: [
+        dot(
+          data,
+          dodgeX("middle", {
+            fx: "cause",
+            y: "hectares",
+            fill: "cause",
+            r: "hectares",
+          }),
+        ),
+      ],
+    }),
+  "sda/output/chart.png",
 );
-// We keep only the fires that are larger than 1 hectare.
-await table.filter(`hectares > 1`);
-// We rename the causes.
-await table.replace("cause", { "H": "Human", "N": "Natural", "U": "Unknown" });
-await table.logTable();
-
-// Let's create a beeswarm chart with a log scale.
-// We facet over the causes.
-const chart = (data: unknown[]) =>
-  plot({
-    height: 600,
-    width: 800,
-    color: { legend: true },
-    y: { type: "log", label: "Hectares" },
-    r: { range: [1, 20] },
-    marks: [
-      dot(
-        data,
-        dodgeX("middle", {
-          fx: "cause",
-          y: "hectares",
-          fill: "cause",
-          r: "hectares",
-        }),
-      ),
-    ],
-  });
-
-const path = "sda/output/chart.png";
 
-await table.writeChart(chart, path);
-
-await sdb.done();
+await sdb.close();
 ```
 
 ![Beeswarm chart showing the size of wildfires in Canada in 2023.](./assets/beeswarm.png)
 
+#### Maps
+
 If you want to create [Observable Plot](https://github.com/observablehq/plot)
 maps, you can use
 [`writeMap`](https://jsr.io/@nshiab/simple-data-analysis/doc/~/SimpleTable.prototype.writeMap).
@@ -367,193 +406,265 @@ import { SimpleDB } from "@nshiab/simple-data-analysis";
 import { geo, plot } from "@observablehq/plot";
 
 const sdb = new SimpleDB();
-const provinces = sdb.newTable("provinces");
 
-// We fetch the Canadian provinces boundaries.
-await provinces.loadGeoData(
-  "https://raw.githubusercontent.com/nshiab/simple-data-analysis/main/test/geodata/files/CanadianProvincesAndTerritories.json",
+const provinces = await sdb
+  .newTable("provinces")
+  .loadGeoData(
+    "https://raw.githubusercontent.com/nshiab/simple-data-analysis/main/test/geodata/files/CanadianProvincesAndTerritories.json",
+  )
+  .log();
+
+const fires = await sdb
+  .newTable("fires")
+  .loadData(
+    "https://raw.githubusercontent.com/nshiab/simple-data-analysis/main/test/geodata/files/firesCanada2023.csv",
+  )
+  .createPoints("lat", "lon", "geom")
+  .replace("cause", { "H": "Human", "N": "Natural", "U": "Unknown" })
+  .selectColumns(["geom", "hectares", "cause"])
+  .filter(`hectares > 0`)
+  .log();
+
+// We put the provinces and fires in the same table and add an isFire column
+// to easily distinguish between them.
+const provincesAndFires = await provinces
+  .clone({
+    name: "provincesAndFires",
+  })
+  .insertTables(fires, { unifyColumns: true })
+  .addColumn("isFire", "boolean", `hectares > 0`)
+  .log();
+
+await provincesAndFires.writeMap(
+  (geoData) => {
+    const fires = geoData.features.filter((d) => d.properties.isFire);
+    const provinces = geoData.features.filter((d) => !d.properties.isFire);
+
+    return plot({
+      projection: {
+        type: "conic-conformal",
+        rotate: [100, -60],
+        domain: geoData,
+      },
+      color: {
+        legend: true,
+      },
+      r: { range: [0.5, 25] },
+      marks: [
+        geo(provinces, {
+          stroke: "lightgray",
+          fill: "whitesmoke",
+        }),
+        geo(fires, {
+          r: "hectares",
+          fill: "cause",
+          fillOpacity: 0.25,
+          stroke: "cause",
+          strokeOpacity: 0.5,
+        }),
+      ],
+    });
+  },
+  "sda/output/map.png",
 );
-await provinces.logTable();
 
-// We fetch the fires.
-const fires = sdb.newTable("fires");
-await fires.loadData(
-  "https://raw.githubusercontent.com/nshiab/simple-data-analysis/main/test/geodata/files/firesCanada2023.csv",
-);
-// We create a new column to store the points as geometries.
-await fires.points("lat", "lon", "geom");
-// We select the columns of interest and filter out
-// fires less than 1 hectare.
-await fires.replace("cause", { "H": "Human", "N": "Natural", "U": "Unknown" });
-await fires.selectColumns(["geom", "hectares", "cause"]);
-await fires.filter(`hectares > 0`);
-await fires.logTable();
-
-// Now, we want the provinces and the fires in the same table
-// to draw our map with the writeMap method.
-// First, we clone the provinces table.
-const provincesAndFires = await provinces.cloneTable({
-  outputTable: "provincesAndFires",
-});
-// Now we can insert the fires into the provincesAndFires table.
-// By default, SDA will throw an error if the tables don't have the
-// same columns. So we set the unifyColumns option to true.
-await provincesAndFires.insertTables(fires, { unifyColumns: true });
-// To make our lives easier, we add a column to
-// distinguish between provinces and fires.
-await provincesAndFires.addColumn("isFire", "boolean", `hectares > 0`);
-await provincesAndFires.logTable();
-
-// This is our function to draw the map, using the Plot library.
-// The geoData will come from the our provincesAndFires table
-// as GeoJSON data. Each row of the table is a feature, and each
-// feature has properties matching the columns of the table.
-const map = (geoData: {
-  features: {
-    properties: { [key: string]: unknown };
-  }[];
-}) => {
-  const fires = geoData.features.filter((d) => d.properties.isFire);
-  const provinces = geoData.features.filter((d) => !d.properties.isFire);
-
-  return plot({
-    projection: {
-      type: "conic-conformal",
-      rotate: [100, -60],
-      domain: geoData,
-    },
-    color: {
-      legend: true,
-    },
-    r: { range: [0.5, 25] },
-    marks: [
-      geo(provinces, {
-        stroke: "lightgray",
-        fill: "whitesmoke",
-      }),
-      geo(fires, {
-        r: "hectares",
-        fill: "cause",
-        fillOpacity: 0.25,
-        stroke: "cause",
-        strokeOpacity: 0.5,
-      }),
-    ],
-  });
-};
-
-// This is the path where the map will be saved.
-const path = "sda/output/map.png";
-
-// Now we can call writeMap.
-await provincesAndFires.writeMap(map, path);
-
-await sdb.done();
+await sdb.close();
 ```
 
 ![Map showing the wildfires in Canada in 2023.](./assets/map.png)
 
-## Caching fetched and computed data
+### AI
+
+SDA can use LLMs and embedding models to enrich data, search text, and answer
+questions based on the contents of a table. The examples below rely on
+environment variables to connect to an AI provider and select a model. Click the
+relevant documentation links below for more information.
+
+SDA's AI capabilities come from
+[`journalism-ai`](https://jsr.io/@nshiab/journalism-ai). By default, LLM
+responses and embeddings are cached in the hidden `.journalism-cache` folder,
+avoiding repeated model calls for the same request.
+
+#### Enrich rows with AI
+
+The
+[`aiRowByRow`](https://jsr.io/@nshiab/simple-data-analysis/doc/~/SimpleTable.prototype.aiRowByRow)
+method sends the values of a column to an LLM and stores the structured
+responses in one or more new columns. It processes requests concurrently and can
+record row-level errors, making it useful for cleaning, extracting, classifying,
+and enriching data at scale.
+
+```ts
+import { SimpleDB } from "@nshiab/simple-data-analysis";
+
+const sdb = new SimpleDB();
+const cities = await sdb
+  .newTable("cities")
+  .loadArray([
+    { city: "Marrakech" },
+    { city: "Kyoto" },
+    { city: "Auckland" },
+  ])
+  .aiRowByRow(
+    "city",
+    ["country", "continent"],
+    "Give me the country and continent of the city.",
+    { concurrency: 5, errorColumn: "error" },
+  )
+  .log();
+
+await sdb.close();
+```
+
+#### Semantic search
+
+The
+[`hybridSearch`](https://jsr.io/@nshiab/simple-data-analysis/doc/~/SimpleTable.prototype.hybridSearch)
+method lets you find exact keyword matches and semantically similar matches
+together. SDA generates the embeddings using the provider and model configured
+through environment variables. For keyword search or vector search alone, the
+[`bm25`](https://jsr.io/@nshiab/simple-data-analysis/doc/~/SimpleTable.prototype.bm25)
+and
+[`aiVectorSimilarity`](https://jsr.io/@nshiab/simple-data-analysis/doc/~/SimpleTable.prototype.aiVectorSimilarity)
+methods used by `hybridSearch` are also available directly.
+
+```ts
+import { SimpleDB } from "@nshiab/simple-data-analysis";
+
+const sdb = new SimpleDB();
+// We search both the meaning and the wording of each recipe.
+const results = await sdb
+  .newTable("recipes")
+  .loadData(
+    "https://raw.githubusercontent.com/nshiab/simple-data-analysis/main/test/data/files/recipesClean.parquet",
+  )
+  .hybridSearch(
+    "buttery pastry for breakfast",
+    "Dish",
+    "Recipe",
+    5,
+    { outputTable: "results" },
+  )
+  .log(); // For example: "Butter Pie" (keyword) and "Croissant" (semantic).
+await sdb.close();
+```
+
+#### Retrieval-augmented generation (RAG)
+
+The
+[`aiRAG`](https://jsr.io/@nshiab/simple-data-analysis/doc/~/SimpleTable.prototype.aiRAG)
+method first retrieves relevant rows with hybrid search, then asks an LLM to
+answer using only those rows.
+
+```ts
+import { SimpleDB } from "@nshiab/simple-data-analysis";
+
+const sdb = new SimpleDB();
+// We retrieve the most relevant recipes and ask the AI to answer
+// based only on their contents.
+const answer = await sdb
+  .newTable("recipes")
+  .loadData(
+    "https://raw.githubusercontent.com/nshiab/simple-data-analysis/main/test/data/files/recipesClean.parquet",
+  )
+  .aiRAG(
+    "I am vegan. What can I eat for lunch that is spicy?",
+    "Dish",
+    "Recipe",
+    10,
+  );
+
+console.log(answer);
+await sdb.close();
+```
+
+#### Natural language query
+
+The
+[`aiQuery`](https://jsr.io/@nshiab/simple-data-analysis/doc/~/SimpleTable.prototype.aiQuery)
+method turns a natural-language instruction into a SQL query and executes it on
+the table.
+
+```ts
+import { SimpleDB } from "@nshiab/simple-data-analysis";
+
+const sdb = new SimpleDB();
+const averageTemperatures = await sdb
+  .newTable("temperatures")
+  .loadData(
+    "https://raw.githubusercontent.com/nshiab/simple-data-analysis/main/test/data/files/dailyTemperatures.csv",
+  )
+  .renameColumns({ t: "temperature", id: "station" })
+  .aiQuery(
+    "Compute the average temperature for each station with two decimals.",
+  )
+  .log();
+await sdb.close();
+```
+
+### Caching fetched and computed data
 
 Instead of running the same code over and over again, you can cache the results.
 This can speed up your workflow, especially when fetching data or performing
 computationally expensive operations.
 
-Here's the previous example adapted to cache data. For more information, check
-the
-[cache method documentation](https://jsr.io/@nshiab/simple-data-analysis-core/doc/~/SimpleTable#method_cache_0).
+When you use the
+[cache method](https://jsr.io/@nshiab/simple-data-analysis-core/doc/~/SimpleTable.prototype.cache),
+the data is cached in a hidden `.sda-cache` folder. With `cacheVerbose` enabled,
+the logs explain whether the callback, the table, and any additional inputs
+match the cached entry. `cache()` passes the table to its callback and
+automatically tracks changes made to that table before the cached step, as well
+as changes to other `SimpleTable`s used by the callback. Use `inputs` for other
+values that should invalidate the cache, such as numbers or strings.
 
-The data is cached in the hidden folder `.sda-cache` at the root of your code
-repository. Make sure to add it to your `.gitignore`. If you want to clean your
-cache, just delete the folder.
-
-If you set up with `@nshiab/setup-data-project` (see _Quick setup_ at the top),
-`.sda-cache` is automatically added to your `.gitignore` and you can use
-`npm run clean` or `bun run clean` or `deno task clean` to clear the cache.
+Here's an example caching fetched data and the result of a spatial join.
 
 ```ts
 import { SimpleDB } from "@nshiab/simple-data-analysis";
 
-// We enable two options to make our lives easier.
-// cacheVerbose will log information about the cached
-// data, and logDuration will log the total duration between
-// the creation of this SimpleDB instance and its last operation.
 const sdb = new SimpleDB({ cacheVerbose: true, logDuration: true });
 
-const fires = sdb.newTable("fires");
-
-// We cache these steps with a ttl of 60 seconds.
-// On the first run, the data will be fetched
-// and stored in the hidden folder .sda-cache.
-// If you rerun the script less than 60 seconds
-// later, the data won't be fetched but loaded
-// from the local cache. However, if you run the
-// code after 60 seconds, the data will be
-// considered outdated and fetched again.
-// After another 60 seconds, the new data in the cache will
-// expire again. This is useful when working with scraped data.
-// If you update the code passed to the cache method,
-// everything starts over.
-await fires.cache(
-  async () => {
-    await fires.loadData(
-      "https://raw.githubusercontent.com/nshiab/simple-data-analysis/main/test/geodata/files/firesCanada2023.csv",
-    );
-    await fires.points("lat", "lon", "geom");
-  },
-  { ttl: 60 },
-);
-
-const provinces = sdb.newTable("provinces");
-
-// Same thing here, except there is no ttl option,
-// so the cached data will never expire unless you delete
-// the hidden folder .sda-cache. Again, if you update
-// the code passed to the cache method, everything
-// starts over.
-await provinces.cache(async () => {
-  await provinces.loadGeoData(
+// Cache the data until the callback or table changes.
+const provinces = await sdb.newTable("provinces").cache((table) => {
+  table.loadGeoData(
     "https://raw.githubusercontent.com/nshiab/simple-data-analysis/main/test/geodata/files/CanadianProvincesAndTerritories.json",
   );
 });
 
-const firesInsideProvinces = sdb.newTable("firesInsideProvinces");
-
-// While caching is quite useful when fetching data,
-// it's also handy for computationally expensive
-// operations like joins and summaries.
-// Since the fires table has a ttl of 60 seconds
-// and we depend on it here, we need a ttl equal
-// or lower. Otherwise, we won't work with
-// up-to-date data.
-await firesInsideProvinces.cache(
-  async () => {
-    await firesInsideProvinces.insertTables(fires);
-    await firesInsideProvinces.joinGeo(provinces, "inside");
-    await firesInsideProvinces.removeMissing();
-    await firesInsideProvinces.summarize({
-      values: "hectares",
-      categories: "nameEnglish",
-      summaries: ["count", "sum"],
-      decimals: 0,
-    });
-    await firesInsideProvinces.renameColumns({
-      count: "nbFires",
-      sum: "burntArea",
-    });
-    await firesInsideProvinces.sort({ burntArea: "desc" });
+// Cache the data for 60 seconds or until the callback or table changes.
+const fires = await sdb.newTable("fires").cache(
+  (table) => {
+    table
+      .loadData(
+        "https://raw.githubusercontent.com/nshiab/simple-data-analysis/main/test/geodata/files/firesCanada2023.csv",
+      )
+      .createPoints("lat", "lon", "geom");
   },
   { ttl: 60 },
 );
 
-await firesInsideProvinces.logTable({ nbRowsToLog: 13, types: true });
+// Refresh this cache when the callback, the firesInsideProvinces table,
+// or either table used by the callback changes.
+const firesInsideProvinces = await sdb.newTable("firesInsideProvinces").cache(
+  (table) => {
+    table
+      .insertTables(fires)
+      .joinGeo(provinces, "inside")
+      .removeMissing()
+      .summarize({
+        columns: "hectares",
+        by: "nameEnglish",
+        stats: { nbFires: "count", burntArea: "sum" },
+        decimals: 0,
+      })
+      .sort({ burntArea: "desc" });
+  },
+);
+
+await firesInsideProvinces.log("all");
 await firesInsideProvinces.logBarChart("nameEnglish", "burntArea");
 
-// It's important to call done() at the end.
-// This method will remove the unused files
-// in the cache. It will also log the total duration
-// if the logDuration option was set to true.
-await sdb.done();
+await sdb.close();
 ```
 
 After the first run, here's what you'll see in your terminal. For each
@@ -562,43 +673,48 @@ After the first run, here's what you'll see in your terminal. For each
 The whole script took around a second to complete.
 
 ```
-cache() for fires
-Nothing in cache. Running and storing in cache.
-Computations done in 350 ms.
-Wrote in cache in 3 ms.
-
-
 cache() for provinces
-Nothing in cache. Running and storing in cache.
+Cache miss.
+No matching cache entry exists for this computation.
+Running computations and storing a new cache entry.
 Computations done in 247 ms.
 Wrote in cache in 1 ms.
 
 
+cache() for fires
+Cache miss.
+No matching cache entry exists for this computation.
+Running computations and storing a new cache entry.
+Computations done in 350 ms.
+Wrote in cache in 3 ms.
+
+
 cache() for firesInsideProvinces
-Nothing in cache. Running and storing in cache.
+Cache miss.
+No matching cache entry exists for this computation.
+Running computations and storing a new cache entry.
 Computations done in 68 ms.
 Wrote in cache in 0 ms.
 
 
 Table firesInsideProvinces:
-┌────────────────┬───────────────────────────┬────────────────┬───────────────┐
-│ value          │ nameEnglish               │ nbFires        │ burntArea     │
-│ VARCHAR/string │ VARCHAR/string            │ INTEGER/number │ DOUBLE/number │
-├────────────────┼───────────────────────────┼────────────────┼───────────────┤
-│ hectares       │ Quebec                    │ 706            │ 5024737       │
-│ hectares       │ Northwest Territories     │ 314            │ 4253907       │
-│ hectares       │ Alberta                   │ 1208           │ 3214444       │
-│ hectares       │ British Columbia          │ 2496           │ 2856625       │
-│ hectares       │ Saskatchewan              │ 560            │ 1801903       │
-│ hectares       │ Ontario                   │ 741            │ 441581        │
-│ hectares       │ Yukon                     │ 227            │ 395461        │
-│ hectares       │ Manitoba                  │ 301            │ 199200        │
-│ hectares       │ Nova Scotia               │ 208            │ 25017         │
-│ hectares       │ Newfoundland and Labrador │ 85             │ 21833         │
-│ hectares       │ Nunavut                   │ 1              │ 2700          │
-│ hectares       │ New Brunswick             │ 202            │ 854           │
-└────────────────┴───────────────────────────┴────────────────┴───────────────┘
-12 rows in total (nbRowsToLog: 13)
+┌───────────────────────────┬─────────┬───────────┐
+│ nameEnglish               │ nbFires │ burntArea │
+├───────────────────────────┼─────────┼───────────┤
+│ Quebec                    │ 706     │ 5024737   │
+│ Northwest Territories     │ 314     │ 4253907   │
+│ Alberta                   │ 1208    │ 3214444   │
+│ British Columbia          │ 2496    │ 2856625   │
+│ Saskatchewan              │ 560     │ 1801903   │
+│ Ontario                   │ 741     │ 441581    │
+│ Yukon                     │ 227     │ 395461    │
+│ Manitoba                  │ 301     │ 199200    │
+│ Nova Scotia               │ 208     │ 25017     │
+│ Newfoundland and Labrador │ 85      │ 21833     │
+│ Nunavut                   │ 1       │ 2700      │
+│ New Brunswick             │ 202     │ 854       │
+└───────────────────────────┴─────────┴───────────┘
+12 rows in total (count: 12)
 
 Bar chart of "burntArea" per "nameEnglish":
                           ┌
@@ -628,18 +744,29 @@ Newfoundland and Labrador ┤ 21,833
                           └
 
 
-SimpleDB - Done in 681 ms / 4 ms spent writing the cache
+SimpleDB ran for 681 ms / 4 ms spent writing the cache
 ```
 
 If you run the script less than 60 seconds after the first run, here's what
 you'll see.
 
-Thanks to caching, the script ran 25 times faster!
+Most computations are skipped and their cached data is loaded instead. In this
+example, the total runtime drops from 681 ms to 27 ms, making the second run
+around 25 times faster.
 
 ```
+cache() for provinces
+Cache hit.
+Compute function unchanged.
+Data loaded in 0 ms.
+Running computations previously took 247 ms.
+You saved 247 ms.
+
+
 cache() for fires
-Found in cache.
-ttl of 1 min, 0 sec, 0 ms has not expired.
+Cache hit.
+Compute function unchanged.
+TTL of 1 min, 0 sec, 0 ms has not expired.
 The creation date is May 28, 2026, at 4:37 p.m..
 There are 54 sec, 941 ms left.
 Data loaded in 21 ms.
@@ -647,61 +774,53 @@ Running computations previously took 312 ms.
 You saved 291 ms.
 
 
-cache() for provinces
-Found in cache.
-Data loaded in 0 ms.
-Running computations previously took 247 ms.
-You saved 247 ms.
-
-
 cache() for firesInsideProvinces
-Found in cache.
-ttl of 1 min, 0 sec, 0 ms has not expired.
-The creation date is May 28, 2026, at 4:37 p.m..
-There are 54 sec, 972 ms left.
+Cache hit.
+Compute function unchanged.
+Table dependencies unchanged: "fires", "provinces".
 Data loaded in 0 ms.
 Running computations previously took 50 ms.
 You saved 50 ms.
 
 [Note to readers: I have cut the table and chart.]
 
-SimpleDB - Done in 27 ms / 588 ms saved by using the cache
+SimpleDB ran for 27 ms / 588 ms saved by using the cache
 ```
 
-And if you run the script 60 seconds later, the fires and join/summary caches
-will have expired, but not the provinces one. Some of the code will have run,
-but not everything. The script still ran roughly 1.5 times faster. This is quite
-handy in complex analysis with big datasets. The less you wait, the more fun you
-have!
+After 60 seconds, the fires cache expires while the provinces cache is reused.
+Because `firesInsideProvinces` uses both tables in its callback, its cache
+refreshes automatically when the fires table changes.
 
 ```
-cache() for fires
-Found in cache.
-ttl of 1 min, 0 sec, 0 ms has expired.
-The creation date is May 28, 2026, at 4:37 p.m..
-It's is 1 min, 17 sec, 465 ms ago.
-Running and storing in cache.
-Computations done in 340 ms.
-Wrote in cache in 3 ms.
-
-
 cache() for provinces
-Found in cache.
+Cache hit.
+Compute function unchanged.
 Data loaded in 1 ms.
 Running computations previously took 247 ms.
 You saved 246 ms.
 
 
-cache() for firesInsideProvinces
-Found in cache.
-ttl of 1 min, 0 sec, 0 ms has expired.
+cache() for fires
+Cache entry is stale.
+Compute function unchanged.
+TTL of 1 min, 0 sec, 0 ms has expired.
 The creation date is May 28, 2026, at 4:37 p.m..
-It's is 1 min, 17 sec, 758 ms ago.
-Running and storing in cache.
+It was created 1 min, 17 sec, 465 ms ago.
+Running computations and refreshing the cache entry.
+Computations done in 340 ms.
+Wrote in cache in 3 ms.
+
+
+cache() for firesInsideProvinces
+Cache miss.
+Compute function unchanged.
+Table dependencies changed: "fires".
+Table dependencies unchanged: "provinces".
+Running computations and storing a new cache entry.
 Computations done in 48 ms.
 Wrote in cache in 1 ms.
 
 [Note to readers: I have cut the table and chart.]
 
-SimpleDB - Done in 399 ms / 246 ms saved by using the cache / 4 ms spent writing the cache
+SimpleDB ran for 399 ms / 246 ms saved by using the cache / 4 ms spent writing the cache
 ```