datanotes 3.0.0 → 3.2.0

package/dist/docs/administration/limits.md

@@ -6,21 +6,86 @@ The engine keeps every row in a Markdown note, so its speed depends on the numbe
 
 ## Measured times
 
-We measured the engine on databases of 10,000 and 50,000 notes. On Windows 11, Microsoft Defender scanned the files. On Linux, the engine ran in a container with the notes on a Linux volume.
+We measured Datanotes 3.1.0 on synthetic databases of 10,000 and 50,000 notes, made with `bench/scale/`. One machine ran every test:
 
-| Measure                                 | 10,000 notes | 50,000 notes |
-| --------------------------------------- | ------------ | ------------ |
-| Start, Windows 11 with Defender         | 15 to 18 s   | 63 to 115 s  |
-| Start, Linux container                  | 4 to 5 s     | 22 s         |
-| `select_rows` with `where`              | 30 to 35 ms  | 178 ms       |
-| `insert_row`                            | 91 to 128 ms | 414 ms       |
-| Memory (resident set size) after writes | Not measured | Up to 2 GB   |
+* Windows 11 Pro on an Intel Core i7-13700HX, with 32 GB of memory and an NVMe SSD
+* Node.js 24.21
+* Microsoft Defender scanning every file, with no exclusion
+* Other programs running at the same time
 
-These numbers set the size the engine handles well:
+The times are medians. The first start reads every note and builds the [vault index](#start-faster-with-the-vault-index). A restart reads only the notes that changed.
 
-* Up to about 10,000 notes, it starts within 20 seconds and answers within a few hundred milliseconds
-* From 10,000 to 20,000 notes, start it before your agents and keep it running
-* Above about 20,000 notes, run it as a service on Linux, and give it 2 GB of memory
+| Measure                                               | 10,000 notes | 50,000 notes |
+| ----------------------------------------------------- | ------------ | ------------ |
+| First start                                           | 2.7 s        | 15 s         |
+| Restart                                               | 0.7 s        | 2.0 to 2.3 s |
+| First `select_rows` after a restart                   | 0.15 s       | 1.1 to 2 s   |
+| Background work after a restart                       | 0.6 s        | 4 to 6 s     |
+| Memory (resident set size) after a restart            | 130 MB       | 330 MB       |
+| Memory (resident set size) after a series of searches | 150 MB       | 460 MB       |
+| Vault index on disk                                   | 21 MB        | 104 MB       |
+
+We measured the memory once the engine was idle, after a garbage collection, with the read cache off.
+
+At 50,000 notes, the operations took these times:
+
+| Operation                                      | Median time |
+| ---------------------------------------------- | ----------- |
+| `insert_row`                                   | 37 ms       |
+| `update_row`                                   | 47 ms       |
+| `select_rows` on a table of 117 rows           | 2.8 ms      |
+| `select_rows` of 5 notes by path               | 0.3 ms      |
+| `describe_database`                            | 24 ms       |
+| `graph_neighbors` right after a write          | 2.5 ms      |
+| `search_notes`, two words found in 1,213 notes | 14 ms       |
+| `search_notes`, a person’s name                | 7 ms        |
+| `search_notes`, a word found in 15,905 notes   | 140 ms      |
+
+The first `select_rows` waits until the engine has compared the vault with its index. These numbers set the size the engine handles well:
+
+* Up to about 10,000 notes, it starts within 3 seconds and answers within a few hundred milliseconds
+* From 10,000 to 50,000 notes, start it before your agents and keep it running
+* Above about 20,000 notes, run it as a service, and give it 2 GB of memory
+
+## Start faster with the vault index
+
+The engine keeps a vault index in its store, in `index/vault.sqlite`. It records what the engine read from each note: its properties, tags, links and text. It also indexes the words of each note, so `search_notes` reads only the notes that can match. It keeps each text compressed. At the next start, the engine reads only the notes that changed since.
+
+* A note counts as unchanged while its size, its modification time and its change time stay the same.
+* The change time moves with every write, also one that restores the old modification time, as `cp -p` and `rsync -t` do.
+* A note written within 2 seconds of being recorded is read again. Some file systems keep times to the second only.
+* A note you moved while the engine was stopped is found by its file id, or by its content.
+
+The index is a cache, never the source of truth. Keep these points in mind:
+
+* You can delete it at any time. The next start builds it again.
+* The store backup leaves it out, and so does Time Machine on macOS.
+* It needs Node.js 22.13 or later. On an older Node.js, every start reads every note.
+* After each start, the engine checks the index in the background, one table at a time.
+* When SQLite or that check finds damage, the engine stops using the index until the next start rebuilds it.
+* A query with only one- or two-letter words reads every note.
+
+To turn it off, set `vaultIndexEnabled` to `false` in the [engine’s settings](https://datanotes.dev/administration/configuration/). The engine then deletes the index, and every start reads every note.
+
+## Run a large database on Windows
+
+Microsoft Defender scans every file the engine opens. On a large vault, these scans take most of the start. The engine marks its store as not content indexed, so Windows Search skips it. It never changes your antivirus settings.
+
+To speed up the engine, choose one of these options:
+
+* Exclude the store folder from real-time scanning. Run this in a PowerShell window opened as administrator:
+
+  ```powershell
+  Add-MpPreference -ExclusionPath "$env:LOCALAPPDATA\Datanotes"
+  ```
+
+* Exclude the vault folder too, but only when its notes come from sources you trust.
+
+* Put the vault and the data home on a Dev Drive. Its performance mode defers the scans.
+
+Never exclude the `node.exe` process: that exempts every Node.js program on the machine.
+
+Keep the store out of OneDrive, Dropbox and iCloud folders. Sync clients and scanners lock files, and the engine then retries its renames. On macOS, the engine writes a `.metadata_never_index` file in its store, so Spotlight skips it. It also keeps the vault index out of Time Machine, since the engine can rebuild it. Time Machine still backs up the rest of the store.
 
 ## Start the engine before the agent
 
@@ -67,3 +132,22 @@ volumes:
 ```
 
 On a Linux host, a bind mount delivers file events: set `DATANOTES_WATCH=events` there too. [Run in Docker](https://datanotes.dev/administration/docker/) explains the container.
+
+## Check the engine’s indexes
+
+The engine keeps indexes of the vault in memory, so a read does not go through every note. It learns each change from the vault’s file events: its own writes, your edits, renames and deletes. It never trusts them over the notes, which stay the source of truth.
+
+When an answer looks wrong after many changes outside the engine, check the indexes. Start the engine with `DATANOTES_INDEX_CHECK=1`:
+
+```bash
+DATANOTES_INDEX_CHECK=1 \
+  npx -y datanotes serve --database your_database_folder
+```
+
+After every operation, the engine then compares its indexes with a full read of the vault:
+
+* A difference shows as an error in the engine’s log, with the notes it concerns.
+* The engine then reads the vault again, so the next operations answer right.
+* Each comparison reads every note, so keep the check for debugging, never in daily use.
+
+Every 10 minutes, once no write ran for a minute, the engine also compares the transaction log’s view of the rows with the vault. A difference shows as a warning, and the engine logs the changes it missed.

package/dist/docs/administration/run-the-service.md

@@ -22,6 +22,24 @@ The first start creates the engine’s store and its settings file, `settings.js
 npx -y datanotes store path --database your_database_folder
 ```
 
+Add `--json` to get the answer as an object:
+
+| Field             | Meaning                                                                       |
+| ----------------- | ----------------------------------------------------------------------------- |
+| `vault`           | The folder you gave, as an absolute path                                      |
+| `store`           | The store folder                                                              |
+| `database_id`     | The database’s id                                                             |
+| `database_folder` | The database folder, relative to the folder you gave; empty with `--database` |
+| `database_path`   | The database folder on disk, with `metastore.yaml` at its root                |
+
+Every field but `vault` is `null` when the engine cannot tell where the store would be. That happens only when all three hold:
+
+* You pass no `--store`, and `DATANOTES_STORE` is not set.
+* This machine never ran the engine on the folder.
+* `metastore.yaml` has no id yet.
+
+With `--store`, only `database_id` can be `null`: neither the store nor `metastore.yaml` records the id.
+
 To run the engine in a container, follow [Run in Docker](https://datanotes.dev/administration/docker/).
 
 ## Where the engine keeps its store

package/dist/docs/api/operations/add_column.md

@@ -13,6 +13,7 @@ Column names are permanent identifiers; only `rename_column` changes them.
 Enum columns: `values` and, for each, its meaning in `value_descriptions` (`{"stated": "the user said it", …}`): describe_table shows them and a refused value lists them.
 Generated link columns: `generated: "link({occurred_at:YYYY-MM-DD})"` with `ref` = the days table links each row to the row of `ref` named by the template (its folder read from the schema; the target may not exist yet).
 Filtered link targets (link columns): `ref_where` (a `where` every target must match) and `ref_by` (a column of the row picks the target level: `{"column": "kind", "levels": ["area", "project", "workstream"]}` or `{"column": "kind", "map": {"area": null, "project": ["area"]}}`; null = top level, no link; `target_column` when the target names it differently: an enum target column must have every value the map allows, a string one takes any value). A link to no row of the table, or to the row itself, is refused too. Existing rows that break them refuse the change; `list_link_targets` lists the valid targets of a row.
+Acyclic links: `acyclic: true` on a link (or list of links) to the table itself refuses every write that closes a cycle, on either side of an `inverse`; refused while existing rows form one (they are listed).
 
 The schema changes only through operations like this one: never edit metastore.yaml by hand (no validation, no undo).
 

package/dist/docs/api/operations/add_constraint.md

@@ -18,6 +18,7 @@ A write that changes a linked row re-checks the rows that link to it and is refu
 it breaks them, naming the changed row (in a batch, re-link them first: each step is checked on the vault
 the earlier ones left); the inverse links a write adds, undo_migration and restore_table are checked too.
 rename_column / rename_value follow into CHECKs; drop_column of a column a CHECK reads is refused.
+`hint`: what to do when a row breaks the rule, in your words; a refused write puts it first in its hint, and each violation of the CHECK carries it.
 
 The schema changes only through operations like this one: never edit metastore.yaml by hand (no validation, no undo).
 
@@ -36,6 +37,7 @@ The schema changes only through operations like this one: never edit metastore.y
 | `table` | `string` | yes |  |  |
 | `name` | `string` | yes |  |  |
 | `check` | `object` | yes |  |  |
+| `hint` | `string \| null` | no | `null` | What a refused write tells the agent when a row breaks this CHECK: how to fix the row. It comes first in the refusal's hint; describe_table lists it in constraint_hints. |
 | `dry_run` | `boolean` | no | `false` |  |
 
 ## Responses

package/dist/docs/api/operations/alter_column.md

@@ -3,7 +3,7 @@
 `POST /datanotes/operations/alter_column/` (operation `alter_column`)
 
 Change a column's attributes (type, min/max/step, values, required, default, label,
-aliases, description, note, `ref_where`, `ref_by` (see list_link_targets); `null` removes an attribute, except
+aliases, description, note, `ref_where`, `ref_by` (see list_link_targets), `acyclic` (refused while the rows already form a cycle: they are listed); `null` removes an attribute, except
 `description`, which changes but never goes (null or "" is refused); `dense` true/false). Refused with
 the list of violating rows when existing values do not satisfy the new definition. A
 different scale means a different column: add a new one instead of rescaling. To rename one

package/dist/docs/api/operations/alter_table.md

@@ -5,7 +5,7 @@
 Alter a table: move it (`catalog`, `schema`), rename it (`name`), change
 `partition` / `filename` patterns, `description` (changes, never goes: "" is refused), `aliases`,
 `exclude` folders, `row_template`
-("" removes it), `history` (true | false | "default"), `autolink_target`
+("" removes it), `instructions` (how to fill a row, for agents; "" removes them), `history` (true | false | "default"), `autolink_target`
 (true | false | "default": may automatic links link its rows? default: yes, unless its rows are named by
 a date or a time) or `tags` (key/value
 labels merged into the table's, e.g. `{"domain": "health"}`; a `null` value removes that tag;
@@ -13,7 +13,7 @@ labels merged into the table's, e.g. `{"domain": "health"}`; a `null` value remo
 `archive` (folder name inside the table folder) or `fields` (untyped fields, name → default; a
 `null` value removes one). Rows, archive and transaction log follow; links are updated; renamed
 rows keep their old name in `aliases`. A change of metadata only (tags, description, aliases,
-history, autolink_target, row_template, fields) rewrites the table entry alone: no row is moved or rewritten. Refused
+history, autolink_target, row_template, instructions, fields) rewrites the table entry alone: no row is moved or rewritten. Refused
 when a row cannot fill a pattern. Renaming or moving the table is refused while a CHECK that follows
 links, or a `ref_where` / `ref_by`, of another table reads it through a link (link targets are not
 rewritten): the conflicts name them. Use `dry_run` to see `planned_moves`; undo with undo_migration.
@@ -44,6 +44,7 @@ The schema changes only through operations like this one: never edit metastore.y
 | `autolink_target` | `boolean \| "default" \| null` | no | `null` | May automatic links link the table's rows? true, false, or "default" (yes, unless its rows are named by a date or a time). |
 | `exclude` | `string[] \| null` | no | `null` |  |
 | `row_template` | `string \| null` | no | `null` | Markdown body of new rows; "" removes it. |
+| `instructions` | `string \| null` | no | `null` | How to fill a row, for agents (Markdown); "" removes them. |
 | `tags` | `object \| null` | no | `null` | Tags to add or change (strings); a null value removes that tag. The other tags are kept. |
 | `clear_tags` | `boolean` | no | `false` | Remove every tag first (with `tags`: the table keeps exactly those). |
 | `archive` | `string \| null` | no | `null` | Archive folder: a folder name inside the table folder (e.g. "_archive"); "" = the database default (`archive_folder`). Archived rows follow. |

package/dist/docs/api/operations/check_table.md

@@ -3,7 +3,7 @@
 `POST /datanotes/operations/check_table/` (operation `check_table`)
 
 Validate every row against the table (required, types, ranges, enums, links, unknown
-columns, CHECK constraints, those that follow links included, filtered link targets (ref_where / ref_by), stale generated/identity values, missing inverse links, front matter that cannot be read: code `unreadable`, with no column, for a note whose `table` line names the table but whose YAML is broken, which every other read leaves out). A CHECK violation has code `check` and the CHECK's name as `constraint` (every other code names its `column`), as in the `violations` of list_changes. Use it to find rows edited by hand that break the schema. Without `table`, every table is checked: each row then names its `table`, `tables_checked` counts the tables checked, and a table that cannot be checked is listed in `errors` ({table, error}) while the others are. The rows with violations are paged: `limit` (default 500, at most 5000) from `offset`; `total` counts them all. (Hand edits are also checked as they happen: see `violations` in list_changes.)
+columns, CHECK constraints, those that follow links included, filtered link targets (ref_where / ref_by), stale generated/identity values, missing inverse links, cycles of hierarchy and acyclic columns, front matter that cannot be read: code `unreadable`, with no column, for a note whose `table` line names the table but whose YAML is broken, which every other read leaves out). A CHECK violation has code `check` and the CHECK's name as `constraint` (every other code names its `column`), as in the `violations` of list_changes. Use it to find rows edited by hand that break the schema. Without `table`, every table is checked: each row then names its `table`, `tables_checked` counts the tables checked, and a table that cannot be checked is listed in `errors` ({table, error}) while the others are. The rows with violations are paged: `limit` (default 500, at most 5000) from `offset`; `total` counts them all. (Hand edits are also checked as they happen: see `violations` in list_changes.)
 
 ## Request body
 

package/dist/docs/api/operations/create_table.md

@@ -16,7 +16,7 @@ digits, `_` (no dots: pass the catalog and the schema as their own arguments).
   order: filters/avg use the position), ref (link target, full `catalog.schema.table` or "self"),
   unique, label, description, aliases, `generated` (e.g. `"date(occurred_at)"`, `"{name} {surname}"`),
   `identity` ("filename" | "created"), `vocabulary` (allowed values = names of another table's rows),
-  `hierarchy: true` (one parent link to its own table), `dense` (written on every row),
+  `hierarchy: true` (one parent link to its own table), `acyclic: true` (links to rows of its own table that never loop back, such as tasks blocked by tasks; with `inverse`, the rule holds on both sides), `dense` (written on every row),
   `inverse` on a link column: the column of the `ref` table that links back (`friends` ↔
   `friends`, `children` ↔ `parents`, `partner` ↔ `partner`); writes keep both sides in step.
 - `time_column`: rows named by that date/datetime; otherwise `filename` (e.g. `"{name}"`).
@@ -25,6 +25,7 @@ digits, `_` (no dots: pass the catalog and the schema as their own arguments).
   database's setting), `autolink_target` (true / false: may automatic links link its rows? default:
   yes, unless its rows are named by a date or a time, as with `time_column`).
 - `row_template`: markdown body of new rows. Shared columns come from bases: `set_table_bases`.
+- `instructions`: how to fill a row, for agents (Markdown); describe_table returns them.
 - `tags`: key/value labels of the table, strings (`{"domain": "health"}`); list_tables and
   describe_table return them, alter_table changes them.
 Enum columns: `values` and, for each, its meaning in `value_descriptions` (`{"stated": "the user said it", …}`): describe_table shows them and a refused value lists them.
@@ -59,6 +60,7 @@ The schema changes only through operations like this one: never edit metastore.y
 | `history` | `boolean \| null` | no | `null` | Keep the table's versions (time travel, restore); default: the database's setting. |
 | `autolink_target` | `boolean \| null` | no | `null` | May automatic links link the table's rows? Default: yes, unless its rows are named by a date or a time. |
 | `row_template` | `string \| null` | no | `null` | Markdown body of new rows; {column} placeholders are filled from the row. |
+| `instructions` | `string \| null` | no | `null` | How to fill a row, for agents (Markdown): which columns first, what to ask the user. describe_table returns them. |
 | `fields` | `object \| null` | no | `null` | Untyped fields of the table (name → default), kept on every row; a field cannot share a name with a column. |
 | `tags` | `object \| null` | no | `null` | Key/value labels of the table (strings), e.g. {"domain": "health"}. |
 | `dry_run` | `boolean` | no | `false` |  |

package/dist/docs/api/operations/describe_table.md

@@ -6,7 +6,7 @@ Describe a table (`catalog.schema.table` or a shorter unambiguous name): `qualif
 `catalog`, `schema`, `description`, `aliases`, `layout` (`folder`, `row_folder`: the folder pattern
 of new rows, `archive`), `options` (`partition`, `filename`, `archive`, `exclude`), typed `columns`
 with label/aliases/description and the dense flag, `untyped_fields`, `row_count`, and when set:
-`tags`, `bases` (the bases it inherits from, in order), `constraints` (its CHECKs, name → where), `row_template`, `history`,
+`tags`, `bases` (the bases it inherits from, in order), `constraints` (its CHECKs, name → where), `constraint_hints` (their own hints, name → hint), `instructions` (how to fill a row: follow them before you write one), `row_template`, `history`,
 `autolink_target`, `fields` (their values).
 
 Before adding a column (e.g. a new metric), read `columns` and match your concept against

package/dist/docs/api/operations/graph_neighbors.md

@@ -5,8 +5,10 @@
 Direct neighbours of notes: what they link to (`out`) and what links to them (`in`,
 backlinks), with counts per edge label. `direction`: out, in or both (default).
 
-`from_nodes` selects the notes: `{"paths": [...]}` and/or `{"table": ..., "where": {...}}`.
-Edges are labelled with their link column (`people`, `event`, …) or `link` for other links.
+`from_nodes` selects the notes: `{"paths": [...]}`, `{"table": ..., "where": {...}}` and/or
+`{"tables": [...]}`: the rows of several tables, by name or pattern (`["*"]` every table, `["personal.*"]` a
+catalog's). The whole vault's graph in one call: `{"tables": ["*"]}` with direction `out` (at most 5000
+start notes). Edges are labelled with their link column (`people`, `event`, …) or `link` for other links.
 
 ## Request body
 

package/dist/docs/api/operations/graph_path.md

@@ -1,9 +1,9 @@
-# Shortest chain of links between two sets of notes (each `{"paths": [...]}` and/or
+# Shortest chain of links between two sets of notes (each `{"paths": [...]}`, `{"table", "where"}`
 
 `POST /datanotes/operations/graph_path/` (operation `graph_path`)
 
-Shortest chain of links between two sets of notes (each `{"paths": [...]}` and/or
-`{"table", "where"}`): `found`, `length` and `path`: the steps (`from`, `label`, `direction`, `to`).
+Shortest chain of links between two sets of notes (each `{"paths": [...]}`, `{"table", "where"}`
+and/or `{"tables": [...]}`): `found`, `length` and `path`: the steps (`from`, `label`, `direction`, `to`).
 `direction` both by default; `max_depth` default 6 (max 10).
 
 ## Request body

package/dist/docs/api/operations/graph_traverse.md

@@ -1,8 +1,8 @@
-# Breadth-first walk over links from `from_nodes` (`{"paths": [...]}` and/or `{"table", "where"}`).
+# Breadth-first walk over links from `from_nodes` (`{"paths": [...]}`, `{"table", "where"}` and/or `{"tables": [...]}`).
 
 `POST /datanotes/operations/graph_traverse/` (operation `graph_traverse`)
 
-Breadth-first walk over links from `from_nodes` (`{"paths": [...]}` and/or `{"table", "where"}`).
+Breadth-first walk over links from `from_nodes` (`{"paths": [...]}`, `{"table", "where"}` and/or `{"tables": [...]}`).
 
 - `direction`: out (default, follow links), in (follow backlinks) or both.
 - `edges`: edge labels to follow (link columns such as `people`, `event`, or `link`).

package/dist/docs/api/operations/insert_row.md

@@ -1,8 +1,8 @@
-# Insert a row (new note). Values are validated against the columns; the note path comes from
+# Insert a row (new note). When describe_table gives the table's `instructions`, fill the row as they say. Values are validated against the columns; the note path comes from
 
 `POST /datanotes/operations/insert_row/` (operation `insert_row`)
 
-Insert a row (new note). Values are validated against the columns; the note path comes from
+Insert a row (new note). When describe_table gives the table's `instructions`, fill the row as they say. Values are validated against the columns; the note path comes from
 the table's `path`/`filename` patterns (e.g. an event logged today about last night lands in
 last night's day folder when you set `occurred_at` to last night). `body` is the note text.
 Generated / identity columns are computed (do not pass them); CHECK constraints are enforced.

package/dist/docs/api/operations/record_tool_call.md

@@ -23,6 +23,7 @@
 | `session` | `string` | no |  |  |
 | `result_chars` | `number` | no |  |  |
 | `result_tokens` | `number` | no |  |  |
+| `summary` | `object` | no |  | The result's summary, when `value` is sent already cut: the engine records it (the answer says `takes_summary`). |
 
 ## Responses
 

package/dist/docs/api/operations/tags/queries.md

@@ -11,5 +11,5 @@ Operations that read rows and notes: select_rows, link targets, text and semanti
 - [The rows a link column may point to, for one row: what an agent or a picker offers before writing](https://datanotes.dev/api/operations/list_link_targets/): `POST /datanotes/operations/list_link_targets/`
 - [Find notes by meaning, not exact words: `{query, results: [{path, score, heading, snippet,](https://datanotes.dev/api/operations/semantic_search/): `POST /datanotes/operations/semantic_search/`
 - [Direct neighbours of notes: what they link to (`out`) and what links to them (`in`,](https://datanotes.dev/api/operations/graph_neighbors/): `POST /datanotes/operations/graph_neighbors/`
-- [Breadth-first walk over links from `from_nodes` (`{"paths": [...]}` and/or `{"table", "where"}`).](https://datanotes.dev/api/operations/graph_traverse/): `POST /datanotes/operations/graph_traverse/`
-- [Shortest chain of links between two sets of notes (each `{"paths": [...]}` and/or](https://datanotes.dev/api/operations/graph_path/): `POST /datanotes/operations/graph_path/`
+- [Breadth-first walk over links from `from_nodes` (`{"paths": [...]}`, `{"table", "where"}` and/or `{"tables": [...]}`).](https://datanotes.dev/api/operations/graph_traverse/): `POST /datanotes/operations/graph_traverse/`
+- [Shortest chain of links between two sets of notes (each `{"paths": [...]}`, `{"table", "where"}`](https://datanotes.dev/api/operations/graph_path/): `POST /datanotes/operations/graph_path/`

package/dist/docs/api/operations/tags/rows-dml.md

@@ -4,7 +4,7 @@ Operations that write rows: insert, update, merge, delete, move, adopt and batch
 
 ## Operations
 
-- [Insert a row (new note). Values are validated against the columns; the note path comes from](https://datanotes.dev/api/operations/insert_row/): `POST /datanotes/operations/insert_row/`
+- [Insert a row (new note). When describe_table gives the table's `instructions`, fill the row as they say. Values are validated against the columns; the note path comes from](https://datanotes.dev/api/operations/insert_row/): `POST /datanotes/operations/insert_row/`
 - [Update every row matching `where` (required: `{}` is every row). `where`: keys AND-ed; values are equality or operators](https://datanotes.dev/api/operations/update_rows/): `POST /datanotes/operations/update_rows/`
 - [Update exactly ONE row, addressed by its `id` (the system UUID) or its `path`; refused when](https://datanotes.dev/api/operations/update_row/): `POST /datanotes/operations/update_row/`
 - [Delete rows matching `where`. `mode`: "archive" (default, move to the table's archive](https://datanotes.dev/api/operations/delete_rows/): `POST /datanotes/operations/delete_rows/`

package/dist/docs/api.md

@@ -1,6 +1,6 @@
 # Datanotes HTTP API
 
-> OpenAPI 3.1.0 description of the Datanotes 3.0.0 HTTP API: https://datanotes.dev/openapi.json
+> OpenAPI 3.1.0 description of the Datanotes 3.2.0 HTTP API: https://datanotes.dev/openapi.json
 
 Database of Markdown notes: tables of notes, rows, queries, history, a change feed and the vault's notes.
 
@@ -47,7 +47,7 @@ Operations that define the database: tables, columns, constraints, bases and vie
 
 Operations that write rows: insert, update, merge, delete, move, adopt and batch.
 
-- [Insert a row (new note). Values are validated against the columns; the note path comes from](https://datanotes.dev/api/operations/insert_row/): `POST /datanotes/operations/insert_row/`
+- [Insert a row (new note). When describe_table gives the table's `instructions`, fill the row as they say. Values are validated against the columns; the note path comes from](https://datanotes.dev/api/operations/insert_row/): `POST /datanotes/operations/insert_row/`
 - [Update every row matching `where` (required: `{}` is every row). `where`: keys AND-ed; values are equality or operators](https://datanotes.dev/api/operations/update_rows/): `POST /datanotes/operations/update_rows/`
 - [Update exactly ONE row, addressed by its `id` (the system UUID) or its `path`; refused when](https://datanotes.dev/api/operations/update_row/): `POST /datanotes/operations/update_row/`
 - [Delete rows matching `where`. `mode`: "archive" (default, move to the table's archive](https://datanotes.dev/api/operations/delete_rows/): `POST /datanotes/operations/delete_rows/`
@@ -69,8 +69,8 @@ Operations that read rows and notes: select_rows, link targets, text and semanti
 - [The rows a link column may point to, for one row: what an agent or a picker offers before writing](https://datanotes.dev/api/operations/list_link_targets/): `POST /datanotes/operations/list_link_targets/`
 - [Find notes by meaning, not exact words: `{query, results: [{path, score, heading, snippet,](https://datanotes.dev/api/operations/semantic_search/): `POST /datanotes/operations/semantic_search/`
 - [Direct neighbours of notes: what they link to (`out`) and what links to them (`in`,](https://datanotes.dev/api/operations/graph_neighbors/): `POST /datanotes/operations/graph_neighbors/`
-- [Breadth-first walk over links from `from_nodes` (`{"paths": [...]}` and/or `{"table", "where"}`).](https://datanotes.dev/api/operations/graph_traverse/): `POST /datanotes/operations/graph_traverse/`
-- [Shortest chain of links between two sets of notes (each `{"paths": [...]}` and/or](https://datanotes.dev/api/operations/graph_path/): `POST /datanotes/operations/graph_path/`
+- [Breadth-first walk over links from `from_nodes` (`{"paths": [...]}`, `{"table", "where"}` and/or `{"tables": [...]}`).](https://datanotes.dev/api/operations/graph_traverse/): `POST /datanotes/operations/graph_traverse/`
+- [Shortest chain of links between two sets of notes (each `{"paths": [...]}`, `{"table", "where"}`](https://datanotes.dev/api/operations/graph_path/): `POST /datanotes/operations/graph_path/`
 
 ### History
 

package/dist/docs/develop/concurrency.md

@@ -24,7 +24,42 @@ Every client goes through that engine:
 
 ## Writes run one at a time
 
-The engine applies writes one at a time, under the lock of its transaction log. Reads don’t wait for that lock: they run in parallel, with each other and with writes.
+The engine runs writes one at a time, in the order they arrive, under the lock of its transaction log. Reads run in parallel with each other.
+
+A write that still runs after a minute lets the next write start, so one stuck write can’t hold up the rest. The engine logs a warning that names the operation.
+
+## Reads wait for writes
+
+The engine runs on one thread. Under a stream of reads, a write would share that thread with every read that arrives while it runs, and take several times longer. So writes go first:
+
+* **A write runs or waits**: a read that arrives waits until no write is left.
+* **A read already under way**: it finishes normally. The write doesn’t wait for it.
+* **A read waits at most `writePriorityMaxWaitMs`**: 100 ms by default. It also starts once 8 writes started after it came. A burst of writes never starves the reads.
+* **A long poll of the change feed**: `list_changes` with `wait_ms` never waits, and never holds a write.
+
+What each operation does is unchanged: a read sees every write that ended before it started. Set `writePriority` to `false` in the [settings](https://datanotes.dev/reference/settings/) to let reads and writes start in the order they arrive. The span `engine.schedule.wait` shows the wait in [traces](https://datanotes.dev/develop/telemetry/#the-spans-the-engine-sends).
+
+## Repeated reads come from memory
+
+A read you send again with the same arguments, with the same token, often has the same answer. The engine keeps the answers of recent reads in memory, and sends the kept answer while nothing it read has changed:
+
+* **A write**: a read of the tables it changed is computed again. Reads of other tables keep their answers.
+* **Undo, a hand edit on disk, a sync tool**: they count as writes. The engine sees them before it answers.
+* **A schema change**: every answer is computed again.
+* **A view**: it depends on the tables its query reads.
+
+Some reads depend on less than a table, or on more:
+
+* **`read_note`**: it depends on its note. Deleting or renaming a folder also counts.
+* **`list_folder`**: it depends on the notes and folders created, deleted or renamed in its folder. Deleting or renaming any folder also counts.
+* **`list_tables`**: it depends on the schema only. A write to a table keeps its answer.
+* **`describe_database`**: it depends on every table, and on the notes whose properties can’t be read.
+* **A filter through a link column**: a key such as `project.status` follows the link column `project`. The read also depends on the tables the link leads to.
+* **Reads that follow links or search the text**: `expand`, `within`, the graph operations and `search_notes` depend on the whole vault.
+
+Reads that compare with the current time (`{now}`) are never kept. A read with `{today}` is kept 15 minutes at most. Reads of history, of the change feed and of `system.information_schema` are never kept either.
+
+`readCacheMaxBytes` sets the memory for the answers: 33554432 bytes (32 MiB) by default. The least recently used answers go first. Set it to `0` to turn the cache off.
 
 The engine checks each write against the database as the writes before it left it. A write is checked first, then applied. If another write is applied while the engine checks yours, the engine checks yours again on the new state. It tries up to 5 times before it gives up.
 

package/dist/docs/develop/http-api.md

@@ -32,7 +32,7 @@ An operation’s body takes the same arguments as the agent’s tool of the same
 * **[`list_folder`](https://datanotes.dev/reference/operations/queries/#list_folder)**: the notes and folders in a `folder`
 * **[`search_notes`](https://datanotes.dev/reference/operations/queries/#search_notes)**: the notes that contain a `query`
 
-Every route and field keeps working in every 3.x version: see [Compatibility](https://datanotes.dev/reference/compatibility/).
+Routes and fields follow semantic versioning: see [Compatibility](https://datanotes.dev/reference/compatibility/).
 
 ## Record who makes each write
 
@@ -72,6 +72,16 @@ Several clients can write at the same time. [Concurrent writes](https://datanote
 
 The engine binds to `127.0.0.1` unless you start it with `--host`. It answers no Cross-Origin Resource Sharing (CORS) preflight, so web pages can’t call it. To reach it from another device, follow the [security model](https://datanotes.dev/guides/security/security-model/).
 
+## Get large answers compressed
+
+A client on another machine gets answers of 16 KB or more compressed with gzip. It asks for that with the `Accept-Encoding: gzip` header, as most HTTP clients do. A client on the engine’s machine gets them as they are: compressing would cost the engine more time than it saves.
+
+The `httpCompression` [setting](https://datanotes.dev/reference/settings/) changes this:
+
+* **`remote`**: clients on other machines only (the default).
+* **`always`**: this machine’s clients too.
+* **`off`**: no compression.
+
 ## Generate a client or explore the routes
 
 The [HTTP API reference](https://datanotes.dev/api/) documents every route, generated from the same OpenAPI description, [openapi.json](https://datanotes.dev/openapi.json). From JavaScript, use the [JavaScript client](https://datanotes.dev/develop/client/). Agents use the [Model Context Protocol (MCP) server](https://datanotes.dev/start/connect-agents/).