datanotes 3.0.0 → 3.2.0

package/dist/docs/release-notes.md

@@ -4,6 +4,151 @@
 
 This page lists each version of Datanotes, newest first. Datanotes follows [semantic versioning](https://semver.org): a major version can break what you rely on, and its notes list those changes first. The package is on [npm](https://www.npmjs.com/package/datanotes), and each version has a [GitHub release](https://github.com/davidefornelli/datanotes/releases).
 
+## 3.2.0
+
+Datanotes 3.2.0 keeps the links between rows of one table free of cycles. It also lets a table tell agents how to fill its rows. It only adds to 3.1: no operation, argument or setting goes away.
+
+### Links without cycles
+
+A link column, or a list of links, to its own table takes `acyclic: true`. The engine then refuses a write that makes a row depend on itself through that column. An example is a task blocked by a task it blocks. [Keep dependencies free of cycles with `acyclic`](https://datanotes.dev/guides/database-objects/relations/#keep-dependencies-free-of-cycles-with-acyclic) shows it.
+
+* **Both sides**: with `inverse`, a write to either column is checked. The links the engine adds on the other side count too.
+* **Existing rows**: `add_column` and `alter_column` refuse `acyclic: true` while rows already form a cycle, and list them.
+* **Hand edits**: `check_table` and the change feed report a cycle made outside the engine.
+* **The refusal**: it shows the cycle, such as `would close a cycle in "blocked_by" (A → B → A)`. It has its own hint and docs.
+* **`system.information_schema.columns`**: a new column, `is_acyclic`.
+
+### Tell agents how to fill a table
+
+* **`instructions`**: `create_table` and `alter_table` take Markdown text that says how to fill a row. It can list which columns come first and what to ask the user. `describe_table` returns it, and `instructions: ""` removes it. [Tell agents how to fill a row with `instructions`](https://datanotes.dev/guides/database-objects/tables/#tell-agents-how-to-fill-a-row-with-instructions) shows it.
+* **A hint for each CHECK**: `add_constraint` takes `hint`, what to do when a row breaks the rule. A refused write puts it first in its `hint`, and each violation of that CHECK carries it as `hint`. `describe_table` lists the hints in `constraint_hints`, next to `constraints`.
+* **`system.information_schema`**: `tables` has a new column, `instructions`, and `check_constraints` a new column, `hint`.
+
+### Fixes
+
+* `alter_column` with `ref: "self"` no longer refuses the links a column already holds.
+
+## 3.1.0
+
+Datanotes 3.1.0 makes the engine fast at scale and adds OpenTelemetry traces. It only adds to 3.0: no operation, argument or setting goes away. One default changes: the transaction log now writes segments. Read [Before you downgrade to 3.0](#before-you-downgrade-to-30) if you may go back.
+
+### Speed
+
+Measured with `bench/scale/` on a synthetic 50,000-note vault on Windows, median times. “Before” is the engine at the start of this work, with the same benchmark:
+
+| Operation                             | Before   | 3.1.0  |
+| ------------------------------------- | -------- | ------ |
+| `insert_row`                          | 662 ms   | 37 ms  |
+| `update_row`                          | 124 ms   | 47 ms  |
+| `select_rows` on a small table        | 23 ms    | 2.8 ms |
+| `describe_database`                   | 1,019 ms | 24 ms  |
+| `graph_neighbors` right after a write | 1,472 ms | 2.5 ms |
+| Archive 100 rows                      | 71.6 s   | 1.3 s  |
+| `move_rows`, 20 rows                  | 13.9 s   | 0.5 s  |
+| `select_rows` by 5 paths              | 36.5 ms  | 0.3 ms |
+
+The gains come from these changes:
+
+* **An index of the vault in memory**: the engine keeps each table’s rows, unique values, links and backlinks up to date as notes change. A read no longer walks every note of the vault.
+
+* **Links resolved one note at a time**: creating, deleting or renaming a note no longer resolves every link of the vault again.
+
+* **Each note parsed once per version**: the engine keeps the parse until the file changes.
+
+* **Faster queries**:
+
+  * `where` filters are compiled once;
+  * sorts read each key once;
+  * pages keep only the rows they need;
+  * time-zone formatting is cached, so sorting 10,000 rows by `file_modified` takes about 50 ms, not 4 s.
+
+* **Writes before reads**: a write waits for no new read. A write under four concurrent readers takes about 35 ms, not 147 ms. Settings `writePriority` and `writePriorityMaxWaitMs` tune it.
+
+* **Search through the index on disk**: `search_notes` looks the words up in the index on disk, then reads only the notes that can match. On 50,000 notes, two words found in 1,213 notes take about 14 ms, not 70 to 83 ms. A name takes 7 ms, not 100 ms. A word found in 15,905 notes still takes about 140 ms. A query of one- or two-letter words reads every note. After a start, the engine reads the search’s row ids in the background. The first search no longer waits for them.
+
+* **Repeated reads from memory**: a read the engine answered before, on data that has not changed since, comes from a cache. Setting `readCacheMaxBytes` sizes it, and `0` turns it off.
+
+* **Faster undo**: `undo_migration` takes tens of milliseconds, not seconds.
+
+* **Faster history**:
+
+  * `table_changes` replays each version once;
+  * `describe_history` reads only commit headers;
+  * repeated time-travel reads come from memory.
+
+* **Less memory**:
+
+  * note texts stay within a budget;
+  * repeated values, names and paths are kept once;
+  * row bodies are read only when asked for;
+  * the transaction log keeps hashes, not texts.
+
+* **A faster start**:
+
+  * the engine walks the vault once, without blocking;
+  * it reads many notes at a time;
+  * the change feed starts after the server listens.
+
+* **An index on disk**: the engine keeps `<store>/index/vault.sqlite` through Node.js’s built-in SQLite. A start then reads only the notes that changed since the last stop: at 50,000 notes a restart takes about 2 s, not 40 s. The index is a cache: the notes stay the source of truth, and the engine rebuilds the index when it is missing or damaged. It holds each note’s text, compressed, and takes about twice the vault’s size on disk. Setting `vaultIndexEnabled` turns it off. Without `node:sqlite`, the engine reads every note as before.
+
+* **More reads at once**: the `datanotes` command gives Node.js 16 threads for file reads, unless `UV_THREADPOOL_SIZE` sets another number.
+
+### The whole graph in one call
+
+`graph_neighbors`, `graph_traverse` and `graph_path` take `from_nodes.tables`: table names or patterns, with `"*"` for every table. One call returns the graph of the whole vault.
+
+### Traces
+
+The engine and `datanotes mcp` send OpenTelemetry traces next to the logs, to any OTLP backend:
+
+* one span per request, continuing an incoming `traceparent`;
+* one span per operation, with the database span attributes;
+* spans for the phases: lock wait and hold, query plan and scan, validation, journal and transaction log;
+* one trace per agent call, across the MCP server and the engine.
+
+Log records carry their trace and span ids. New settings: `telemetryTraces`, `telemetryTraceSampleRatio`, `telemetryMaxSpansPerTrace`, `telemetrySlowSpanMs` and `telemetryEnvironment`. The `OTEL_*` variables work as well. `datanotes/client` takes `trace` and `tracer` to continue a trace into the engine. See [Send logs to your observability backend](https://datanotes.dev/develop/telemetry/).
+
+### The store
+
+* **The transaction log in segments, by default**: a table’s versions go to append-only segment files instead of one file per version. The store keeps far fewer files, which matters to antivirus scans and to the store backup. Setting `transactionLogFormat` chooses `segments` or `files`. A 3.0 store needs no conversion: new versions go to segments next to the old files.
+* **Smaller checkpoints**: a checkpoint is a manifest and compressed parts, written after the write that made it due.
+* **Durable store writes**: the files an operation wrote are synced at its end. Setting `storeSync` sets the policy. A version cut short by a crash no longer makes a table’s history unreadable. `vacuum_table` and `datanotes store convert-log` sync the files they write before they delete the old ones.
+* **A lighter backup**: a backup tick with no change runs no git command.
+* **The store left out of indexing**: on Windows the engine marks the store folder as not content indexed, and on macOS it adds a Spotlight marker.
+* **The vault index out of Time Machine**: on macOS the engine excludes `<store>/index/` from Time Machine. The engine rebuilds the index when it is missing, so a restore loses nothing. Time Machine still backs up the journal and the history.
+* **The vault index checked**: after each start, the engine checks the index file in the background, one table at a time. The engine turns a damaged index off for the run and rebuilds it at the next start.
+
+### Before you downgrade to 3.0
+
+Datanotes 3.0 reads only one file per version. Before you go back to 3.0, convert the logs:
+
+1. Stop the engine. The conversion runs only while no engine serves the database.
+
+2. Convert the logs of the database:
+
+   ```bash
+   datanotes store convert-log --to files --database <folder>
+   ```
+
+3. Start Datanotes 3.0.
+
+A table vacuumed with more than 32 MB of text keeps a checkpoint 3.0 cannot read, so 3.0 starts its history again from empty. To keep one file per version from the start, set `transactionLogFormat` to `files`.
+
+### Other changes
+
+* **`dnotes`**: the package also installs the command under a short name. `dnotes status` is the same as `datanotes status`, and `datanotes` stays.
+* **`search_notes`**: match contexts now fall on the right text in notes with decomposed accents, Hangul or emoji.
+* **Compressed answers**: setting `httpCompression` sends gzip to remote clients. It is on by default, and clients on the same machine get plain answers.
+* **Semantic search**: vectors take a quarter of the memory, and a repeated query skips the embedding service.
+* **`select_rows` by path**: a filter on `path` reads only the notes it names. A row elsewhere in the table that can’t be read no longer fails it.
+* **Index checks**: `DATANOTES_INDEX_CHECK=1` compares the engine’s indexes with a full read after every operation.
+* **`/tool-calls/record/`**: takes an optional `summary`, so `datanotes mcp` no longer sends whole tool results back to the engine.
+* **Benchmarks**: `bench/http-latency.mjs` and `bench/scale/` measure any vault.
+* **`datanotes store path --json`**: the answer adds `database_path`, the database folder on disk.
+* **`datanotes store new-id`, `backup` and `restore`**: each holds the store while it runs, as `convert-log` does. An engine you start meanwhile refuses to start. A live engine whose heartbeat stopped now makes them refuse too. `restore` leaves the store in place untouched until the clone is checked. When the clone cannot take its place, `restore` removes the clone. It puts the old store back, or names the folder the old store is in.
+* **Traces of background work**: `job auto_recompute` recomputes columns after a hand edit, and `job inverse_links` mirrors inverse links. Both are traces of their own, never part of the request that saw the edit.
+* **Writes**: a write runs again only when another write overtook it. Before, a write whose values held the text of that conflict ran up to five times.
+
 ## 3.0.0
 
 Datanotes 3.0.0 is the first stable release. Versions 1.x and 2.x, published on 7 and 8 October 2026, were previews. Datanotes runs a folder of Markdown notes as typed database tables. You reach it through an HTTP API, a command line and an MCP server for agents.

package/dist/docs/start/how-it-works.md

@@ -80,14 +80,43 @@ The transaction log records each version of a table, so you can read or restore
 
 Change the database’s setting with [`configure_database`](https://datanotes.dev/reference/operations/auxiliary/#configure_database) and `history`. [Table history and time travel](https://datanotes.dev/guides/history/table-history/) shows it at work.
 
-Each table’s log lives in the engine’s store, in `transaction_log/your_catalog.your_schema.your_table/`, with one numbered JSON file per change. A commit holds these entries:
+Each table’s log lives in the engine’s store, in `transaction_log/your_catalog.your_schema.your_table/`. Each version is a commit with these entries:
 
 * **`commitInfo`**: the operation, its time and, when known, its actor
 * **`metaData`**: the table’s definition, when the commit changes it
 * **`add`**: the path, the hash and the full content of each row written
 * **`remove`**: the path and the hash of each row removed
 
-The engine writes a checkpoint, a full copy of the table, once the commits since the last checkpoint outweigh it. That means twice the table’s size and at least 256 KB, or 1,000 versions. The log therefore grows with what you write, not with the table’s size. Hand edits appear as `EXTERNAL_EDIT` commits.
+The engine writes a checkpoint, a full copy of the table, once the commits since the last checkpoint outweigh it. That means twice the table’s size and at least 256 KB. With one file per version, it also writes one after 1,000 versions, or after one version per 16 KB of rows for a large table. The log therefore grows with what you write, not with the table’s size. Hand edits appear as `EXTERNAL_EDIT` commits.
+
+### How the engine writes the log
+
+The setting `transactionLogFormat` picks how the engine writes new versions:
+
+* **`segments`**, the default since 3.1: versions added to the end of a few large files, such as `00000000000000001000.segment.jsonl`. Fewer files mean less work for an antivirus, git and the store backup. Datanotes 3.0 can’t read them.
+* **`files`**: one numbered JSON file per version, such as `00000000000000000007.json`. Every Datanotes 3 release reads it.
+
+A segment holds up to 1,000 versions or 8 MB. The engine then seals it: it renames the file after its first and last version, and never writes to it again. The engine reads both formats, also mixed in one folder.
+
+A store from Datanotes 3.0 needs no conversion. Its versions stay in their files, and the new versions go to segments in the same folder. To keep one file per version, set `transactionLogFormat` to `files`.
+
+A checkpoint is a manifest and its parts. The manifest, `your_version.manifest.jsonl`, lists each row’s path and hash. The parts, `your_version.part-0.jsonl.gz`, hold the rows’ texts, compressed. No file holds more than 4 MB of text, far under the 95 MB the store backup pushes. The engine writes a checkpoint after the write that made it due, so the write doesn’t wait for it. When it starts, it reads the manifests only.
+
+A vacuum writes its checkpoint as one file, `your_version.checkpoint.json`, which Datanotes 3.0 reads too. A table with more than 32 MB of text gets a manifest instead.
+
+A crash or a power loss can leave the newest version half written. The engine leaves that version out, as if it never wrote it. The next write sets the file aside as `your_file.torn-your_time` and writes the version again.
+
+### Convert the log to another format
+
+To switch formats, stop the engine and run `datanotes store convert-log`:
+
+```bash
+datanotes store convert-log --to segments --vault ~/notes
+```
+
+The command rewrites every table’s log in the new format and sets `transactionLogFormat` to match. Run it with `--to files` before you go back to Datanotes 3.0, as [Compatibility](https://datanotes.dev/reference/compatibility/#downgrade-to-30) explains.
+
+The command holds the store while it runs. An engine you start meanwhile refuses to start.
 
 History costs disk space in the store. Each version keeps the full text of every note it writes, so a note saved ten times is stored about ten times, plus the checkpoints. A table you rewrite often grows its log fastest. [`vacuum_table`](https://datanotes.dev/reference/operations/history/#vacuum_table) deletes the old versions, and `history: false` stops the log of a table that doesn’t need it.
 
@@ -135,9 +164,12 @@ The store holds these files:
 * **`blobs/your_hash.txt`**: texts of 2 KB or more, such as `metastore.yaml` before and after a change, stored once
 * **`transaction_log/`**, **`changes/`** and **`semantic/`**: the history of tables, the change feed and the semantic index
 * **`logs/YYYY-MM-DD.your_source.jsonl`**: the engine’s log records, when `fileLogEnabled` is on
+* **`index/vault.sqlite`**: the vault index, what the engine read from each note, so a start reads only the notes changed since; you can delete it at any time
 * **`store.json`**: the vault and the database folder the store belongs to
 
-When the engine starts, it drops journal entries older than `journalRetentionDays` (90) and deletes log files older than `logRetentionDays` (30).
+When the engine starts, and then once a day, it drops journal entries older than `journalRetentionDays` (90). It also deletes log files older than `logRetentionDays` (30). It reads the journal without the write lock, so writes don’t wait for it.
+
+At the end of each operation, the engine syncs the store files it wrote to the disk: the journal line, the new versions, the blobs and the change feed. A power loss then can’t take the undo data or the history of an operation that returned. The setting `storeSync` set to `off` skips that, for faster writes. The engine never syncs notes: it writes them atomically, as Obsidian does.
 
 Your vault holds only your notes: sync tools, git and Obsidian never see the store. The store belongs to the machine the engine runs on, so the journal and the history do too. Results name store files with a `store:` prefix, such as `store:journal/2026-10.jsonl`.
 

package/dist/docs/start/quick-start.md

@@ -8,6 +8,8 @@ You need [Node.js](https://nodejs.org) 22 or newer and a folder for your databas
 
 On Windows, use Node.js 24.16 or later, or 26.1 or later. Node.js 24.0 to 24.15 and 26.0 can crash at random on local connections. `datanotes serve` warns at start on those versions.
 
+The package installs two commands: `datanotes` and its short name `dnotes`. Every command works under both, so `dnotes status` is the same as `datanotes status`.
+
 1. Start the engine on your folder:
 
    ```bash

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "datanotes",
-  "version": "3.0.0",
+  "version": "3.2.0",
   "mcpName": "io.github.davidefornelli/datanotes",
   "description": "The engine of your second brain: Markdown notes as database tables, for you and your agents.",
   "keywords": [
@@ -28,7 +28,8 @@
   "author": "Davide Fornelli",
   "type": "module",
   "bin": {
-    "datanotes": "dist/datanotes.cjs"
+    "datanotes": "dist/datanotes.cjs",
+    "dnotes": "dist/datanotes.cjs"
   },
   "exports": {
     ".": {