datanotes 3.0.0 → 3.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +1 -5
- package/dist/api/client.d.cts +37 -4
- package/dist/api/client.d.ts +37 -4
- package/dist/api/index.cjs +388 -52
- package/dist/api/index.d.cts +3 -2
- package/dist/api/index.d.ts +3 -2
- package/dist/api/index.js +388 -52
- package/dist/api/mcp.cjs +6 -3
- package/dist/api/mcp.js +6 -3
- package/dist/api/otlp.d.cts +63 -15
- package/dist/api/otlp.d.ts +63 -15
- package/dist/api/redact.d.cts +25 -0
- package/dist/api/redact.d.ts +25 -0
- package/dist/api/tokens.d.cts +5 -0
- package/dist/api/tokens.d.ts +5 -0
- package/dist/api/tracing.d.cts +107 -0
- package/dist/api/tracing.d.ts +107 -0
- package/dist/datanotes.cjs +9380 -2554
- package/dist/docs/administration/limits.md +96 -12
- package/dist/docs/administration/run-the-service.md +18 -0
- package/dist/docs/api/operations/graph_neighbors.md +4 -2
- package/dist/docs/api/operations/graph_path.md +3 -3
- package/dist/docs/api/operations/graph_traverse.md +2 -2
- package/dist/docs/api/operations/record_tool_call.md +1 -0
- package/dist/docs/api/operations/tags/queries.md +2 -2
- package/dist/docs/api.md +3 -3
- package/dist/docs/develop/concurrency.md +36 -1
- package/dist/docs/develop/http-api.md +11 -1
- package/dist/docs/develop/telemetry.md +224 -25
- package/dist/docs/guides/data-quality/check-constraints.md +10 -10
- package/dist/docs/guides/data-quality/hand-edits.md +8 -8
- package/dist/docs/guides/database-objects/bases.md +11 -11
- package/dist/docs/guides/database-objects/catalogs-and-schemas.md +10 -10
- package/dist/docs/guides/database-objects/columns.md +38 -38
- package/dist/docs/guides/database-objects/relations.md +27 -27
- package/dist/docs/guides/database-objects/tables.md +37 -101
- package/dist/docs/guides/database-objects/views.md +44 -44
- package/dist/docs/guides/history/back-up.md +12 -4
- package/dist/docs/guides/history/change-data-feed.md +34 -34
- package/dist/docs/guides/history/journal-and-undo.md +12 -12
- package/dist/docs/guides/history/restore.md +20 -20
- package/dist/docs/guides/history/table-history.md +31 -31
- package/dist/docs/guides/history/vacuum.md +1 -10
- package/dist/docs/guides/read-and-write/automatic-links.md +15 -15
- package/dist/docs/guides/read-and-write/import-notes.md +20 -20
- package/dist/docs/guides/read-and-write/joins-and-aggregates.md +7 -7
- package/dist/docs/guides/read-and-write/read-and-filter.md +10 -10
- package/dist/docs/guides/read-and-write/write-rows.md +10 -10
- package/dist/docs/guides/search/link-graph.md +95 -0
- package/dist/docs/guides/search/text.md +1 -1
- package/dist/docs/guides/security/who-did-what.md +2 -0
- package/dist/docs/reference/cli.md +10 -3
- package/dist/docs/reference/compatibility.md +33 -13
- package/dist/docs/reference/operations/queries.md +3 -3
- package/dist/docs/reference/operations.md +11 -11
- package/dist/docs/reference/settings.md +37 -25
- package/dist/docs/release-notes.md +121 -0
- package/dist/docs/start/how-it-works.md +35 -3
- package/dist/docs/start/quick-start.md +2 -0
- package/package.json +3 -2
|
@@ -50,17 +50,17 @@ Operations that write rows: insert, update, merge, delete, move, adopt and batch
|
|
|
50
50
|
|
|
51
51
|
Operations that read rows and notes: select_rows, link targets, text and semantic search, the link graph. [Query operations](https://datanotes.dev/reference/operations/queries/) lists their parameters.
|
|
52
52
|
|
|
53
|
-
| Operation | What it does
|
|
54
|
-
| ----------------------------------------------------------------------- |
|
|
55
|
-
| [`select_rows`](https://datanotes.dev/reference/operations/queries/#select_rows) | Query rows: `{table, total, rows: [{path, ...}]}`.
|
|
56
|
-
| [`list_link_targets`](https://datanotes.dev/reference/operations/queries/#list_link_targets) | The rows a link column may point to, for one row: what an agent or a picker offers before writing the link.
|
|
57
|
-
| [`read_note`](https://datanotes.dev/reference/operations/queries/#read_note) | Read a note or another file of the vault: `{path, content, front_matter, tags, modified_at, created_at, size}`.
|
|
58
|
-
| [`list_folder`](https://datanotes.dev/reference/operations/queries/#list_folder) | The direct children of a vault folder: `{folder, folders, files}`, each a vault path, sorted; dot-files and dot-folders are left out.
|
|
59
|
-
| [`search_notes`](https://datanotes.dev/reference/operations/queries/#search_notes) | Plain-text search: every word of `query` must occur, case- and accent-insensitive, in the note’s path or text; there are no operators.
|
|
60
|
-
| [`semantic_search`](https://datanotes.dev/reference/operations/queries/#semantic_search) | Find notes by meaning, not exact words: `{query, results: [{path, score, heading, snippet, table?}]}`, one result per note (its best passage), best first.
|
|
61
|
-
| [`graph_neighbors`](https://datanotes.dev/reference/operations/queries/#graph_neighbors) | Direct neighbours of notes: what they link to (`out`) and what links to them (`in`, backlinks), with counts per edge label.
|
|
62
|
-
| [`graph_traverse`](https://datanotes.dev/reference/operations/queries/#graph_traverse) | Breadth-first walk over links from `from_nodes` (`{"paths": [...]}
|
|
63
|
-
| [`graph_path`](https://datanotes.dev/reference/operations/queries/#graph_path) | Shortest chain of links between two sets of notes (each `{"paths": [...]}
|
|
53
|
+
| Operation | What it does | Examples |
|
|
54
|
+
| ----------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
55
|
+
| [`select_rows`](https://datanotes.dev/reference/operations/queries/#select_rows) | Query rows: `{table, total, rows: [{path, ...}]}`. | [Catalogs, schemas and the database](https://datanotes.dev/guides/database-objects/catalogs-and-schemas/), [Tables](https://datanotes.dev/guides/database-objects/tables/), [Columns and types](https://datanotes.dev/guides/database-objects/columns/), [Views](https://datanotes.dev/guides/database-objects/views/), [Bases](https://datanotes.dev/guides/database-objects/bases/), [Relations and hierarchies](https://datanotes.dev/guides/database-objects/relations/), [Validation](https://datanotes.dev/guides/data-quality/validation/), [CHECK constraints](https://datanotes.dev/guides/data-quality/check-constraints/), [Hand edits](https://datanotes.dev/guides/data-quality/hand-edits/), [Read and filter rows](https://datanotes.dev/guides/read-and-write/read-and-filter/), [Joins and aggregates](https://datanotes.dev/guides/read-and-write/joins-and-aggregates/), [Write and move rows](https://datanotes.dev/guides/read-and-write/write-rows/), [Batch writes](https://datanotes.dev/guides/read-and-write/batch/), [Import existing notes](https://datanotes.dev/guides/read-and-write/import-notes/), [Journal and undo](https://datanotes.dev/guides/history/journal-and-undo/), [Table history and time travel](https://datanotes.dev/guides/history/table-history/), [Restore a table or the database](https://datanotes.dev/guides/history/restore/) |
|
|
56
|
+
| [`list_link_targets`](https://datanotes.dev/reference/operations/queries/#list_link_targets) | The rows a link column may point to, for one row: what an agent or a picker offers before writing the link. | [Relations and hierarchies](https://datanotes.dev/guides/database-objects/relations/) |
|
|
57
|
+
| [`read_note`](https://datanotes.dev/reference/operations/queries/#read_note) | Read a note or another file of the vault: `{path, content, front_matter, tags, modified_at, created_at, size}`. | [Relations and hierarchies](https://datanotes.dev/guides/database-objects/relations/), [Hand edits](https://datanotes.dev/guides/data-quality/hand-edits/), [Import existing notes](https://datanotes.dev/guides/read-and-write/import-notes/), [Read and search notes](https://datanotes.dev/guides/search/text/) |
|
|
58
|
+
| [`list_folder`](https://datanotes.dev/reference/operations/queries/#list_folder) | The direct children of a vault folder: `{folder, folders, files}`, each a vault path, sorted; dot-files and dot-folders are left out. | [Catalogs, schemas and the database](https://datanotes.dev/guides/database-objects/catalogs-and-schemas/), [Tables](https://datanotes.dev/guides/database-objects/tables/), [Columns and types](https://datanotes.dev/guides/database-objects/columns/), [Write and move rows](https://datanotes.dev/guides/read-and-write/write-rows/), [Read and search notes](https://datanotes.dev/guides/search/text/) |
|
|
59
|
+
| [`search_notes`](https://datanotes.dev/reference/operations/queries/#search_notes) | Plain-text search: every word of `query` must occur, case- and accent-insensitive, in the note’s path or text; there are no operators. | [Read and search notes](https://datanotes.dev/guides/search/text/) |
|
|
60
|
+
| [`semantic_search`](https://datanotes.dev/reference/operations/queries/#semantic_search) | Find notes by meaning, not exact words: `{query, results: [{path, score, heading, snippet, table?}]}`, one result per note (its best passage), best first. | [Find notes by meaning](https://datanotes.dev/guides/search/semantic/) |
|
|
61
|
+
| [`graph_neighbors`](https://datanotes.dev/reference/operations/queries/#graph_neighbors) | Direct neighbours of notes: what they link to (`out`) and what links to them (`in`, backlinks), with counts per edge label. | [Walk the link graph](https://datanotes.dev/guides/search/link-graph/) |
|
|
62
|
+
| [`graph_traverse`](https://datanotes.dev/reference/operations/queries/#graph_traverse) | Breadth-first walk over links from `from_nodes` (`{"paths": [...]}`, `{"table", "where"}` and/or `{"tables": [...]}`). | [Walk the link graph](https://datanotes.dev/guides/search/link-graph/) |
|
|
63
|
+
| [`graph_path`](https://datanotes.dev/reference/operations/queries/#graph_path) | Shortest chain of links between two sets of notes (each `{"paths": [...]}`, `{"table", "where"}` and/or `{"tables": [...]}`): `found`, `length` and `path`: the steps (`from`, `label`, `direction`, `to`). | [Walk the link graph](https://datanotes.dev/guides/search/link-graph/) |
|
|
64
64
|
|
|
65
65
|
## History
|
|
66
66
|
|
|
@@ -4,28 +4,40 @@
|
|
|
4
4
|
|
|
5
5
|
The headless engine reads `settings.json` in its store, outside the vault (`datanotes store path` shows where), or the file given with `--settings`. Missing keys take their default.
|
|
6
6
|
|
|
7
|
-
| Key | Default | Meaning
|
|
8
|
-
| ---------------------------- | ------------------------- |
|
|
9
|
-
| `journalRetentionDays` | `90` | Journal entries older than this many days are dropped (undo is no longer possible for them).
|
|
10
|
-
| `fileLogEnabled` | `true` | Write log records to `logs/<day>.<source>.jsonl` in the engine’s store.
|
|
11
|
-
| `logRetentionDays` | `30` | Log files older than this many days are deleted.
|
|
12
|
-
| `tableKey` | `"table"` | Row property naming the catalog table a note belongs to (`table: personal.metrics.mood`).
|
|
13
|
-
| `telemetryEnabled` | `false` | Send the engine’s logs to an OpenTelemetry collector or OTLP backend (OTLP/HTTP).
|
|
14
|
-
| `telemetryEndpoint` | `"http://127.0.0.1:4318"` | OTLP/HTTP base URL; logs are POSTed to `<endpoint>/v1/logs`.
|
|
15
|
-
| `telemetryServiceName` | `"datanotes"` | `service.name` resource attribute.
|
|
16
|
-
| `telemetryHeaders` | `{}` | HTTP headers sent with every export: the API key or `Authorization` the backend asks for. Kept in the settings file; `OTEL_EXPORTER_OTLP_HEADERS` adds to them.
|
|
17
|
-
| `telemetryProtocol` | `"http/protobuf"` | OTLP encoding: `http/protobuf` (the OpenTelemetry default) or `http/json`.
|
|
18
|
-
| `
|
|
19
|
-
| `
|
|
20
|
-
| `
|
|
21
|
-
| `
|
|
22
|
-
| `
|
|
23
|
-
| `
|
|
24
|
-
| `
|
|
25
|
-
| `
|
|
26
|
-
| `
|
|
27
|
-
| `
|
|
28
|
-
| `
|
|
29
|
-
| `
|
|
30
|
-
| `
|
|
31
|
-
| `
|
|
7
|
+
| Key | Default | Meaning |
|
|
8
|
+
| ---------------------------- | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
9
|
+
| `journalRetentionDays` | `90` | Journal entries older than this many days are dropped (undo is no longer possible for them). |
|
|
10
|
+
| `fileLogEnabled` | `true` | Write log records to `logs/<day>.<source>.jsonl` in the engine’s store. |
|
|
11
|
+
| `logRetentionDays` | `30` | Log files older than this many days are deleted. |
|
|
12
|
+
| `tableKey` | `"table"` | Row property naming the catalog table a note belongs to (`table: personal.metrics.mood`). |
|
|
13
|
+
| `telemetryEnabled` | `false` | Send the engine’s logs and traces to an OpenTelemetry collector or OTLP backend (OTLP/HTTP). |
|
|
14
|
+
| `telemetryEndpoint` | `"http://127.0.0.1:4318"` | OTLP/HTTP base URL; logs are POSTed to `<endpoint>/v1/logs`, traces to `<endpoint>/v1/traces`. |
|
|
15
|
+
| `telemetryServiceName` | `"datanotes"` | `service.name` resource attribute. |
|
|
16
|
+
| `telemetryHeaders` | `{}` | HTTP headers sent with every export: the API key or `Authorization` the backend asks for. Kept in the settings file; `OTEL_EXPORTER_OTLP_HEADERS` adds to them. |
|
|
17
|
+
| `telemetryProtocol` | `"http/protobuf"` | OTLP encoding: `http/protobuf` (the OpenTelemetry default) or `http/json`. |
|
|
18
|
+
| `telemetryTraces` | `true` | Send traces too while telemetry is on: a span for every HTTP request, operation and phase of an operation (lock, query, checks, journal, vault calls), with its duration and parent. `OTEL_TRACES_EXPORTER=none` turns them off. |
|
|
19
|
+
| `telemetryTraceSampleRatio` | `1` | Share of new traces kept, from 0 to 1 (1: every one). A trace continued from a caller (`traceparent`) follows the caller’s choice. `OTEL_TRACES_SAMPLER` overrides it. |
|
|
20
|
+
| `telemetryMaxSpansPerTrace` | `1000` | Most spans one trace may hold (1 to 100000); the others are dropped and counted in `datanotes.spans_dropped`. |
|
|
21
|
+
| `telemetrySlowSpanMs` | `0` | A request, operation or background job lasting at least this many milliseconds is also logged as a WARN (0: never). |
|
|
22
|
+
| `telemetryEnvironment` | `""` | `deployment.environment.name` of the logs and traces, such as `dev` or `home` (empty: none). |
|
|
23
|
+
| `autoRecompute` | `true` | After a row is edited by hand, recompute its generated / identity columns (a few seconds after the last change, as a journaled `recompute_columns` operation). |
|
|
24
|
+
| `repairLinksOnHandMove` | `true` | Rewrite the links to a note renamed or moved by hand (in a file manager, with git or a sync tool), across the vault, once the vault has been quiet for 4 s and not while git pulls, rebases or merges. The repair is one journaled `repair_links` operation that undo takes back. Links an editor already rewrote stay as they are. |
|
|
25
|
+
| `semanticEnabled` | `false` | Semantic search: embed notes with `semanticProvider` (off by default: content is sent to it). |
|
|
26
|
+
| `semanticProvider` | `"ollama"` | Embedding provider: `ollama`, `lmstudio`, `openrouter`, `openai` or `custom` (any OpenAI-compatible API). |
|
|
27
|
+
| `semanticBaseUrl` | `""` | Empty: the provider’s default URL. |
|
|
28
|
+
| `semanticModel` | `""` | Embedding model, e.g. `nomic-embed-text` (Ollama) or `openai/text-embedding-3-small` (OpenRouter). |
|
|
29
|
+
| `semanticScope` | `"vault"` | “vault”: every note; “tables”: only table rows. |
|
|
30
|
+
| `semanticExclude` | `[]` | Folders left out of the index. |
|
|
31
|
+
| `semanticBatchSize` | `32` | Passages per embedding request. |
|
|
32
|
+
| `writePriority` | `true` | Write priority: while a write runs or waits, new reads wait for it (reads under way finish), so a stream of reads does not slow a write down. Writes start one at a time, in order. |
|
|
33
|
+
| `writePriorityMaxWaitMs` | `100` | The longest a read waits for writes under `writePriority`, in milliseconds (0 to 60000). |
|
|
34
|
+
| `readCacheMaxBytes` | `33554432` | Memory for the answers of repeated reads, in bytes (0: no cache). A read asked again with the same arguments is answered from it while nothing it depends on has changed. |
|
|
35
|
+
| `vaultIndexEnabled` | `true` | Keep an index of the vault’s notes in the engine’s store (`index/vault.sqlite`), so that a start reads only the notes changed since the last one, and `search_notes` only the notes that can match (headless engine, Node 22.13 or later). The index is a cache: deleting it is safe, the next start builds it again. Off: every start and every search reads every note, and the index is deleted. |
|
|
36
|
+
| `httpServerPort` | `27150` | Port of the HTTP API (on 127.0.0.1 unless the service is told otherwise): 1024 to 65535, or 0 for any free port. Any other value stops the engine from starting. |
|
|
37
|
+
| `httpServerToken` | `""` | The owner’s bearer token (generated at the first start). |
|
|
38
|
+
| `httpCompression` | `"remote"` | Which clients of the HTTP API get answers of 16 KB or more compressed with gzip, when their request says `Accept-Encoding: gzip`: `remote` (default) those on other machines, `always` this machine’s too, `off` none. Compressing costs the engine time that a client on the same machine does not win back. |
|
|
39
|
+
| `storeBackupEnabled` | `false` | Back up the engine’s store to a git repository: commits as `storeBackupIntervalSeconds` says, each pushed to `storeBackupRemote` (headless engine; `datanotes store backup` sets it). |
|
|
40
|
+
| `storeBackupRemote` | `""` | The git remote the store backup pushes to, such as a private GitHub repository. Empty: commits in the store only, nothing pushed. |
|
|
41
|
+
| `storeBackupIntervalSeconds` | `60` | How often the store backup looks for changes, in seconds (at least 10): when something changed, it commits between two operations (a consistent snapshot), then pushes. |
|
|
42
|
+
| `storeSync` | `"operation"` | How the engine makes its store files durable: `operation` syncs (fsync) every store file an operation wrote once, when the operation ends (the journal line, the transaction log, the blobs; the change feed after each of its appends); `off` leaves it to the operating system, which is faster but lets a power loss take the last operations’ undo data and history. Notes are never synced (they are written atomically, as Obsidian writes them). Headless engine. |
|
|
43
|
+
| `transactionLogFormat` | `"segments"` | How the transaction log writes new versions: `segments` (default since 3.1) appends them to a few large files (far fewer files for the antivirus, git and the store backup), which Datanotes 3.0 cannot read: run `datanotes store convert-log --to files` before you go back to it; `files` writes one file per version, which every Datanotes 3 reads. The engine reads both, also mixed in one folder: a log of 3.0 keeps its files and grows in segments. |
|
|
@@ -4,6 +4,127 @@
|
|
|
4
4
|
|
|
5
5
|
This page lists each version of Datanotes, newest first. Datanotes follows [semantic versioning](https://semver.org): a major version can break what you rely on, and its notes list those changes first. The package is on [npm](https://www.npmjs.com/package/datanotes), and each version has a [GitHub release](https://github.com/davidefornelli/datanotes/releases).
|
|
6
6
|
|
|
7
|
+
## 3.1.0
|
|
8
|
+
|
|
9
|
+
Datanotes 3.1.0 makes the engine fast at scale and adds OpenTelemetry traces. It only adds to 3.0: no operation, argument or setting goes away. One default changes: the transaction log now writes segments. Read [Before you downgrade to 3.0](#before-you-downgrade-to-30) if you may go back.
|
|
10
|
+
|
|
11
|
+
### Speed
|
|
12
|
+
|
|
13
|
+
Measured with `bench/scale/` on a synthetic 50,000-note vault on Windows, median times. “Before” is the engine at the start of this work, with the same benchmark:
|
|
14
|
+
|
|
15
|
+
| Operation | Before | 3.1.0 |
|
|
16
|
+
| ------------------------------------- | -------- | ------ |
|
|
17
|
+
| `insert_row` | 662 ms | 37 ms |
|
|
18
|
+
| `update_row` | 124 ms | 47 ms |
|
|
19
|
+
| `select_rows` on a small table | 23 ms | 2.8 ms |
|
|
20
|
+
| `describe_database` | 1,019 ms | 24 ms |
|
|
21
|
+
| `graph_neighbors` right after a write | 1,472 ms | 2.5 ms |
|
|
22
|
+
| Archive 100 rows | 71.6 s | 1.3 s |
|
|
23
|
+
| `move_rows`, 20 rows | 13.9 s | 0.5 s |
|
|
24
|
+
| `select_rows` by 5 paths | 36.5 ms | 0.3 ms |
|
|
25
|
+
|
|
26
|
+
The gains come from these changes:
|
|
27
|
+
|
|
28
|
+
* **An index of the vault in memory**: the engine keeps each table’s rows, unique values, links and backlinks up to date as notes change. A read no longer walks every note of the vault.
|
|
29
|
+
|
|
30
|
+
* **Links resolved one note at a time**: creating, deleting or renaming a note no longer resolves every link of the vault again.
|
|
31
|
+
|
|
32
|
+
* **Each note parsed once per version**: the engine keeps the parse until the file changes.
|
|
33
|
+
|
|
34
|
+
* **Faster queries**:
|
|
35
|
+
|
|
36
|
+
* `where` filters are compiled once;
|
|
37
|
+
* sorts read each key once;
|
|
38
|
+
* pages keep only the rows they need;
|
|
39
|
+
* time-zone formatting is cached, so sorting 10,000 rows by `file_modified` takes about 50 ms, not 4 s.
|
|
40
|
+
|
|
41
|
+
* **Writes before reads**: a write waits for no new read. A write under four concurrent readers takes about 35 ms, not 147 ms. Settings `writePriority` and `writePriorityMaxWaitMs` tune it.
|
|
42
|
+
|
|
43
|
+
* **Search through the index on disk**: `search_notes` looks the words up in the index on disk, then reads only the notes that can match. On 50,000 notes, two words found in 1,213 notes take about 14 ms, not 70 to 83 ms. A name takes 7 ms, not 100 ms. A word found in 15,905 notes still takes about 140 ms. A query of one- or two-letter words reads every note. After a start, the engine reads the search’s row ids in the background. The first search no longer waits for them.
|
|
44
|
+
|
|
45
|
+
* **Repeated reads from memory**: a read the engine answered before, on data that has not changed since, comes from a cache. Setting `readCacheMaxBytes` sizes it, and `0` turns it off.
|
|
46
|
+
|
|
47
|
+
* **Faster undo**: `undo_migration` takes tens of milliseconds, not seconds.
|
|
48
|
+
|
|
49
|
+
* **Faster history**:
|
|
50
|
+
|
|
51
|
+
* `table_changes` replays each version once;
|
|
52
|
+
* `describe_history` reads only commit headers;
|
|
53
|
+
* repeated time-travel reads come from memory.
|
|
54
|
+
|
|
55
|
+
* **Less memory**:
|
|
56
|
+
|
|
57
|
+
* note texts stay within a budget;
|
|
58
|
+
* repeated values, names and paths are kept once;
|
|
59
|
+
* row bodies are read only when asked for;
|
|
60
|
+
* the transaction log keeps hashes, not texts.
|
|
61
|
+
|
|
62
|
+
* **A faster start**:
|
|
63
|
+
|
|
64
|
+
* the engine walks the vault once, without blocking;
|
|
65
|
+
* it reads many notes at a time;
|
|
66
|
+
* the change feed starts after the server listens.
|
|
67
|
+
|
|
68
|
+
* **An index on disk**: the engine keeps `<store>/index/vault.sqlite` through Node.js’s built-in SQLite. A start then reads only the notes that changed since the last stop: at 50,000 notes a restart takes about 2 s, not 40 s. The index is a cache: the notes stay the source of truth, and the engine rebuilds the index when it is missing or damaged. It holds each note’s text, compressed, and takes about twice the vault’s size on disk. Setting `vaultIndexEnabled` turns it off. Without `node:sqlite`, the engine reads every note as before.
|
|
69
|
+
|
|
70
|
+
* **More reads at once**: the `datanotes` command gives Node.js 16 threads for file reads, unless `UV_THREADPOOL_SIZE` sets another number.
|
|
71
|
+
|
|
72
|
+
### The whole graph in one call
|
|
73
|
+
|
|
74
|
+
`graph_neighbors`, `graph_traverse` and `graph_path` take `from_nodes.tables`: table names or patterns, with `"*"` for every table. One call returns the graph of the whole vault.
|
|
75
|
+
|
|
76
|
+
### Traces
|
|
77
|
+
|
|
78
|
+
The engine and `datanotes mcp` send OpenTelemetry traces next to the logs, to any OTLP backend:
|
|
79
|
+
|
|
80
|
+
* one span per request, continuing an incoming `traceparent`;
|
|
81
|
+
* one span per operation, with the database span attributes;
|
|
82
|
+
* spans for the phases: lock wait and hold, query plan and scan, validation, journal and transaction log;
|
|
83
|
+
* one trace per agent call, across the MCP server and the engine.
|
|
84
|
+
|
|
85
|
+
Log records carry their trace and span ids. New settings: `telemetryTraces`, `telemetryTraceSampleRatio`, `telemetryMaxSpansPerTrace`, `telemetrySlowSpanMs` and `telemetryEnvironment`. The `OTEL_*` variables work as well. `datanotes/client` takes `trace` and `tracer` to continue a trace into the engine. See [Send logs to your observability backend](https://datanotes.dev/develop/telemetry/).
|
|
86
|
+
|
|
87
|
+
### The store
|
|
88
|
+
|
|
89
|
+
* **The transaction log in segments, by default**: a table’s versions go to append-only segment files instead of one file per version. The store keeps far fewer files, which matters to antivirus scans and to the store backup. Setting `transactionLogFormat` chooses `segments` or `files`. A 3.0 store needs no conversion: new versions go to segments next to the old files.
|
|
90
|
+
* **Smaller checkpoints**: a checkpoint is a manifest and compressed parts, written after the write that made it due.
|
|
91
|
+
* **Durable store writes**: the files an operation wrote are synced at its end. Setting `storeSync` sets the policy. A version cut short by a crash no longer makes a table’s history unreadable. `vacuum_table` and `datanotes store convert-log` sync the files they write before they delete the old ones.
|
|
92
|
+
* **A lighter backup**: a backup tick with no change runs no git command.
|
|
93
|
+
* **The store left out of indexing**: on Windows the engine marks the store folder as not content indexed, and on macOS it adds a Spotlight marker.
|
|
94
|
+
* **The vault index out of Time Machine**: on macOS the engine excludes `<store>/index/` from Time Machine. The engine rebuilds the index when it is missing, so a restore loses nothing. Time Machine still backs up the journal and the history.
|
|
95
|
+
* **The vault index checked**: after each start, the engine checks the index file in the background, one table at a time. The engine turns a damaged index off for the run and rebuilds it at the next start.
|
|
96
|
+
|
|
97
|
+
### Before you downgrade to 3.0
|
|
98
|
+
|
|
99
|
+
Datanotes 3.0 reads only one file per version. Before you go back to 3.0, convert the logs:
|
|
100
|
+
|
|
101
|
+
1. Stop the engine. The conversion runs only while no engine serves the database.
|
|
102
|
+
|
|
103
|
+
2. Convert the logs of the database:
|
|
104
|
+
|
|
105
|
+
```bash
|
|
106
|
+
datanotes store convert-log --to files --database <folder>
|
|
107
|
+
```
|
|
108
|
+
|
|
109
|
+
3. Start Datanotes 3.0.
|
|
110
|
+
|
|
111
|
+
A table vacuumed with more than 32 MB of text keeps a checkpoint 3.0 cannot read, so 3.0 starts its history again from empty. To keep one file per version from the start, set `transactionLogFormat` to `files`.
|
|
112
|
+
|
|
113
|
+
### Other changes
|
|
114
|
+
|
|
115
|
+
* **`dnotes`**: the package also installs the command under a short name. `dnotes status` is the same as `datanotes status`, and `datanotes` stays.
|
|
116
|
+
* **`search_notes`**: match contexts now fall on the right text in notes with decomposed accents, Hangul or emoji.
|
|
117
|
+
* **Compressed answers**: setting `httpCompression` sends gzip to remote clients. It is on by default, and clients on the same machine get plain answers.
|
|
118
|
+
* **Semantic search**: vectors take a quarter of the memory, and a repeated query skips the embedding service.
|
|
119
|
+
* **`select_rows` by path**: a filter on `path` reads only the notes it names. A row elsewhere in the table that can’t be read no longer fails it.
|
|
120
|
+
* **Index checks**: `DATANOTES_INDEX_CHECK=1` compares the engine’s indexes with a full read after every operation.
|
|
121
|
+
* **`/tool-calls/record/`**: takes an optional `summary`, so `datanotes mcp` no longer sends whole tool results back to the engine.
|
|
122
|
+
* **Benchmarks**: `bench/http-latency.mjs` and `bench/scale/` measure any vault.
|
|
123
|
+
* **`datanotes store path --json`**: the answer adds `database_path`, the database folder on disk.
|
|
124
|
+
* **`datanotes store new-id`, `backup` and `restore`**: each holds the store while it runs, as `convert-log` does. An engine you start meanwhile refuses to start. A live engine whose heartbeat stopped now makes them refuse too. `restore` leaves the store in place untouched until the clone is checked. When the clone cannot take its place, `restore` removes the clone. It puts the old store back, or names the folder the old store is in.
|
|
125
|
+
* **Traces of background work**: `job auto_recompute` recomputes columns after a hand edit, and `job inverse_links` mirrors inverse links. Both are traces of their own, never part of the request that saw the edit.
|
|
126
|
+
* **Writes**: a write runs again only when another write overtook it. Before, a write whose values held the text of that conflict ran up to five times.
|
|
127
|
+
|
|
7
128
|
## 3.0.0
|
|
8
129
|
|
|
9
130
|
Datanotes 3.0.0 is the first stable release. Versions 1.x and 2.x, published on 7 and 8 October 2026, were previews. Datanotes runs a folder of Markdown notes as typed database tables. You reach it through an HTTP API, a command line and an MCP server for agents.
|
|
@@ -80,14 +80,43 @@ The transaction log records each version of a table, so you can read or restore
|
|
|
80
80
|
|
|
81
81
|
Change the database’s setting with [`configure_database`](https://datanotes.dev/reference/operations/auxiliary/#configure_database) and `history`. [Table history and time travel](https://datanotes.dev/guides/history/table-history/) shows it at work.
|
|
82
82
|
|
|
83
|
-
Each table’s log lives in the engine’s store, in `transaction_log/your_catalog.your_schema.your_table
|
|
83
|
+
Each table’s log lives in the engine’s store, in `transaction_log/your_catalog.your_schema.your_table/`. Each version is a commit with these entries:
|
|
84
84
|
|
|
85
85
|
* **`commitInfo`**: the operation, its time and, when known, its actor
|
|
86
86
|
* **`metaData`**: the table’s definition, when the commit changes it
|
|
87
87
|
* **`add`**: the path, the hash and the full content of each row written
|
|
88
88
|
* **`remove`**: the path and the hash of each row removed
|
|
89
89
|
|
|
90
|
-
The engine writes a checkpoint, a full copy of the table, once the commits since the last checkpoint outweigh it. That means twice the table’s size and at least 256 KB,
|
|
90
|
+
The engine writes a checkpoint, a full copy of the table, once the commits since the last checkpoint outweigh it. That means twice the table’s size and at least 256 KB. With one file per version, it also writes one after 1,000 versions, or after one version per 16 KB of rows for a large table. The log therefore grows with what you write, not with the table’s size. Hand edits appear as `EXTERNAL_EDIT` commits.
|
|
91
|
+
|
|
92
|
+
### How the engine writes the log
|
|
93
|
+
|
|
94
|
+
The setting `transactionLogFormat` picks how the engine writes new versions:
|
|
95
|
+
|
|
96
|
+
* **`segments`**, the default since 3.1: versions added to the end of a few large files, such as `00000000000000001000.segment.jsonl`. Fewer files mean less work for an antivirus, git and the store backup. Datanotes 3.0 can’t read them.
|
|
97
|
+
* **`files`**: one numbered JSON file per version, such as `00000000000000000007.json`. Every Datanotes 3 release reads it.
|
|
98
|
+
|
|
99
|
+
A segment holds up to 1,000 versions or 8 MB. The engine then seals it: it renames the file after its first and last version, and never writes to it again. The engine reads both formats, also mixed in one folder.
|
|
100
|
+
|
|
101
|
+
A store from Datanotes 3.0 needs no conversion. Its versions stay in their files, and the new versions go to segments in the same folder. To keep one file per version, set `transactionLogFormat` to `files`.
|
|
102
|
+
|
|
103
|
+
A checkpoint is a manifest and its parts. The manifest, `your_version.manifest.jsonl`, lists each row’s path and hash. The parts, `your_version.part-0.jsonl.gz`, hold the rows’ texts, compressed. No file holds more than 4 MB of text, far under the 95 MB the store backup pushes. The engine writes a checkpoint after the write that made it due, so the write doesn’t wait for it. When it starts, it reads the manifests only.
|
|
104
|
+
|
|
105
|
+
A vacuum writes its checkpoint as one file, `your_version.checkpoint.json`, which Datanotes 3.0 reads too. A table with more than 32 MB of text gets a manifest instead.
|
|
106
|
+
|
|
107
|
+
A crash or a power loss can leave the newest version half written. The engine leaves that version out, as if it never wrote it. The next write sets the file aside as `your_file.torn-your_time` and writes the version again.
|
|
108
|
+
|
|
109
|
+
### Convert the log to another format
|
|
110
|
+
|
|
111
|
+
To switch formats, stop the engine and run `datanotes store convert-log`:
|
|
112
|
+
|
|
113
|
+
```bash
|
|
114
|
+
datanotes store convert-log --to segments --vault ~/notes
|
|
115
|
+
```
|
|
116
|
+
|
|
117
|
+
The command rewrites every table’s log in the new format and sets `transactionLogFormat` to match. Run it with `--to files` before you go back to Datanotes 3.0, as [Compatibility](https://datanotes.dev/reference/compatibility/#downgrade-to-30) explains.
|
|
118
|
+
|
|
119
|
+
The command holds the store while it runs. An engine you start meanwhile refuses to start.
|
|
91
120
|
|
|
92
121
|
History costs disk space in the store. Each version keeps the full text of every note it writes, so a note saved ten times is stored about ten times, plus the checkpoints. A table you rewrite often grows its log fastest. [`vacuum_table`](https://datanotes.dev/reference/operations/history/#vacuum_table) deletes the old versions, and `history: false` stops the log of a table that doesn’t need it.
|
|
93
122
|
|
|
@@ -135,9 +164,12 @@ The store holds these files:
|
|
|
135
164
|
* **`blobs/your_hash.txt`**: texts of 2 KB or more, such as `metastore.yaml` before and after a change, stored once
|
|
136
165
|
* **`transaction_log/`**, **`changes/`** and **`semantic/`**: the history of tables, the change feed and the semantic index
|
|
137
166
|
* **`logs/YYYY-MM-DD.your_source.jsonl`**: the engine’s log records, when `fileLogEnabled` is on
|
|
167
|
+
* **`index/vault.sqlite`**: the vault index, what the engine read from each note, so a start reads only the notes changed since; you can delete it at any time
|
|
138
168
|
* **`store.json`**: the vault and the database folder the store belongs to
|
|
139
169
|
|
|
140
|
-
When the engine starts, it drops journal entries older than `journalRetentionDays` (90)
|
|
170
|
+
When the engine starts, and then once a day, it drops journal entries older than `journalRetentionDays` (90). It also deletes log files older than `logRetentionDays` (30). It reads the journal without the write lock, so writes don’t wait for it.
|
|
171
|
+
|
|
172
|
+
At the end of each operation, the engine syncs the store files it wrote to the disk: the journal line, the new versions, the blobs and the change feed. A power loss then can’t take the undo data or the history of an operation that returned. The setting `storeSync` set to `off` skips that, for faster writes. The engine never syncs notes: it writes them atomically, as Obsidian does.
|
|
141
173
|
|
|
142
174
|
Your vault holds only your notes: sync tools, git and Obsidian never see the store. The store belongs to the machine the engine runs on, so the journal and the history do too. Results name store files with a `store:` prefix, such as `store:journal/2026-10.jsonl`.
|
|
143
175
|
|
|
@@ -8,6 +8,8 @@ You need [Node.js](https://nodejs.org) 22 or newer and a folder for your databas
|
|
|
8
8
|
|
|
9
9
|
On Windows, use Node.js 24.16 or later, or 26.1 or later. Node.js 24.0 to 24.15 and 26.0 can crash at random on local connections. `datanotes serve` warns at start on those versions.
|
|
10
10
|
|
|
11
|
+
The package installs two commands: `datanotes` and its short name `dnotes`. Every command works under both, so `dnotes status` is the same as `datanotes status`.
|
|
12
|
+
|
|
11
13
|
1. Start the engine on your folder:
|
|
12
14
|
|
|
13
15
|
```bash
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "datanotes",
|
|
3
|
-
"version": "3.
|
|
3
|
+
"version": "3.1.0",
|
|
4
4
|
"mcpName": "io.github.davidefornelli/datanotes",
|
|
5
5
|
"description": "The engine of your second brain: Markdown notes as database tables, for you and your agents.",
|
|
6
6
|
"keywords": [
|
|
@@ -28,7 +28,8 @@
|
|
|
28
28
|
"author": "Davide Fornelli",
|
|
29
29
|
"type": "module",
|
|
30
30
|
"bin": {
|
|
31
|
-
"datanotes": "dist/datanotes.cjs"
|
|
31
|
+
"datanotes": "dist/datanotes.cjs",
|
|
32
|
+
"dnotes": "dist/datanotes.cjs"
|
|
32
33
|
},
|
|
33
34
|
"exports": {
|
|
34
35
|
".": {
|