datanotes 2.0.1 → 3.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/NOTICE +9 -0
- package/README.md +203 -28
- package/SECURITY.md +92 -45
- package/dist/THIRD-PARTY-NOTICES.txt +682 -0
- package/dist/api/catalog.d.cts +69 -0
- package/dist/api/catalog.d.ts +50 -2
- package/dist/api/cli-options.d.cts +39 -0
- package/dist/api/cli-options.d.ts +39 -0
- package/dist/api/client.d.cts +122 -0
- package/dist/api/client.d.ts +56 -34
- package/dist/api/docs.d.cts +211 -0
- package/dist/api/docs.d.ts +211 -0
- package/dist/api/guidance.d.cts +81 -0
- package/dist/api/guidance.d.ts +81 -0
- package/dist/api/index.cjs +7470 -2771
- package/dist/api/index.d.cts +14 -0
- package/dist/api/index.d.ts +11 -7
- package/dist/api/index.js +7470 -2771
- package/dist/api/local.cjs +118 -6
- package/dist/api/local.d.cts +95 -0
- package/dist/api/local.d.ts +70 -6
- package/dist/api/local.js +104 -6
- package/dist/api/mcp.cjs +17387 -0
- package/dist/api/mcp.d.cts +108 -0
- package/dist/api/mcp.d.ts +108 -0
- package/dist/api/mcp.js +17337 -0
- package/dist/api/otlp.d.cts +113 -0
- package/dist/api/otlp.d.ts +63 -15
- package/dist/api/redact.d.cts +42 -0
- package/dist/api/redact.d.ts +42 -0
- package/dist/api/requests.d.cts +26 -0
- package/dist/api/requests.d.ts +14 -16
- package/dist/api/results.d.cts +16 -0
- package/dist/api/results.d.ts +2 -1
- package/dist/api/tokens.d.cts +12 -0
- package/dist/api/tokens.d.ts +5 -0
- package/dist/api/tracing.d.cts +107 -0
- package/dist/api/tracing.d.ts +107 -0
- package/dist/datanotes.cjs +69597 -28329
- package/dist/docs/administration/configuration.md +59 -0
- package/dist/docs/administration/docker.md +80 -0
- package/dist/docs/administration/git-sync.md +71 -0
- package/dist/docs/administration/limits.md +153 -0
- package/dist/docs/administration/run-the-service.md +237 -0
- package/dist/docs/administration/status.md +61 -0
- package/dist/docs/administration/use-with-obsidian.md +75 -0
- package/dist/docs/api/operations/add_column.md +49 -0
- package/dist/docs/api/operations/add_constraint.md +55 -0
- package/dist/docs/api/operations/adopt_rows.md +50 -0
- package/dist/docs/api/operations/align_relations.md +41 -0
- package/dist/docs/api/operations/alter_column.md +46 -0
- package/dist/docs/api/operations/alter_table.md +67 -0
- package/dist/docs/api/operations/batch.md +48 -0
- package/dist/docs/api/operations/changes_stream.md +28 -0
- package/dist/docs/api/operations/check_table.md +29 -0
- package/dist/docs/api/operations/clone_table.md +48 -0
- package/dist/docs/api/operations/comment_on.md +42 -0
- package/dist/docs/api/operations/configure_database.md +52 -0
- package/dist/docs/api/operations/create_table.md +78 -0
- package/dist/docs/api/operations/create_view.md +54 -0
- package/dist/docs/api/operations/create_webhook.md +37 -0
- package/dist/docs/api/operations/define_base.md +52 -0
- package/dist/docs/api/operations/delete_rows.md +43 -0
- package/dist/docs/api/operations/delete_webhook.md +31 -0
- package/dist/docs/api/operations/describe_database.md +30 -0
- package/dist/docs/api/operations/describe_history.md +40 -0
- package/dist/docs/api/operations/describe_semantic_index.md +24 -0
- package/dist/docs/api/operations/describe_table.md +38 -0
- package/dist/docs/api/operations/docs.md +13 -0
- package/dist/docs/api/operations/drop_column.md +41 -0
- package/dist/docs/api/operations/drop_constraint.md +39 -0
- package/dist/docs/api/operations/drop_table.md +43 -0
- package/dist/docs/api/operations/drop_view.md +38 -0
- package/dist/docs/api/operations/engine_status.md +23 -0
- package/dist/docs/api/operations/graph_neighbors.md +34 -0
- package/dist/docs/api/operations/graph_path.md +32 -0
- package/dist/docs/api/operations/graph_traverse.md +43 -0
- package/dist/docs/api/operations/identify_session.md +24 -0
- package/dist/docs/api/operations/infer_table.md +43 -0
- package/dist/docs/api/operations/insert_row.md +48 -0
- package/dist/docs/api/operations/list_changes.md +35 -0
- package/dist/docs/api/operations/list_folder.md +28 -0
- package/dist/docs/api/operations/list_link_targets.md +44 -0
- package/dist/docs/api/operations/list_migrations.md +35 -0
- package/dist/docs/api/operations/list_tables.md +32 -0
- package/dist/docs/api/operations/list_webhooks.md +23 -0
- package/dist/docs/api/operations/manual_index.md +14 -0
- package/dist/docs/api/operations/manual_page.md +22 -0
- package/dist/docs/api/operations/merge_rows.md +44 -0
- package/dist/docs/api/operations/move_rows.md +52 -0
- package/dist/docs/api/operations/openapi.md +13 -0
- package/dist/docs/api/operations/openapi_post.md +20 -0
- package/dist/docs/api/operations/read_note.md +29 -0
- package/dist/docs/api/operations/recompute_columns.md +44 -0
- package/dist/docs/api/operations/record_tool_call.md +37 -0
- package/dist/docs/api/operations/rename_column.md +44 -0
- package/dist/docs/api/operations/rename_value.md +45 -0
- package/dist/docs/api/operations/restore_database.md +51 -0
- package/dist/docs/api/operations/restore_table.md +46 -0
- package/dist/docs/api/operations/search_notes.md +28 -0
- package/dist/docs/api/operations/select_rows.md +61 -0
- package/dist/docs/api/operations/semantic_search.md +34 -0
- package/dist/docs/api/operations/semantic_search_docs.md +26 -0
- package/dist/docs/api/operations/set_table_bases.md +44 -0
- package/dist/docs/api/operations/status.md +13 -0
- package/dist/docs/api/operations/stop_engine.md +23 -0
- package/dist/docs/api/operations/sync_semantic_index.md +33 -0
- package/dist/docs/api/operations/table_changes.md +31 -0
- package/dist/docs/api/operations/tags/auxiliary.md +14 -0
- package/dist/docs/api/operations/tags/change-feed.md +8 -0
- package/dist/docs/api/operations/tags/history.md +17 -0
- package/dist/docs/api/operations/tags/queries.md +15 -0
- package/dist/docs/api/operations/tags/rows-dml.md +16 -0
- package/dist/docs/api/operations/tags/schema-ddl.md +22 -0
- package/dist/docs/api/operations/tags/service.md +15 -0
- package/dist/docs/api/operations/tags/tokens.md +13 -0
- package/dist/docs/api/operations/tokens_create.md +34 -0
- package/dist/docs/api/operations/tokens_delete.md +29 -0
- package/dist/docs/api/operations/tokens_list.md +21 -0
- package/dist/docs/api/operations/tokens_me.md +20 -0
- package/dist/docs/api/operations/tokens_rotate.md +29 -0
- package/dist/docs/api/operations/tokens_update.md +34 -0
- package/dist/docs/api/operations/undo_migration.md +46 -0
- package/dist/docs/api/operations/update_row.md +60 -0
- package/dist/docs/api/operations/update_rows.md +46 -0
- package/dist/docs/api/operations/vacuum_table.md +37 -0
- package/dist/docs/api.md +135 -0
- package/dist/docs/develop/build-an-app.md +320 -0
- package/dist/docs/develop/build-on-datanotes.md +138 -0
- package/dist/docs/develop/change-feed-and-webhooks.md +70 -0
- package/dist/docs/develop/client.md +116 -0
- package/dist/docs/develop/concurrency.md +188 -0
- package/dist/docs/develop/http-api.md +89 -0
- package/dist/docs/develop/telemetry.md +285 -0
- package/dist/docs/guides/agents/docs-offline.md +99 -0
- package/dist/docs/guides/agents/how-agents-use-datanotes.md +124 -0
- package/dist/docs/guides/agents/prompts.md +495 -0
- package/dist/docs/guides/agents/reading-errors.md +112 -0
- package/dist/docs/guides/data-quality/check-constraints.md +379 -0
- package/dist/docs/guides/data-quality/check-table.md +63 -0
- package/dist/docs/guides/data-quality/hand-edits.md +507 -0
- package/dist/docs/guides/data-quality/sync-between-devices.md +51 -0
- package/dist/docs/guides/data-quality/validation.md +90 -0
- package/dist/docs/guides/database-objects/bases.md +480 -0
- package/dist/docs/guides/database-objects/catalogs-and-schemas.md +897 -0
- package/dist/docs/guides/database-objects/columns.md +1332 -0
- package/dist/docs/guides/database-objects/relations.md +1454 -0
- package/dist/docs/guides/database-objects/tables.md +1340 -0
- package/dist/docs/guides/database-objects/views.md +1003 -0
- package/dist/docs/guides/history/back-up.md +203 -0
- package/dist/docs/guides/history/change-data-feed.md +247 -0
- package/dist/docs/guides/history/journal-and-undo.md +262 -0
- package/dist/docs/guides/history/restore.md +341 -0
- package/dist/docs/guides/history/table-history.md +657 -0
- package/dist/docs/guides/history/vacuum.md +26 -0
- package/dist/docs/guides/read-and-write/automatic-links.md +768 -0
- package/dist/docs/guides/read-and-write/batch.md +139 -0
- package/dist/docs/guides/read-and-write/import-notes.md +896 -0
- package/dist/docs/guides/read-and-write/joins-and-aggregates.md +251 -0
- package/dist/docs/guides/read-and-write/read-and-filter.md +536 -0
- package/dist/docs/guides/read-and-write/write-rows.md +573 -0
- package/dist/docs/guides/search/link-graph.md +241 -0
- package/dist/docs/guides/search/semantic.md +133 -0
- package/dist/docs/guides/search/text.md +142 -0
- package/dist/docs/guides/security/security-model.md +110 -0
- package/dist/docs/guides/security/tokens.md +60 -0
- package/dist/docs/guides/security/who-did-what.md +44 -0
- package/dist/docs/index.md +113 -0
- package/dist/docs/llms.txt +33 -0
- package/dist/docs/reference/cli.md +143 -0
- package/dist/docs/reference/compatibility.md +83 -0
- package/dist/docs/reference/errors.md +72 -0
- package/dist/docs/reference/glossary.md +161 -0
- package/dist/docs/reference/operations/auxiliary.md +116 -0
- package/dist/docs/reference/operations/history.md +168 -0
- package/dist/docs/reference/operations/queries.md +169 -0
- package/dist/docs/reference/operations/rows.md +185 -0
- package/dist/docs/reference/operations/schema.md +316 -0
- package/dist/docs/reference/operations.md +96 -0
- package/dist/docs/reference/settings.md +43 -0
- package/dist/docs/release-notes.md +188 -0
- package/dist/docs/start/connect-agents.md +204 -0
- package/dist/docs/start/how-it-works.md +349 -0
- package/dist/docs/start/quick-start.md +128 -0
- package/package.json +60 -15
|
@@ -0,0 +1,70 @@
|
|
|
1
|
+
# Follow the change feed and webhooks
|
|
2
|
+
|
|
3
|
+
> React to every change in the vault from a program: poll or long-poll the change feed, stream it, or have the engine post it to a webhook.
|
|
4
|
+
|
|
5
|
+
The change feed numbers every change in the vault: writes through the engine, hand edits with the rules they break, schema changes and, when asked for, the tool calls of agents. A program follows it to react when a note changes, without reading the tables again. [Change data feed](https://datanotes.dev/guides/history/change-data-feed/) shows its events on a real engine.
|
|
6
|
+
|
|
7
|
+
## Choose how to follow the feed
|
|
8
|
+
|
|
9
|
+
A program learns about changes in three ways:
|
|
10
|
+
|
|
11
|
+
* **Polling**: `POST /datanotes/operations/list_changes/` returns the changes since a point in the feed
|
|
12
|
+
* **Streaming**: `GET /datanotes/changes/stream` sends each change as a Server-Sent Events (SSE) message
|
|
13
|
+
* **Webhooks**: the owner creates them, and the engine posts change events to their URL
|
|
14
|
+
|
|
15
|
+
Each event has its number, `seq`, the table, its `kind` (`insert`, `update`, `delete`, `move` or `metastore`), the operation that made it in `op`, the note’s path, the changed columns with their values before and after, and the actor. A hand edit has `source: "external"`, and its last event in a batch lists the rules it breaks in `violations`. When you rename a note by hand, the engine repairs the links to it. The events of that repair come after the move and name it in `cause`. For a row, `cause` also gives the `seq` of its `move` event.
|
|
16
|
+
|
|
17
|
+
## Long-poll the feed from code
|
|
18
|
+
|
|
19
|
+
`list_changes` returns the changes after `since`, oldest first. Pass the answer’s `last` as `since` on the next call, and `wait_ms` (at most 60,000) to wait for the next change instead of polling. `head` is the feed’s newest number, so a program that starts from `head` sees only what happens after it starts:
|
|
20
|
+
|
|
21
|
+
```ts
|
|
22
|
+
type Feed = {
|
|
23
|
+
head: number;
|
|
24
|
+
last: number;
|
|
25
|
+
events: { kind: string; path: string }[];
|
|
26
|
+
};
|
|
27
|
+
|
|
28
|
+
|
|
29
|
+
const first = await db.call("list_changes", { limit: 1 });
|
|
30
|
+
if (!first.ok) throw new Error(String(first.value));
|
|
31
|
+
let since = (first.value as Feed).head;
|
|
32
|
+
for (;;) {
|
|
33
|
+
const r = await db.call("list_changes", {
|
|
34
|
+
since,
|
|
35
|
+
wait_ms: 20_000,
|
|
36
|
+
tables: ["film"],
|
|
37
|
+
});
|
|
38
|
+
if (!r.ok) throw new Error(String(r.value));
|
|
39
|
+
const feed = r.value as Feed;
|
|
40
|
+
for (const event of feed.events) console.log(event.kind, event.path);
|
|
41
|
+
since = feed.last;
|
|
42
|
+
}
|
|
43
|
+
```
|
|
44
|
+
|
|
45
|
+
The filters narrow the feed before it reaches you:
|
|
46
|
+
|
|
47
|
+
* **`tables`**, **`kinds`** and **`columns`**: the tables, kinds of change and columns to keep
|
|
48
|
+
* **`where`**: rows that match a filter, as in `select_rows`, on the values after the change
|
|
49
|
+
* **`exclude_actor`**: leaves out the changes of one harness, such as the program’s own writes when it sends its name as `harness`
|
|
50
|
+
|
|
51
|
+
An answer with `gap: true` means the engine dropped older changes that you hadn’t read: read the tables again. To pick up where the program stopped after a restart, save `last` and start from it.
|
|
52
|
+
|
|
53
|
+
## Stream the feed
|
|
54
|
+
|
|
55
|
+
`GET /datanotes/changes/stream` sends each change as a Server-Sent Events (SSE) message, with its number as the event’s `id`. It takes the same filters as query parameters, and resumes from `Last-Event-ID` or `since`. A comment line every 25s keeps proxies from closing the connection, and an `event: gap` message tells the client that events were dropped.
|
|
56
|
+
|
|
57
|
+
## Receive changes with a webhook
|
|
58
|
+
|
|
59
|
+
A webhook has the engine post the feed to a web address, for a service that would rather be called than poll. Only the owner token creates webhooks, with [`create_webhook`](https://datanotes.dev/reference/operations/history/#create_webhook), and the answer shows the webhook’s `secret` once: give it to the receiver. An agent connected with `datanotes mcp` gets `create_webhook` only with `--owner-tools` or `--profile all` ([Choose the tools an agent sees](https://datanotes.dev/start/connect-agents/#choose-the-tools-an-agent-sees)).
|
|
60
|
+
|
|
61
|
+
The engine posts batches of up to 100 events, in order, as `{"webhook": "…", "events": [ … ], "last": 42}`. Two headers come with each one:
|
|
62
|
+
|
|
63
|
+
* **`X-Datanotes-Signature`**: `sha256=` and the hex HMAC-SHA256 of the body, keyed with the secret; check it before you trust the body
|
|
64
|
+
* **`X-Datanotes-Delivery`**: the number of the batch’s last event
|
|
65
|
+
|
|
66
|
+
A 2xx answer moves the webhook on. Any other answer is retried with backoff, from 5s up to 10 minutes, so a receiver that was down gets every event when it comes back. `datanotes status` lists a failing webhook with its last error.
|
|
67
|
+
|
|
68
|
+
## Choose the feed or a webhook
|
|
69
|
+
|
|
70
|
+
A named token can read the feed, while only the owner creates webhooks. So a program with its own token long-polls or streams the feed, as [Build an app on Datanotes](https://datanotes.dev/develop/build-an-app/) does, and a service the owner sets up receives a webhook.
|
|
@@ -0,0 +1,116 @@
|
|
|
1
|
+
# Call the engine from JavaScript
|
|
2
|
+
|
|
3
|
+
> Use datanotes/client to run operations from Node, browsers and other applications.
|
|
4
|
+
|
|
5
|
+
The `datanotes` package exports `datanotes/client`, a client for the [HTTP API](https://datanotes.dev/develop/http-api/) with no dependencies. It runs in Node, in browsers and inside other applications that bring their own transport. The Model Context Protocol (MCP) server for agents uses it too.
|
|
6
|
+
|
|
7
|
+
## Install the client
|
|
8
|
+
|
|
9
|
+
The client ships in the `datanotes` package:
|
|
10
|
+
|
|
11
|
+
```bash
|
|
12
|
+
npm install datanotes
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
## Connect and run operations
|
|
16
|
+
|
|
17
|
+
Create one client per engine, with its address and a token. The token is the owner token or a [named token](https://datanotes.dev/guides/security/tokens/):
|
|
18
|
+
|
|
19
|
+
```ts
|
|
20
|
+
import { DatanotesClient } from "datanotes/client";
|
|
21
|
+
|
|
22
|
+
|
|
23
|
+
const db = new DatanotesClient({
|
|
24
|
+
url: "http://127.0.0.1:27150",
|
|
25
|
+
token: process.env.DATANOTES_TOKEN ?? "",
|
|
26
|
+
actor: { harness: "my-script" },
|
|
27
|
+
});
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
The engine records `actor` with every write the client makes, unless a call names another actor.
|
|
31
|
+
|
|
32
|
+
Run an [operation](https://datanotes.dev/reference/operations/) by name, with the same arguments agents use. Each call answers `{ ok, value }`:
|
|
33
|
+
|
|
34
|
+
```ts
|
|
35
|
+
const films = await db.call("select_rows", {
|
|
36
|
+
table: "film",
|
|
37
|
+
order_by: "-minutes",
|
|
38
|
+
});
|
|
39
|
+
if (films.ok) console.log((films.value as { rows: unknown[] }).rows);
|
|
40
|
+
|
|
41
|
+
|
|
42
|
+
const r = await db.call("insert_row", {
|
|
43
|
+
table: "film",
|
|
44
|
+
values: { title: "Metropolis", year: 1927, minutes: 153 },
|
|
45
|
+
});
|
|
46
|
+
if (!r.ok) console.log(r.value);
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
The first call lists films, longest first. The second adds a film. When the engine refuses a write, `ok` is `false`, `value` gives a readable reason, and the engine writes nothing. To write only when a row didn’t change since you read it, and to retry safely, see [Concurrent writes](https://datanotes.dev/develop/concurrency/).
|
|
50
|
+
|
|
51
|
+
For a complete program built on the client, see [Build an app on Datanotes](https://datanotes.dev/develop/build-an-app/).
|
|
52
|
+
|
|
53
|
+
## Call operations and routes
|
|
54
|
+
|
|
55
|
+
The client has three methods for the engine. Each one that takes `actor` uses it instead of the client’s default:
|
|
56
|
+
|
|
57
|
+
* **`call(name, args, actor?)`**: runs an engine operation and answers `{ ok, value }`; a name that isn’t an operation answers `ok: false` without a request
|
|
58
|
+
* **`post(route, body, actor?)`**: sends `POST /datanotes` followed by `route`, for example `post("/tokens/me/")`, and throws on an error status
|
|
59
|
+
* **`ping()`**: tells whether the engine answers
|
|
60
|
+
|
|
61
|
+
The package also exports the catalog of operations:
|
|
62
|
+
|
|
63
|
+
* **`OPERATIONS`**: each operation’s name, description and input schema
|
|
64
|
+
* **`operationRequest(name, args)`**: the HTTP request behind an operation
|
|
65
|
+
|
|
66
|
+
To offer the operations as tools of your own Model Context Protocol (MCP) server, use `datanotes/mcp`, as [Build on Datanotes](https://datanotes.dev/develop/build-on-datanotes/#build-your-own-mcp-server) shows.
|
|
67
|
+
|
|
68
|
+
## Handle errors
|
|
69
|
+
|
|
70
|
+
Failures that aren’t answers from an operation throw a `DatanotesError`. Its `kind` tells you what went wrong:
|
|
71
|
+
|
|
72
|
+
* **`unreachable`**: the engine didn’t answer
|
|
73
|
+
* **`auth`**: the engine refused the token
|
|
74
|
+
* **`http`**: `post` got an error status, given in `status`
|
|
75
|
+
|
|
76
|
+
Its `message` ends with a `Hint:` line, how to fix the call, and a `Docs:` line, the page that shows the fix; the error also has them as `hint` and `docs`. A failed `call` returns the same text as its `value`. To check an agent’s arguments before you send them, `checkArguments(name, args)` names the unknown, missing and wrongly typed ones. [Read the engine’s errors](https://datanotes.dev/guides/agents/reading-errors/) shows the shape of every error.
|
|
77
|
+
|
|
78
|
+
## Find the local engine
|
|
79
|
+
|
|
80
|
+
`datanotes/local` finds the engine that runs on a vault of this machine. It works in Node only:
|
|
81
|
+
|
|
82
|
+
```ts
|
|
83
|
+
import { localEngine } from "datanotes/local";
|
|
84
|
+
|
|
85
|
+
|
|
86
|
+
const engine = localEngine("your_vault_folder");
|
|
87
|
+
```
|
|
88
|
+
|
|
89
|
+
`localEngine` returns `{ url, token, lock }` while an engine runs on the vault, and `null` when none does. It throws when the engine that holds the vault runs on another machine.
|
|
90
|
+
|
|
91
|
+
It finds the engine as the engine finds its store: by the database’s id, in the data home. It honors the same variables, `DATANOTES_DATA_HOME` and `DATANOTES_STORE`, and takes them as options too:
|
|
92
|
+
|
|
93
|
+
```ts
|
|
94
|
+
const engine = localEngine("your_vault_folder", {
|
|
95
|
+
store: "your_store_folder",
|
|
96
|
+
});
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
`storeLocation("your_vault_folder")` returns the store’s folder, the database id and the database folder, without an engine running.
|
|
100
|
+
|
|
101
|
+
## Use your own transport
|
|
102
|
+
|
|
103
|
+
An application with its own way to make HTTP requests passes it as the transport. An Obsidian plugin, for example, passes `requestUrl`, so its requests aren’t subject to Cross-Origin Resource Sharing (CORS) rules:
|
|
104
|
+
|
|
105
|
+
```ts
|
|
106
|
+
const db = new DatanotesClient({
|
|
107
|
+
url,
|
|
108
|
+
token,
|
|
109
|
+
transport: async (req) => {
|
|
110
|
+
const r = await requestUrl({ ...req, throw: false });
|
|
111
|
+
return { status: r.status, text: r.text };
|
|
112
|
+
},
|
|
113
|
+
});
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
The transport receives each request and returns its status and body text.
|
|
@@ -0,0 +1,188 @@
|
|
|
1
|
+
# Concurrent writes
|
|
2
|
+
|
|
3
|
+
> What happens when several agents, programs and devices write to one engine at once, and how to write safely with expect, batch and retries.
|
|
4
|
+
|
|
5
|
+
Several clients often write to one database at once. Think of an agent, a script and the app on your phone. This page explains what the engine does with those writes, and how your program keeps its own writes correct.
|
|
6
|
+
|
|
7
|
+
## One engine per database
|
|
8
|
+
|
|
9
|
+
Run exactly one engine per database folder. While it runs, the engine holds a lock file, `engine.lock`, in its store. A second engine with the same store refuses to start.
|
|
10
|
+
|
|
11
|
+
The lock lives in the store, not in the database folder. So an engine with a store of its own isn’t refused, and two engines on one folder lose each other’s writes. Never run these on the same folder:
|
|
12
|
+
|
|
13
|
+
* the engine on the host and the engine in Docker
|
|
14
|
+
* engines on two devices that sync the vault
|
|
15
|
+
* two engines started with different `--store` folders
|
|
16
|
+
|
|
17
|
+
[Run the engine as a service](https://datanotes.dev/administration/run-the-service/#run-one-engine-per-database) shows the lock and what to do when it’s stale. [Sync the vault between devices](https://datanotes.dev/guides/data-quality/sync-between-devices/) shows how to run one engine on a synced vault.
|
|
18
|
+
|
|
19
|
+
Every client goes through that engine:
|
|
20
|
+
|
|
21
|
+
* **Programs**: call its [HTTP API](https://datanotes.dev/develop/http-api/), directly or with the [JavaScript client](https://datanotes.dev/develop/client/)
|
|
22
|
+
* **Agents**: call the Model Context Protocol (MCP) servers, which call the same HTTP API
|
|
23
|
+
* **People**: edit notes by hand, in Obsidian or another editor, which the engine records but doesn’t check first
|
|
24
|
+
|
|
25
|
+
## Writes run one at a time
|
|
26
|
+
|
|
27
|
+
The engine runs writes one at a time, in the order they arrive, under the lock of its transaction log. Reads run in parallel with each other.
|
|
28
|
+
|
|
29
|
+
A write that still runs after a minute lets the next write start, so one stuck write can’t hold up the rest. The engine logs a warning that names the operation.
|
|
30
|
+
|
|
31
|
+
## Reads wait for writes
|
|
32
|
+
|
|
33
|
+
The engine runs on one thread. Under a stream of reads, a write would share that thread with every read that arrives while it runs, and take several times longer. So writes go first:
|
|
34
|
+
|
|
35
|
+
* **A write runs or waits**: a read that arrives waits until no write is left.
|
|
36
|
+
* **A read already under way**: it finishes normally. The write doesn’t wait for it.
|
|
37
|
+
* **A read waits at most `writePriorityMaxWaitMs`**: 100 ms by default. It also starts once 8 writes started after it came. A burst of writes never starves the reads.
|
|
38
|
+
* **A long poll of the change feed**: `list_changes` with `wait_ms` never waits, and never holds a write.
|
|
39
|
+
|
|
40
|
+
What each operation does is unchanged: a read sees every write that ended before it started. Set `writePriority` to `false` in the [settings](https://datanotes.dev/reference/settings/) to let reads and writes start in the order they arrive. The span `engine.schedule.wait` shows the wait in [traces](https://datanotes.dev/develop/telemetry/#the-spans-the-engine-sends).
|
|
41
|
+
|
|
42
|
+
## Repeated reads come from memory
|
|
43
|
+
|
|
44
|
+
A read you send again with the same arguments, with the same token, often has the same answer. The engine keeps the answers of recent reads in memory, and sends the kept answer while nothing it read has changed:
|
|
45
|
+
|
|
46
|
+
* **A write**: a read of the tables it changed is computed again. Reads of other tables keep their answers.
|
|
47
|
+
* **Undo, a hand edit on disk, a sync tool**: they count as writes. The engine sees them before it answers.
|
|
48
|
+
* **A schema change**: every answer is computed again.
|
|
49
|
+
* **A view**: it depends on the tables its query reads.
|
|
50
|
+
|
|
51
|
+
Some reads depend on less than a table, or on more:
|
|
52
|
+
|
|
53
|
+
* **`read_note`**: it depends on its note. Deleting or renaming a folder also counts.
|
|
54
|
+
* **`list_folder`**: it depends on the notes and folders created, deleted or renamed in its folder. Deleting or renaming any folder also counts.
|
|
55
|
+
* **`list_tables`**: it depends on the schema only. A write to a table keeps its answer.
|
|
56
|
+
* **`describe_database`**: it depends on every table, and on the notes whose properties can’t be read.
|
|
57
|
+
* **A filter through a link column**: a key such as `project.status` follows the link column `project`. The read also depends on the tables the link leads to.
|
|
58
|
+
* **Reads that follow links or search the text**: `expand`, `within`, the graph operations and `search_notes` depend on the whole vault.
|
|
59
|
+
|
|
60
|
+
Reads that compare with the current time (`{now}`) are never kept. A read with `{today}` is kept 15 minutes at most. Reads of history, of the change feed and of `system.information_schema` are never kept either.
|
|
61
|
+
|
|
62
|
+
`readCacheMaxBytes` sets the memory for the answers: 33554432 bytes (32 MiB) by default. The least recently used answers go first. Set it to `0` to turn the cache off.
|
|
63
|
+
|
|
64
|
+
The engine checks each write against the database as the writes before it left it. A write is checked first, then applied. If another write is applied while the engine checks yours, the engine checks yours again on the new state. It tries up to 5 times before it gives up.
|
|
65
|
+
|
|
66
|
+
So the rules of your tables hold even when clients race:
|
|
67
|
+
|
|
68
|
+
* **Unique columns**: two clients insert the same value of a `unique` column at once. One write succeeds; the other is refused with a violation of the unique column.
|
|
69
|
+
* **CHECK constraints that follow links**: a client adds a task to a project while another client closes that project. If the CHECK refuses tasks of closed projects, the task is refused.
|
|
70
|
+
* **A database that keeps changing**: when the state changes on every try, the engine refuses the write with a conflict.
|
|
71
|
+
|
|
72
|
+
That last conflict reads “the database changed while this write was being checked: send it again”. The answer has status 422 and `ok: false`, and the engine writes nothing. Send the write again.
|
|
73
|
+
|
|
74
|
+
## Two writes to the same row
|
|
75
|
+
|
|
76
|
+
Each edit sets only the columns it names. So two writes to one row behave this way:
|
|
77
|
+
|
|
78
|
+
* **Different columns**: both writes apply, and the row keeps both changes.
|
|
79
|
+
* **The same column, without `expect`**: both writes apply, and the last one stays. The last writer wins.
|
|
80
|
+
* **The same column, with `expect`**: the second write is refused, because the row no longer has the value it expects.
|
|
81
|
+
|
|
82
|
+
## Write only if the row didn’t change
|
|
83
|
+
|
|
84
|
+
`expect` on `update_row` makes the write a compare-and-set. You pass the values you read, and the engine writes only if the row still has them. Otherwise it refuses the write with a conflict and writes nothing.
|
|
85
|
+
|
|
86
|
+
```ts
|
|
87
|
+
import { DatanotesClient } from "datanotes/client";
|
|
88
|
+
|
|
89
|
+
|
|
90
|
+
const db = new DatanotesClient({
|
|
91
|
+
url: "http://127.0.0.1:27150",
|
|
92
|
+
token: process.env.DATANOTES_TOKEN ?? "",
|
|
93
|
+
actor: { harness: "film-night" },
|
|
94
|
+
});
|
|
95
|
+
|
|
96
|
+
|
|
97
|
+
const r = await db.call("update_row", {
|
|
98
|
+
table: "film",
|
|
99
|
+
path: "main/public/film/Metropolis.md",
|
|
100
|
+
expect: { status: "to watch" },
|
|
101
|
+
set: { status: "watched" },
|
|
102
|
+
});
|
|
103
|
+
if (!r.ok) console.log(r.value);
|
|
104
|
+
```
|
|
105
|
+
|
|
106
|
+
If someone marked the film `watched` after your read, the engine refuses the write. `r.value` names the column, the value you expected and the value it found.
|
|
107
|
+
|
|
108
|
+
`update_rows` has a guard of its own. `expect_count` refuses the write unless exactly that many rows match `where`. A filter that matches more rows than you meant then changes nothing.
|
|
109
|
+
|
|
110
|
+
## Retry after you read again
|
|
111
|
+
|
|
112
|
+
A refused write is safe to retry only after you read the current state. The row changed for a reason: compute your write again from what it holds now.
|
|
113
|
+
|
|
114
|
+
This loop adds one to a film’s `times_seen`. It reads the row, writes with `expect`, and reads again when the write is refused:
|
|
115
|
+
|
|
116
|
+
```ts
|
|
117
|
+
const path = "main/public/film/Metropolis.md";
|
|
118
|
+
|
|
119
|
+
|
|
120
|
+
async function addViewing(attempts = 5): Promise<boolean> {
|
|
121
|
+
for (let i = 0; i < attempts; i++) {
|
|
122
|
+
const read = await db.call("select_rows", {
|
|
123
|
+
table: "film",
|
|
124
|
+
where: { path },
|
|
125
|
+
columns: ["times_seen"],
|
|
126
|
+
});
|
|
127
|
+
if (!read.ok) throw new Error(String(read.value));
|
|
128
|
+
const [film] = (read.value as {
|
|
129
|
+
rows: { times_seen: number }[];
|
|
130
|
+
}).rows;
|
|
131
|
+
|
|
132
|
+
|
|
133
|
+
const write = await db.call("update_row", {
|
|
134
|
+
table: "film",
|
|
135
|
+
path,
|
|
136
|
+
expect: { times_seen: film.times_seen },
|
|
137
|
+
set: { times_seen: film.times_seen + 1 },
|
|
138
|
+
});
|
|
139
|
+
if (write.ok) return true;
|
|
140
|
+
}
|
|
141
|
+
return false;
|
|
142
|
+
}
|
|
143
|
+
```
|
|
144
|
+
|
|
145
|
+
The loop stops after a few attempts. A write that keeps failing usually breaks a rule, and no retry fixes that. Show its reason to a person instead.
|
|
146
|
+
|
|
147
|
+
Decide on a refusal by its status, never by the text of its message:
|
|
148
|
+
|
|
149
|
+
* **422**: the engine refused the write, for validation or a conflict, and wrote nothing
|
|
150
|
+
* **409**: the engine refused an undo, or a table name is ambiguous
|
|
151
|
+
* **503**: a file failed mid-write, and the engine put back what it wrote (`rolled_back: true`). Nothing changed: send the write again later
|
|
152
|
+
* **500**: a file failed mid-write, and so did the rollback. `failed` lists what stayed: run `undo_migration` first
|
|
153
|
+
|
|
154
|
+
`call` answers `ok: false` for any of these. To read the status, use `post`: it throws a `DatanotesError` whose `status` holds it. Over plain HTTP, read the status of the response. [Call the engine over HTTP](https://datanotes.dev/develop/http-api/#read-errors) lists every status.
|
|
155
|
+
|
|
156
|
+
## Group writes that belong together
|
|
157
|
+
|
|
158
|
+
A `batch` is all or nothing while the engine keeps running. Its operations run in order, each one seeing the ones before it. If the engine refuses one, it undoes the ones already applied, and the answer has `rolled_back: true`.
|
|
159
|
+
|
|
160
|
+
A crash or `kill -9` in the middle of a batch is different. It may leave part of the batch applied:
|
|
161
|
+
|
|
162
|
+
* **Steps already done**: each keeps a journal entry of its own, which `undo_migration` reverts
|
|
163
|
+
* **The step under way**: its notes are written before its journal entry, so part of it may stay outside the journal
|
|
164
|
+
* **The batch entry**: the engine writes it last, so it’s missing
|
|
165
|
+
|
|
166
|
+
Run `check_table` after the next start to find the rows such a stop left. Stop the engine with `datanotes stop` instead, which lets the batch finish.
|
|
167
|
+
|
|
168
|
+
Use a batch when several writes must agree, such as a screening and the film it marks as watched. [Batch writes](https://datanotes.dev/guides/read-and-write/batch/) shows a batch with its answer.
|
|
169
|
+
|
|
170
|
+
A batch isn’t isolated from other clients, though. The engine applies its operations one by one, like separate writes. So other clients can act between two of them:
|
|
171
|
+
|
|
172
|
+
* **Reads**: a client may see the first operations of a refused batch before the engine undoes them.
|
|
173
|
+
* **Writes**: another client’s write can land between two operations of the batch. The engine then undoes the batch and runs it again on the new state.
|
|
174
|
+
* **Undo**: if another client changed a row the batch wrote, the undo may fail. The answer then has `rolled_back: false` and status 500, and `rollback` lists what stayed.
|
|
175
|
+
|
|
176
|
+
When `rolled_back` is `false`, undo the listed writes with `undo_migration`, or repair the rows, before you retry.
|
|
177
|
+
|
|
178
|
+
## Hand edits skip the checks
|
|
179
|
+
|
|
180
|
+
A person who edits a note in Obsidian writes straight to the file. The engine doesn’t validate that edit, because it can’t refuse it.
|
|
181
|
+
|
|
182
|
+
The engine records the edit instead, so you find what it broke:
|
|
183
|
+
|
|
184
|
+
* **Transaction log**: a table with history records the edit as `EXTERNAL_EDIT`
|
|
185
|
+
* **Change feed**: the edit’s event has `source: "external"`, and its last event lists the rules it breaks in `violations`
|
|
186
|
+
* **`check_table`**: lists every row of a table that breaks the table’s rules
|
|
187
|
+
|
|
188
|
+
A program that follows the [change feed](https://datanotes.dev/develop/change-feed-and-webhooks/) sees hand edits as they happen. [Hand edits](https://datanotes.dev/guides/data-quality/hand-edits/) shows how to find and repair the rows they break.
|
|
@@ -0,0 +1,89 @@
|
|
|
1
|
+
# Call the engine over HTTP
|
|
2
|
+
|
|
3
|
+
> Authenticate, call the engine’s routes, check its health and read its errors.
|
|
4
|
+
|
|
5
|
+
`datanotes serve` starts the engine’s HTTP server on `http://127.0.0.1:27150`. Every operation of the engine is a route, so any language that sends HTTP requests can read and write your notes. This page covers tokens, the routes, the engine’s health report and errors; the [HTTP API reference](https://datanotes.dev/api/) documents each route.
|
|
6
|
+
|
|
7
|
+
## Authenticate your requests
|
|
8
|
+
|
|
9
|
+
Every request needs an `Authorization: Bearer your_token` header, with one of two tokens:
|
|
10
|
+
|
|
11
|
+
* **Owner token**: `httpServerToken` in the settings file, which the engine generates when it first starts; it can do everything
|
|
12
|
+
* **Named token**: the engine creates it when you [create a token](https://datanotes.dev/guides/security/tokens/#create-a-token) for a program or an agent, with only the permissions you give it
|
|
13
|
+
|
|
14
|
+
Three routes answer without a token: `GET /`, `GET /datanotes/openapi.json` and the `/datanotes/docs` console. They hold no data from your notes.
|
|
15
|
+
|
|
16
|
+
To use another port, set `httpServerPort` in the settings, or start the engine with `--port`. To listen on another address, start the engine with `--host`. [Choose the port](https://datanotes.dev/administration/run-the-service/#choose-the-port) explains both.
|
|
17
|
+
|
|
18
|
+
## Call the engine’s routes
|
|
19
|
+
|
|
20
|
+
The engine serves its operations and a description of itself:
|
|
21
|
+
|
|
22
|
+
* **`POST /datanotes/operations/<operation>/`**: one route per operation, with the operation’s arguments as the JSON body, for example `POST /datanotes/operations/insert_row/` with `{"table": "film", "values": {…}}`
|
|
23
|
+
* **`GET /`**: whether the engine runs, and its version
|
|
24
|
+
* **`GET /datanotes/openapi.json`**: the OpenAPI 3.1 description of every route, generated from the catalog of operations, to generate clients in any language
|
|
25
|
+
* **`GET /datanotes/docs`**: the same description in an interactive console
|
|
26
|
+
|
|
27
|
+
The console is Swagger UI, which the engine serves itself, with nothing from other hosts. Click **Authorize**, paste the owner token or a named token, then click **Execute** on any route to call this engine. The browser keeps the token in its local storage until you log out in the **Authorize** dialog.
|
|
28
|
+
|
|
29
|
+
An operation’s body takes the same arguments as the agent’s tool of the same name. You read notes, rows or not, with three operations:
|
|
30
|
+
|
|
31
|
+
* **[`read_note`](https://datanotes.dev/reference/operations/queries/#read_note)**: one note, by its `path`
|
|
32
|
+
* **[`list_folder`](https://datanotes.dev/reference/operations/queries/#list_folder)**: the notes and folders in a `folder`
|
|
33
|
+
* **[`search_notes`](https://datanotes.dev/reference/operations/queries/#search_notes)**: the notes that contain a `query`
|
|
34
|
+
|
|
35
|
+
Routes and fields follow semantic versioning: see [Compatibility](https://datanotes.dev/reference/compatibility/).
|
|
36
|
+
|
|
37
|
+
## Record who makes each write
|
|
38
|
+
|
|
39
|
+
The `X-Datanotes-Actor` header tells the engine on whose behalf you write. Its value is JSON, with any of `session`, `harness`, `model` and `tool`:
|
|
40
|
+
|
|
41
|
+
```bash
|
|
42
|
+
curl -X POST http://127.0.0.1:27150/datanotes/operations/insert_row/ \
|
|
43
|
+
-H "Authorization: Bearer your_owner_token" \
|
|
44
|
+
-H "Content-Type: application/json" \
|
|
45
|
+
-H 'X-Datanotes-Actor: {"harness": "my-script"}' \
|
|
46
|
+
-d '{"table": "film", "values": {"title": "His Girl Friday", "year": 1940}}'
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
The engine stores the actor with the write in the journal, and in the table’s history when the table keeps one. It adds the credential as `auth`: `owner`, or the named token’s name, which the header can’t claim.
|
|
50
|
+
|
|
51
|
+
## Check the engine’s health
|
|
52
|
+
|
|
53
|
+
`POST /datanotes/status/` reports the engine’s health, and `datanotes status` prints the same report from a shell. [Check the engine’s status](https://datanotes.dev/administration/status/) shows both.
|
|
54
|
+
|
|
55
|
+
## Read errors
|
|
56
|
+
|
|
57
|
+
Errors come back as JSON `{"error": "…", "hint": "…", "docs": "…"}`: what is wrong, how to fix it, and the page of this site, in Markdown, that shows the fix ([read the engine’s errors](https://datanotes.dev/guides/agents/reading-errors/)). The status is one of these:
|
|
58
|
+
|
|
59
|
+
* **400**: bad input
|
|
60
|
+
* **401**: missing or wrong token
|
|
61
|
+
* **403**: a named token beyond its permissions
|
|
62
|
+
* **404**: unknown table, view, migration or note
|
|
63
|
+
* **409**: ambiguous table name
|
|
64
|
+
* **500**: an unexpected error, or a write that failed partway and couldn’t be [rolled back](https://datanotes.dev/guides/history/journal-and-undo/#the-engine-rolls-back-a-write-it-cannot-finish)
|
|
65
|
+
* **503**: a write that failed partway, which the engine rolled back, so nothing changed: send it again later
|
|
66
|
+
|
|
67
|
+
The engine answers a refused write, from validation or a conflict, with 422 (409 for an undo). The body is the operation’s result, with `ok: false`, the reasons, `hint` and `docs`, and the engine writes nothing. An unexpected error (500) has no hint: it isn’t a mistake in the request.
|
|
68
|
+
|
|
69
|
+
Several clients can write at the same time. [Concurrent writes](https://datanotes.dev/develop/concurrency/) explains what the engine does then, and when to retry a refused write.
|
|
70
|
+
|
|
71
|
+
## Keep the engine local
|
|
72
|
+
|
|
73
|
+
The engine binds to `127.0.0.1` unless you start it with `--host`. It answers no Cross-Origin Resource Sharing (CORS) preflight, so web pages can’t call it. To reach it from another device, follow the [security model](https://datanotes.dev/guides/security/security-model/).
|
|
74
|
+
|
|
75
|
+
## Get large answers compressed
|
|
76
|
+
|
|
77
|
+
A client on another machine gets answers of 16 KB or more compressed with gzip. It asks for that with the `Accept-Encoding: gzip` header, as most HTTP clients do. A client on the engine’s machine gets them as they are: compressing would cost the engine more time than it saves.
|
|
78
|
+
|
|
79
|
+
The `httpCompression` [setting](https://datanotes.dev/reference/settings/) changes this:
|
|
80
|
+
|
|
81
|
+
* **`remote`**: clients on other machines only (the default).
|
|
82
|
+
* **`always`**: this machine’s clients too.
|
|
83
|
+
* **`off`**: no compression.
|
|
84
|
+
|
|
85
|
+
## Generate a client or explore the routes
|
|
86
|
+
|
|
87
|
+
The [HTTP API reference](https://datanotes.dev/api/) documents every route, generated from the same OpenAPI description, [openapi.json](https://datanotes.dev/openapi.json). From JavaScript, use the [JavaScript client](https://datanotes.dev/develop/client/). Agents use the [Model Context Protocol (MCP) server](https://datanotes.dev/start/connect-agents/).
|
|
88
|
+
|
|
89
|
+
To see the routes at work in one program, follow [Build an app on Datanotes](https://datanotes.dev/develop/build-an-app/).
|