datanotes 2.0.0 → 3.0.0

package/dist/docs/develop/change-feed-and-webhooks.md

@@ -0,0 +1,70 @@
+# Follow the change feed and webhooks
+
+> React to every change in the vault from a program: poll or long-poll the change feed, stream it, or have the engine post it to a webhook.
+
+The change feed numbers every change in the vault: writes through the engine, hand edits with the rules they break, schema changes and, when asked for, the tool calls of agents. A program follows it to react when a note changes, without reading the tables again. [Change data feed](https://datanotes.dev/guides/history/change-data-feed/) shows its events on a real engine.
+
+## Choose how to follow the feed
+
+A program learns about changes in three ways:
+
+* **Polling**: `POST /datanotes/operations/list_changes/` returns the changes since a point in the feed
+* **Streaming**: `GET /datanotes/changes/stream` sends each change as a Server-Sent Events (SSE) message
+* **Webhooks**: the owner creates them, and the engine posts change events to their URL
+
+Each event has its number, `seq`, the table, its `kind` (`insert`, `update`, `delete`, `move` or `metastore`), the operation that made it in `op`, the note’s path, the changed columns with their values before and after, and the actor. A hand edit has `source: "external"`, and its last event in a batch lists the rules it breaks in `violations`. When you rename a note by hand, the engine repairs the links to it. The events of that repair come after the move and name it in `cause`. For a row, `cause` also gives the `seq` of its `move` event.
+
+## Long-poll the feed from code
+
+`list_changes` returns the changes after `since`, oldest first. Pass the answer’s `last` as `since` on the next call, and `wait_ms` (at most 60,000) to wait for the next change instead of polling. `head` is the feed’s newest number, so a program that starts from `head` sees only what happens after it starts:
+
+```ts
+type Feed = {
+  head: number;
+  last: number;
+  events: { kind: string; path: string }[];
+};
+
+
+const first = await db.call("list_changes", { limit: 1 });
+if (!first.ok) throw new Error(String(first.value));
+let since = (first.value as Feed).head;
+for (;;) {
+  const r = await db.call("list_changes", {
+    since,
+    wait_ms: 20_000,
+    tables: ["film"],
+  });
+  if (!r.ok) throw new Error(String(r.value));
+  const feed = r.value as Feed;
+  for (const event of feed.events) console.log(event.kind, event.path);
+  since = feed.last;
+}
+```
+
+The filters narrow the feed before it reaches you:
+
+* **`tables`**, **`kinds`** and **`columns`**: the tables, kinds of change and columns to keep
+* **`where`**: rows that match a filter, as in `select_rows`, on the values after the change
+* **`exclude_actor`**: leaves out the changes of one harness, such as the program’s own writes when it sends its name as `harness`
+
+An answer with `gap: true` means the engine dropped older changes that you hadn’t read: read the tables again. To pick up where the program stopped after a restart, save `last` and start from it.
+
+## Stream the feed
+
+`GET /datanotes/changes/stream` sends each change as a Server-Sent Events (SSE) message, with its number as the event’s `id`. It takes the same filters as query parameters, and resumes from `Last-Event-ID` or `since`. A comment line every 25s keeps proxies from closing the connection, and an `event: gap` message tells the client that events were dropped.
+
+## Receive changes with a webhook
+
+A webhook has the engine post the feed to a web address, for a service that would rather be called than poll. Only the owner token creates webhooks, with [`create_webhook`](https://datanotes.dev/reference/operations/history/#create_webhook), and the answer shows the webhook’s `secret` once: give it to the receiver. An agent connected with `datanotes mcp` gets `create_webhook` only with `--owner-tools` or `--profile all` ([Choose the tools an agent sees](https://datanotes.dev/start/connect-agents/#choose-the-tools-an-agent-sees)).
+
+The engine posts batches of up to 100 events, in order, as `{"webhook": "…", "events": [ … ], "last": 42}`. Two headers come with each one:
+
+* **`X-Datanotes-Signature`**: `sha256=` and the hex HMAC-SHA256 of the body, keyed with the secret; check it before you trust the body
+* **`X-Datanotes-Delivery`**: the number of the batch’s last event
+
+A 2xx answer moves the webhook on. Any other answer is retried with backoff, from 5s up to 10 minutes, so a receiver that was down gets every event when it comes back. `datanotes status` lists a failing webhook with its last error.
+
+## Choose the feed or a webhook
+
+A named token can read the feed, while only the owner creates webhooks. So a program with its own token long-polls or streams the feed, as [Build an app on Datanotes](https://datanotes.dev/develop/build-an-app/) does, and a service the owner sets up receives a webhook.

package/dist/docs/develop/client.md

@@ -0,0 +1,116 @@
+# Call the engine from JavaScript
+
+> Use datanotes/client to run operations from Node, browsers and other applications.
+
+The `datanotes` package exports `datanotes/client`, a client for the [HTTP API](https://datanotes.dev/develop/http-api/) with no dependencies. It runs in Node, in browsers and inside other applications that bring their own transport. The Model Context Protocol (MCP) server for agents uses it too.
+
+## Install the client
+
+The client ships in the `datanotes` package:
+
+```bash
+npm install datanotes
+```
+
+## Connect and run operations
+
+Create one client per engine, with its address and a token. The token is the owner token or a [named token](https://datanotes.dev/guides/security/tokens/):
+
+```ts
+import { DatanotesClient } from "datanotes/client";
+
+
+const db = new DatanotesClient({
+  url: "http://127.0.0.1:27150",
+  token: process.env.DATANOTES_TOKEN ?? "",
+  actor: { harness: "my-script" },
+});
+```
+
+The engine records `actor` with every write the client makes, unless a call names another actor.
+
+Run an [operation](https://datanotes.dev/reference/operations/) by name, with the same arguments agents use. Each call answers `{ ok, value }`:
+
+```ts
+const films = await db.call("select_rows", {
+  table: "film",
+  order_by: "-minutes",
+});
+if (films.ok) console.log((films.value as { rows: unknown[] }).rows);
+
+
+const r = await db.call("insert_row", {
+  table: "film",
+  values: { title: "Metropolis", year: 1927, minutes: 153 },
+});
+if (!r.ok) console.log(r.value);
+```
+
+The first call lists films, longest first. The second adds a film. When the engine refuses a write, `ok` is `false`, `value` gives a readable reason, and the engine writes nothing. To write only when a row didn’t change since you read it, and to retry safely, see [Concurrent writes](https://datanotes.dev/develop/concurrency/).
+
+For a complete program built on the client, see [Build an app on Datanotes](https://datanotes.dev/develop/build-an-app/).
+
+## Call operations and routes
+
+The client has three methods for the engine. Each one that takes `actor` uses it instead of the client’s default:
+
+* **`call(name, args, actor?)`**: runs an engine operation and answers `{ ok, value }`; a name that isn’t an operation answers `ok: false` without a request
+* **`post(route, body, actor?)`**: sends `POST /datanotes` followed by `route`, for example `post("/tokens/me/")`, and throws on an error status
+* **`ping()`**: tells whether the engine answers
+
+The package also exports the catalog of operations:
+
+* **`OPERATIONS`**: each operation’s name, description and input schema
+* **`operationRequest(name, args)`**: the HTTP request behind an operation
+
+To offer the operations as tools of your own Model Context Protocol (MCP) server, use `datanotes/mcp`, as [Build on Datanotes](https://datanotes.dev/develop/build-on-datanotes/#build-your-own-mcp-server) shows.
+
+## Handle errors
+
+Failures that aren’t answers from an operation throw a `DatanotesError`. Its `kind` tells you what went wrong:
+
+* **`unreachable`**: the engine didn’t answer
+* **`auth`**: the engine refused the token
+* **`http`**: `post` got an error status, given in `status`
+
+Its `message` ends with a `Hint:` line, how to fix the call, and a `Docs:` line, the page that shows the fix; the error also has them as `hint` and `docs`. A failed `call` returns the same text as its `value`. To check an agent’s arguments before you send them, `checkArguments(name, args)` names the unknown, missing and wrongly typed ones. [Read the engine’s errors](https://datanotes.dev/guides/agents/reading-errors/) shows the shape of every error.
+
+## Find the local engine
+
+`datanotes/local` finds the engine that runs on a vault of this machine. It works in Node only:
+
+```ts
+import { localEngine } from "datanotes/local";
+
+
+const engine = localEngine("your_vault_folder");
+```
+
+`localEngine` returns `{ url, token, lock }` while an engine runs on the vault, and `null` when none does. It throws when the engine that holds the vault runs on another machine.
+
+It finds the engine as the engine finds its store: by the database’s id, in the data home. It honors the same variables, `DATANOTES_DATA_HOME` and `DATANOTES_STORE`, and takes them as options too:
+
+```ts
+const engine = localEngine("your_vault_folder", {
+  store: "your_store_folder",
+});
+```
+
+`storeLocation("your_vault_folder")` returns the store’s folder, the database id and the database folder, without an engine running.
+
+## Use your own transport
+
+An application with its own way to make HTTP requests passes it as the transport. An Obsidian plugin, for example, passes `requestUrl`, so its requests aren’t subject to Cross-Origin Resource Sharing (CORS) rules:
+
+```ts
+const db = new DatanotesClient({
+  url,
+  token,
+  transport: async (req) => {
+    const r = await requestUrl({ ...req, throw: false });
+    return { status: r.status, text: r.text };
+  },
+});
+```
+
+The transport receives each request and returns its status and body text.

package/dist/docs/develop/concurrency.md

@@ -0,0 +1,153 @@
+# Concurrent writes
+
+> What happens when several agents, programs and devices write to one engine at once, and how to write safely with expect, batch and retries.
+
+Several clients often write to one database at once. Think of an agent, a script and the app on your phone. This page explains what the engine does with those writes, and how your program keeps its own writes correct.
+
+## One engine per database
+
+Run exactly one engine per database folder. While it runs, the engine holds a lock file, `engine.lock`, in its store. A second engine with the same store refuses to start.
+
+The lock lives in the store, not in the database folder. So an engine with a store of its own isn’t refused, and two engines on one folder lose each other’s writes. Never run these on the same folder:
+
+* the engine on the host and the engine in Docker
+* engines on two devices that sync the vault
+* two engines started with different `--store` folders
+
+[Run the engine as a service](https://datanotes.dev/administration/run-the-service/#run-one-engine-per-database) shows the lock and what to do when it’s stale. [Sync the vault between devices](https://datanotes.dev/guides/data-quality/sync-between-devices/) shows how to run one engine on a synced vault.
+
+Every client goes through that engine:
+
+* **Programs**: call its [HTTP API](https://datanotes.dev/develop/http-api/), directly or with the [JavaScript client](https://datanotes.dev/develop/client/)
+* **Agents**: call the Model Context Protocol (MCP) servers, which call the same HTTP API
+* **People**: edit notes by hand, in Obsidian or another editor, which the engine records but doesn’t check first
+
+## Writes run one at a time
+
+The engine applies writes one at a time, under the lock of its transaction log. Reads don’t wait for that lock: they run in parallel, with each other and with writes.
+
+The engine checks each write against the database as the writes before it left it. A write is checked first, then applied. If another write is applied while the engine checks yours, the engine checks yours again on the new state. It tries up to 5 times before it gives up.
+
+So the rules of your tables hold even when clients race:
+
+* **Unique columns**: two clients insert the same value of a `unique` column at once. One write succeeds; the other is refused with a violation of the unique column.
+* **CHECK constraints that follow links**: a client adds a task to a project while another client closes that project. If the CHECK refuses tasks of closed projects, the task is refused.
+* **A database that keeps changing**: when the state changes on every try, the engine refuses the write with a conflict.
+
+That last conflict reads “the database changed while this write was being checked: send it again”. The answer has status 422 and `ok: false`, and the engine writes nothing. Send the write again.
+
+## Two writes to the same row
+
+Each edit sets only the columns it names. So two writes to one row behave this way:
+
+* **Different columns**: both writes apply, and the row keeps both changes.
+* **The same column, without `expect`**: both writes apply, and the last one stays. The last writer wins.
+* **The same column, with `expect`**: the second write is refused, because the row no longer has the value it expects.
+
+## Write only if the row didn’t change
+
+`expect` on `update_row` makes the write a compare-and-set. You pass the values you read, and the engine writes only if the row still has them. Otherwise it refuses the write with a conflict and writes nothing.
+
+```ts
+import { DatanotesClient } from "datanotes/client";
+
+
+const db = new DatanotesClient({
+  url: "http://127.0.0.1:27150",
+  token: process.env.DATANOTES_TOKEN ?? "",
+  actor: { harness: "film-night" },
+});
+
+
+const r = await db.call("update_row", {
+  table: "film",
+  path: "main/public/film/Metropolis.md",
+  expect: { status: "to watch" },
+  set: { status: "watched" },
+});
+if (!r.ok) console.log(r.value);
+```
+
+If someone marked the film `watched` after your read, the engine refuses the write. `r.value` names the column, the value you expected and the value it found.
+
+`update_rows` has a guard of its own. `expect_count` refuses the write unless exactly that many rows match `where`. A filter that matches more rows than you meant then changes nothing.
+
+## Retry after you read again
+
+A refused write is safe to retry only after you read the current state. The row changed for a reason: compute your write again from what it holds now.
+
+This loop adds one to a film’s `times_seen`. It reads the row, writes with `expect`, and reads again when the write is refused:
+
+```ts
+const path = "main/public/film/Metropolis.md";
+
+
+async function addViewing(attempts = 5): Promise<boolean> {
+  for (let i = 0; i < attempts; i++) {
+    const read = await db.call("select_rows", {
+      table: "film",
+      where: { path },
+      columns: ["times_seen"],
+    });
+    if (!read.ok) throw new Error(String(read.value));
+    const [film] = (read.value as {
+      rows: { times_seen: number }[];
+    }).rows;
+
+
+    const write = await db.call("update_row", {
+      table: "film",
+      path,
+      expect: { times_seen: film.times_seen },
+      set: { times_seen: film.times_seen + 1 },
+    });
+    if (write.ok) return true;
+  }
+  return false;
+}
+```
+
+The loop stops after a few attempts. A write that keeps failing usually breaks a rule, and no retry fixes that. Show its reason to a person instead.
+
+Decide on a refusal by its status, never by the text of its message:
+
+* **422**: the engine refused the write, for validation or a conflict, and wrote nothing
+* **409**: the engine refused an undo, or a table name is ambiguous
+* **503**: a file failed mid-write, and the engine put back what it wrote (`rolled_back: true`). Nothing changed: send the write again later
+* **500**: a file failed mid-write, and so did the rollback. `failed` lists what stayed: run `undo_migration` first
+
+`call` answers `ok: false` for any of these. To read the status, use `post`: it throws a `DatanotesError` whose `status` holds it. Over plain HTTP, read the status of the response. [Call the engine over HTTP](https://datanotes.dev/develop/http-api/#read-errors) lists every status.
+
+## Group writes that belong together
+
+A `batch` is all or nothing while the engine keeps running. Its operations run in order, each one seeing the ones before it. If the engine refuses one, it undoes the ones already applied, and the answer has `rolled_back: true`.
+
+A crash or `kill -9` in the middle of a batch is different. It may leave part of the batch applied:
+
+* **Steps already done**: each keeps a journal entry of its own, which `undo_migration` reverts
+* **The step under way**: its notes are written before its journal entry, so part of it may stay outside the journal
+* **The batch entry**: the engine writes it last, so it’s missing
+
+Run `check_table` after the next start to find the rows such a stop left. Stop the engine with `datanotes stop` instead, which lets the batch finish.
+
+Use a batch when several writes must agree, such as a screening and the film it marks as watched. [Batch writes](https://datanotes.dev/guides/read-and-write/batch/) shows a batch with its answer.
+
+A batch isn’t isolated from other clients, though. The engine applies its operations one by one, like separate writes. So other clients can act between two of them:
+
+* **Reads**: a client may see the first operations of a refused batch before the engine undoes them.
+* **Writes**: another client’s write can land between two operations of the batch. The engine then undoes the batch and runs it again on the new state.
+* **Undo**: if another client changed a row the batch wrote, the undo may fail. The answer then has `rolled_back: false` and status 500, and `rollback` lists what stayed.
+
+When `rolled_back` is `false`, undo the listed writes with `undo_migration`, or repair the rows, before you retry.
+
+## Hand edits skip the checks
+
+A person who edits a note in Obsidian writes straight to the file. The engine doesn’t validate that edit, because it can’t refuse it.
+
+The engine records the edit instead, so you find what it broke:
+
+* **Transaction log**: a table with history records the edit as `EXTERNAL_EDIT`
+* **Change feed**: the edit’s event has `source: "external"`, and its last event lists the rules it breaks in `violations`
+* **`check_table`**: lists every row of a table that breaks the table’s rules
+
+A program that follows the [change feed](https://datanotes.dev/develop/change-feed-and-webhooks/) sees hand edits as they happen. [Hand edits](https://datanotes.dev/guides/data-quality/hand-edits/) shows how to find and repair the rows they break.

package/dist/docs/develop/http-api.md

@@ -0,0 +1,79 @@
+# Call the engine over HTTP
+
+> Authenticate, call the engine’s routes, check its health and read its errors.
+
+`datanotes serve` starts the engine’s HTTP server on `http://127.0.0.1:27150`. Every operation of the engine is a route, so any language that sends HTTP requests can read and write your notes. This page covers tokens, the routes, the engine’s health report and errors; the [HTTP API reference](https://datanotes.dev/api/) documents each route.
+
+## Authenticate your requests
+
+Every request needs an `Authorization: Bearer your_token` header, with one of two tokens:
+
+* **Owner token**: `httpServerToken` in the settings file, which the engine generates when it first starts; it can do everything
+* **Named token**: the engine creates it when you [create a token](https://datanotes.dev/guides/security/tokens/#create-a-token) for a program or an agent, with only the permissions you give it
+
+Three routes answer without a token: `GET /`, `GET /datanotes/openapi.json` and the `/datanotes/docs` console. They hold no data from your notes.
+
+To use another port, set `httpServerPort` in the settings, or start the engine with `--port`. To listen on another address, start the engine with `--host`. [Choose the port](https://datanotes.dev/administration/run-the-service/#choose-the-port) explains both.
+
+## Call the engine’s routes
+
+The engine serves its operations and a description of itself:
+
+* **`POST /datanotes/operations/<operation>/`**: one route per operation, with the operation’s arguments as the JSON body, for example `POST /datanotes/operations/insert_row/` with `{"table": "film", "values": {…}}`
+* **`GET /`**: whether the engine runs, and its version
+* **`GET /datanotes/openapi.json`**: the OpenAPI 3.1 description of every route, generated from the catalog of operations, to generate clients in any language
+* **`GET /datanotes/docs`**: the same description in an interactive console
+
+The console is Swagger UI, which the engine serves itself, with nothing from other hosts. Click **Authorize**, paste the owner token or a named token, then click **Execute** on any route to call this engine. The browser keeps the token in its local storage until you log out in the **Authorize** dialog.
+
+An operation’s body takes the same arguments as the agent’s tool of the same name. You read notes, rows or not, with three operations:
+
+* **[`read_note`](https://datanotes.dev/reference/operations/queries/#read_note)**: one note, by its `path`
+* **[`list_folder`](https://datanotes.dev/reference/operations/queries/#list_folder)**: the notes and folders in a `folder`
+* **[`search_notes`](https://datanotes.dev/reference/operations/queries/#search_notes)**: the notes that contain a `query`
+
+Every route and field keeps working in every 3.x version: see [Compatibility](https://datanotes.dev/reference/compatibility/).
+
+## Record who makes each write
+
+The `X-Datanotes-Actor` header tells the engine on whose behalf you write. Its value is JSON, with any of `session`, `harness`, `model` and `tool`:
+
+```bash
+curl -X POST http://127.0.0.1:27150/datanotes/operations/insert_row/ \
+  -H "Authorization: Bearer your_owner_token" \
+  -H "Content-Type: application/json" \
+  -H 'X-Datanotes-Actor: {"harness": "my-script"}' \
+  -d '{"table": "film", "values": {"title": "His Girl Friday", "year": 1940}}'
+```
+
+The engine stores the actor with the write in the journal, and in the table’s history when the table keeps one. It adds the credential as `auth`: `owner`, or the named token’s name, which the header can’t claim.
+
+## Check the engine’s health
+
+`POST /datanotes/status/` reports the engine’s health, and `datanotes status` prints the same report from a shell. [Check the engine’s status](https://datanotes.dev/administration/status/) shows both.
+
+## Read errors
+
+Errors come back as JSON `{"error": "…", "hint": "…", "docs": "…"}`: what is wrong, how to fix it, and the page of this site, in Markdown, that shows the fix ([read the engine’s errors](https://datanotes.dev/guides/agents/reading-errors/)). The status is one of these:
+
+* **400**: bad input
+* **401**: missing or wrong token
+* **403**: a named token beyond its permissions
+* **404**: unknown table, view, migration or note
+* **409**: ambiguous table name
+* **500**: an unexpected error, or a write that failed partway and couldn’t be [rolled back](https://datanotes.dev/guides/history/journal-and-undo/#the-engine-rolls-back-a-write-it-cannot-finish)
+* **503**: a write that failed partway, which the engine rolled back, so nothing changed: send it again later
+
+The engine answers a refused write, from validation or a conflict, with 422 (409 for an undo). The body is the operation’s result, with `ok: false`, the reasons, `hint` and `docs`, and the engine writes nothing. An unexpected error (500) has no hint: it isn’t a mistake in the request.
+
+Several clients can write at the same time. [Concurrent writes](https://datanotes.dev/develop/concurrency/) explains what the engine does then, and when to retry a refused write.
+
+## Keep the engine local
+
+The engine binds to `127.0.0.1` unless you start it with `--host`. It answers no Cross-Origin Resource Sharing (CORS) preflight, so web pages can’t call it. To reach it from another device, follow the [security model](https://datanotes.dev/guides/security/security-model/).
+
+## Generate a client or explore the routes
+
+The [HTTP API reference](https://datanotes.dev/api/) documents every route, generated from the same OpenAPI description, [openapi.json](https://datanotes.dev/openapi.json). From JavaScript, use the [JavaScript client](https://datanotes.dev/develop/client/). Agents use the [Model Context Protocol (MCP) server](https://datanotes.dev/start/connect-agents/).
+
+To see the routes at work in one program, follow [Build an app on Datanotes](https://datanotes.dev/develop/build-an-app/).

package/dist/docs/develop/telemetry.md

@@ -0,0 +1,86 @@
+# Send logs to your observability backend
+
+> Send the logs of the engine, the MCP server and the apps to any OpenTelemetry backend.
+
+Every part of Datanotes sends its logs with OpenTelemetry, the vendor-neutral standard, so any backend with an OpenTelemetry Protocol (OTLP) endpoint can receive them. This page shows how to point the engine, the Model Context Protocol (MCP) server and the apps at your backend. It also shows how to follow one agent call across all three. Datanotes has no vendor-specific code.
+
+## What Datanotes sends and where
+
+Datanotes sends logs only, over HTTP, to `your_endpoint/v1/logs`. It encodes them as `http/protobuf`, the OpenTelemetry default, or as `http/json`. Any OpenTelemetry Collector receives them, and so do these backends, among others:
+
+* SigNoz
+* Grafana: Alloy, Loki or Grafana Cloud
+* Elastic, Datadog, Honeycomb, New Relic and Dynatrace
+
+Datanotes sends no traces or metrics. For a backend that accepts only gRPC, put an OpenTelemetry Collector in front of it. When the collector doesn’t answer, Datanotes counts the records as lost and keeps working.
+
+## Configure the export with environment variables
+
+The engine, the MCP server and the apps all read the standard OpenTelemetry variables:
+
+| Variable                                              | Meaning                                                                                                |
+| ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------ |
+| `OTEL_EXPORTER_OTLP_ENDPOINT`                         | The collector’s base address, for example `http://127.0.0.1:4318`                                      |
+| `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT`                    | Or the full `…/v1/logs` address                                                                        |
+| `OTEL_EXPORTER_OTLP_HEADERS`                          | `key=value,key2=value2`, with URL-encoded values: the API key or `Authorization` your backend asks for |
+| `OTEL_EXPORTER_OTLP_PROTOCOL`                         | `http/protobuf` (default) or `http/json`                                                               |
+| `OTEL_SERVICE_NAME`                                   | The service name; defaults to `datanotes`, `datanotes-mcp` (for `datanotes mcp`) or the app’s name     |
+| `OTEL_RESOURCE_ATTRIBUTES`                            | Extra resource attributes, for example `deployment.environment=home`                                   |
+| `OTEL_SDK_DISABLED=true` or `OTEL_LOGS_EXPORTER=none` | Turns the export off                                                                                   |
+
+Datanotes also reads the `…_LOGS_…` variants of the headers and protocol variables. Without an endpoint, it sends nothing.
+
+## Configure the engine in its settings file
+
+Instead of the variables, you can configure the engine in its settings file, `settings.json` in its store (`datanotes store path` shows where). A variable you set overrides the matching setting:
+
+```json
+{
+  "telemetryEnabled": true,
+  "telemetryEndpoint": "http://127.0.0.1:4318",
+  "telemetryServiceName": "datanotes",
+  "telemetryProtocol": "http/protobuf",
+  "telemetryHeaders": {}
+}
+```
+
+`telemetryHeaders` holds what your backend asks for, such as an API key or `Authorization: Bearer …`. The engine keeps it in the settings file. To keep a key out of that file, set `OTEL_EXPORTER_OTLP_HEADERS` instead.
+
+## What the engine logs
+
+The engine logs these events:
+
+* Every HTTP request and operation, with the route, duration, outcome and actor
+* The tool calls MCP servers report, with the tool, duration, result size in characters and tokens, client, model and session
+* Refused requests
+* The semantic index and git sync
+* Errors
+
+Whatever the collector does, the engine also writes its records to files in `logs/`, in its store outside the vault. The `fileLogEnabled` setting turns these files on or off, and `logRetentionDays` sets how long the engine keeps them.
+
+## Configure the MCP server and the apps
+
+The MCP server and the apps read the same variables, and each has its own alternative:
+
+* **`datanotes mcp`**: the `--otlp-endpoint your_endpoint` option
+* **Apps**: an `otlp` block in the app’s configuration, `{"endpoint": "http://127.0.0.1:4318", "headers": {}, "serviceName": "…"}`
+
+## Follow one agent call across services
+
+The engine, the MCP server and the apps log the same `session` and `tool` for each agent call. `session` is the first 16 characters of the MCP session ID. Filter by `session` in your backend to see the call in the MCP server, in the app whose tool it reached, and the writes it made in the engine.
+
+## Send logs from your own program
+
+`datanotes/client` exports the exporter that Datanotes uses, so your own programs can log the same way:
+
+```ts
+import { OtlpLogExporter, otlpFromEnv } from "datanotes/client";
+
+
+const otlp = otlpFromEnv(process.env, { serviceName: "my-app" });
+const log = otlp ? new OtlpLogExporter(otlp) : null;
+log?.log("INFO", "imported films", { count: 8 });
+await log?.close();
+```
+
+`otlpFromEnv` reads the variables above and returns `null` when you set no endpoint or turn the export off; the program then logs nothing. Call `close()` before the program exits to send the remaining records.

package/dist/docs/guides/agents/docs-offline.md

@@ -0,0 +1,99 @@
+# Read the docs offline
+
+> The datanotes package ships these docs as Markdown for its version: agents read and search them with list_docs, read_docs and search_docs, programs through the engine, people from the package.
+
+The `datanotes` package ships this documentation as Markdown: one file per page of the site in `dist/docs/`, with `llms.txt`. The pages match the installed version, and reading them needs no internet access.
+
+## The agent reads the docs of its version
+
+`datanotes mcp` serves these pages to agents. The agent looks up an operation or an example without internet access, and the pages always match the version it talks to. Three tools read them:
+
+* **`list_docs`**: lists the pages, each with its path, title and description. `prefix` keeps one folder, such as `guides/`.
+* **`read_docs`**: returns a page as Markdown, or one section of it by the anchor of its heading, such as `guides/read-and-write/read-and-filter#compare-values-with-operators`. It returns at most 20,000 characters at a time, and a longer text ends with how to read the rest.
+* **`search_docs`**: finds the sections that match a few words, best first. Each hit has the path to pass to `read_docs` and a short excerpt.
+
+The `Docs:` line of an error names the `read_docs` call for its section, so the agent reads it offline at once:
+
+```text
+Docs: read_docs("guides/read-and-write/read-and-filter#compare-values-with-operators")
+```
+
+The server leaves out the web address, since its agents always have the docs tools. A program that uses `DatanotesClient` gets both: the address, then the `read_docs` call in parentheses. Your own MCP server with the docs tools drops the address with `offlineDocs` from `datanotes/mcp`.
+
+An agent that asks for a page or section that doesn’t exist gets the closest ones in the hint. Reading a section instead of a whole page keeps the agent’s context small: `search_docs` finds the section, and `read_docs` returns only that.
+
+## Search with required and excluded words
+
+`search_docs` ranks the sections by the words of the query. A word matches in any form, so `rows` finds `row`. It also matches its synonyms, so `revert` finds `undo`. Words in a heading count most.
+
+Three marks narrow the search:
+
+* **`"a phrase"`**: the section must hold these words in a row, in any form, such as `"time travel"`
+* **`+word`**: the section must hold the word, such as `+view`
+* **`-word`**: the section must not hold the word, such as `-sql`; `-"a phrase"` leaves out a phrase
+
+The other words only rank the sections. This query finds the sections on time travel that mention views and never SQL. Here, `db` is a [client](https://datanotes.dev/develop/client/) connected to the engine, as in [Build your own MCP server](https://datanotes.dev/develop/build-on-datanotes/#build-your-own-mcp-server):
+
+```ts
+import { callDocsTool } from "datanotes/mcp";
+
+
+const hits = await callDocsTool(
+  "search_docs",
+  { query: '"time travel" +view -sql' },
+  { client: db },
+);
+```
+
+The marks filter the results, and never rank them. A section with every required word still ranks by all the words of the query. A word that starts with two dashes, such as `--vault`, is a plain word.
+
+## Search by words and by meaning
+
+When the engine has [semantic search](https://datanotes.dev/guides/search/semantic/) on, `search_docs` becomes a hybrid search. It ranks the sections twice:
+
+* **By words**: the ranking above, made by the MCP server from its own copy of the docs
+* **By meaning**: the engine embeds the query and ranks the sections by the similarity of their vectors
+
+Reciprocal Rank Fusion merges the two rankings. Each section scores the sum of 1 / (60 + its rank) over the rankings that hold it. Only the first 50 sections of each ranking count. A section that both rankings place high comes first, ahead of one that a single ranking finds. The required and excluded words filter both rankings.
+
+Each hit then ends with the rankings that found it: `(keyword, semantic)`, `(keyword)` or `(semantic)`. The last line of the answer says the search was hybrid.
+
+This is the hybrid search of vector databases such as ChromaDB, with no service to add. The engine already has the embedding model of semantic search. It keeps the vectors of the docs in its store, in `semantic/docs.json`, apart from the index of your notes.
+
+The engine builds that index in the background:
+
+* **When it starts**: after it updates the index of your notes
+* **When the docs change**: after an upgrade or a new docs build, the engine embeds again only the sections whose text changed
+* **When the model changes**: another provider, address or model starts a new index
+
+The version of the docs is a hash of their sections, so any change to a section counts. The docs are public: embedding them sends nothing from your vault. With a paid provider, the docs cost about as much to index as 700 short notes.
+
+`search_docs` ranks by words alone, as without semantic search, in these cases:
+
+* **No semantic search**: it’s off, or the engine is older than hybrid search
+* **The index is being built**: the first search after a change starts it
+* **The engine runs another version**: its docs differ from those of the MCP server
+* **The engine doesn’t answer**: `search_docs` asks it again a minute later
+
+## Read the docs from programs and editors
+
+Programs and people read the same pages:
+
+* **Programs**: the engine serves them, with a token like its other routes. `GET /datanotes/manual/` lists them, and `GET /datanotes/manual/guides/read-and-write/read-and-filter.md` returns one page. Add `?section=compare-values-with-operators` for one section.
+* **Search by meaning**: `POST /datanotes/manual/search/` takes a `query` and returns the sections closest in meaning, with `ready: false` while the engine builds the index. Programs that use `datanotes/mcp` pass a `client` to `callDocsTool` to search like `datanotes mcp`.
+* **People and editors**: the files themselves, in `node_modules/datanotes/dist/docs/`.
+
+```bash
+curl -H "Authorization: Bearer your_token" \
+  http://127.0.0.1:27150/datanotes/manual/start/quick-start.md
+```
+
+A program ranks the sections by meaning with the same [client](https://datanotes.dev/develop/client/):
+
+```ts
+const ranked = await db.post("/manual/search/", {
+  query: "undo what an agent did",
+});
+```
+
+In a checkout of the repository, build the docs first with `cd docs && npm ci && npm run build`, which copies the pages to `dist/docs/`. Until then, the tools answer with that command.