datanotes 2.0.1 → 3.1.0

package/dist/docs/develop/telemetry.md

@@ -0,0 +1,285 @@
+# Send logs and traces to your observability backend
+
+> Send the logs and traces of the engine, the MCP server and the apps to any OpenTelemetry backend, and find slow operations.
+
+Every part of Datanotes sends logs and traces with OpenTelemetry, the vendor-neutral standard. Any backend with an OpenTelemetry Protocol (OTLP) endpoint can receive them. This page shows you how to:
+
+* Point the engine, the Model Context Protocol (MCP) server and the apps at your backend
+* Read the spans the engine sends, and what each one measures
+* Follow one agent call across the MCP server and the engine
+* Find the operations that are slow, and why
+
+Datanotes has no vendor-specific code.
+
+## What Datanotes sends and where
+
+Datanotes sends two signals over HTTP:
+
+* **Logs** to `your_endpoint/v1/logs`
+* **Traces** to `your_endpoint/v1/traces`
+
+It encodes them as `http/protobuf`, the OpenTelemetry default, or as `http/json`. Any OpenTelemetry Collector receives them, and so do these backends, among others:
+
+* SigNoz
+* Grafana: Alloy, Tempo, Loki or Grafana Cloud
+* Elastic, Datadog, Honeycomb, New Relic and Dynatrace
+
+Datanotes sends no metrics: your backend computes rates and latency percentiles from the spans. For a backend that accepts only gRPC, put an OpenTelemetry Collector in front of it. When the collector doesn’t answer, Datanotes counts the records and spans as lost and keeps working.
+
+## Configure the export with environment variables
+
+The engine, the MCP server and the apps all read the standard OpenTelemetry variables:
+
+| Variable                                                 | Meaning                                                                                                       |
+| -------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- |
+| `OTEL_EXPORTER_OTLP_ENDPOINT`                            | The collector’s base address, for example `http://127.0.0.1:4318`                                             |
+| `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT`                       | Or the full `…/v1/logs` address                                                                               |
+| `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`                     | Or the full `…/v1/traces` address                                                                             |
+| `OTEL_EXPORTER_OTLP_HEADERS`                             | `key=value,key2=value2`, with URL-encoded values: the API key or `Authorization` your backend asks for        |
+| `OTEL_EXPORTER_OTLP_PROTOCOL`                            | `http/protobuf` (default) or `http/json`                                                                      |
+| `OTEL_SERVICE_NAME`                                      | The service name; defaults to `datanotes`, `datanotes-mcp` (for `datanotes mcp`) or the app’s name            |
+| `OTEL_RESOURCE_ATTRIBUTES`                               | Extra resource attributes, for example `team=home`                                                            |
+| `OTEL_TRACES_SAMPLER` and `OTEL_TRACES_SAMPLER_ARG`      | `traceidratio` with a share from 0 to 1, `always_on` or `always_off`, plain or with the `parentbased_` prefix |
+| `OTEL_SDK_DISABLED=true`                                 | Turns all the export off                                                                                      |
+| `OTEL_LOGS_EXPORTER=none` or `OTEL_TRACES_EXPORTER=none` | Turns off the logs or the traces only                                                                         |
+
+Datanotes also reads the `…_LOGS_…` and `…_TRACES_…` variants of the headers and protocol variables. Without an endpoint, it sends nothing.
+
+## Configure the engine in its settings file
+
+Instead of the variables, you can configure the engine in its settings file, `settings.json` in its store. `datanotes store path` shows where. A variable you set overrides the matching setting:
+
+```json
+{
+  "telemetryEnabled": true,
+  "telemetryEndpoint": "http://127.0.0.1:4318",
+  "telemetryServiceName": "datanotes",
+  "telemetryEnvironment": "home",
+  "telemetryProtocol": "http/protobuf",
+  "telemetryHeaders": {},
+  "telemetryTraces": true,
+  "telemetryTraceSampleRatio": 1,
+  "telemetryMaxSpansPerTrace": 1000,
+  "telemetrySlowSpanMs": 0
+}
+```
+
+Each setting does one thing:
+
+* **`telemetryEnabled`** turns the export of logs and traces on.
+* **`telemetryEnvironment`** names the environment, such as `dev` or `home`. The engine sends it as `deployment.environment.name` and as the older `deployment.environment`, which some backends group by.
+* **`telemetryHeaders`** holds what your backend asks for, such as an API key. To keep a key out of the file, set `OTEL_EXPORTER_OTLP_HEADERS` instead.
+* **`telemetryTraces`** sends the traces too. It is on by default: turning telemetry on gives you both signals.
+* **`telemetryTraceSampleRatio`** keeps that share of new traces, from 0 to 1. The default, 1, keeps every trace. A trace that a caller started follows the caller’s choice.
+* **`telemetryMaxSpansPerTrace`** caps the spans of one trace. The engine drops the rest and counts them in `datanotes.spans_dropped` on the trace’s first span.
+* **`telemetrySlowSpanMs`** logs a request, an operation or a background job as a WARN when it lasts at least that many milliseconds. 0 turns it off.
+
+The engine reads the settings again while it runs, so a change applies within a few seconds. With traces off, tracing costs the engine one check per instrumented call.
+
+## What the engine logs
+
+The engine logs these events:
+
+* Every HTTP request and operation, with the route, duration, outcome and actor
+* The tool calls MCP servers report, with the tool, duration, result size and client
+* Refused requests
+* The semantic index and git sync
+* Errors
+
+A record made during a traced request names its trace. The OTLP record carries the trace and span IDs, so your backend links the log to the trace. The log files carry them as `trace_id` and `span_id`. While a request’s trace is sent, its INFO records become DEBUG records: the span already says what they say.
+
+Whatever the collector does, the engine also writes its records to files in `logs/`, in its store outside the vault. The `fileLogEnabled` setting turns these files on or off, and `logRetentionDays` sets how long the engine keeps them.
+
+## The spans the engine sends
+
+A trace is a tree of spans. Each span measures one piece of work and names its parent. A request to the engine gives this tree:
+
+```text
+POST /datanotes/operations/insert_row/      SERVER: the HTTP request
+└── engine.op insert_row                    the operation
+    ├── engine.schedule.wait                a read waits for writes (only then)
+    ├── engine.settle                       wait for the vault to settle
+    └── engine.insert_row                   the engine's work
+        ├── engine.autolink                 links found in the text
+        ├── engine.checks                   CHECKs and link rules
+        ├── engine.lock.acquire             wait for the write lock
+        └── engine.lock.hold                work under the lock
+            ├── engine.transaction_log      hand edits logged first
+            ├── engine.apply                files written
+            │   └── engine.journal.append   the undo entry
+            └── engine.transaction_log      the operation's commit
+```
+
+A read gives a similar tree under `engine.query.execute`:
+
+* **`engine.query.view`** runs the stored query of a view.
+* **`engine.query.plan`** resolves dates and `within` and compiles the filter.
+* **`engine.query.scan`** reads every row of the table. Its `datanotes.query.plan` is `full_scan`: Datanotes has no indexes, and a file cache spares the reads of unchanged notes.
+* **`engine.query.filter`**, **`engine.query.join`**, **`engine.query.group`**, **`engine.query.sort`** and **`engine.query.project`** follow, as the query needs them.
+
+Other spans cover the rest of the engine’s time:
+
+* **`engine.metastore.parse`** when `metastore.yaml` changed and the engine parses it again
+
+* **`engine.history.read`** for a read as of a version or a time
+
+* **`engine.graph.build`** when a graph operation rebuilds the link graph after a change. Calls that need it at the same time wait for that one build and count it in `datanotes.graph.build_waits`.
+
+* **`engine.change_feed.record`** for the change feed’s events of an operation
+
+* **`POST`** (CLIENT) for each webhook delivery, with `url.full`, `server.address` and `http.response.status_code`
+
+* **`job <name>`** for background work. The jobs are:
+
+  * `job hand_edits_log`
+  * `job adopt_note`
+  * `job schema_reload`
+  * `job semantic_index`
+  * `job store_maintenance`
+  * `job change_feed_prune`
+  * `job change_feed_hand_edits`
+  * `job transaction_log_warmup`
+  * `job auto_recompute`
+  * `job inverse_links`
+  * `job datanotes_start`, the engine’s start
+
+  Background work is always a trace of its own, even when a request saw the change behind it. After a start, `job transaction_log_warmup` reads each table’s transaction log once, one table at a time, so a write waits for one table at most.
+
+### Attributes of the operation span
+
+The `engine.op <operation>` span follows the conventions of database spans. Most backends list these spans as database calls:
+
+| Attribute                                            | Meaning                                                                                                                              |
+| ---------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
+| `db.system.name`, `db.system`                        | `datanotes`                                                                                                                          |
+| `db.operation.name`                                  | The operation, such as `select_rows`                                                                                                 |
+| `db.collection.name`                                 | The table or view the call names                                                                                                     |
+| `db.namespace`                                       | The catalog and schema, when the call gives a full name                                                                              |
+| `db.query.summary`                                   | The operation and its table, such as `select_rows task`                                                                              |
+| `db.response.returned_rows`                          | The rows the call returned                                                                                                           |
+| `datanotes.rows_scanned`                             | The rows the engine read to answer                                                                                                   |
+| `datanotes.notes_changed`, `datanotes.files_created` | What a write changed                                                                                                                 |
+| `datanotes.migration_id`                             | The write’s journal entry, for `undo_migration`                                                                                      |
+| `datanotes.dry_run`                                  | Whether the call was a dry run                                                                                                       |
+| `datanotes.long_poll`                                | `true` for a `list_changes` call that waits for changes                                                                              |
+| `datanotes.schedule.class`                           | `read`, `write` or `free`: how [write priority](https://datanotes.dev/develop/concurrency/#reads-wait-for-writes) treats the call                         |
+| `datanotes.schedule.wait_ms`                         | How long the call waited for writes before it started                                                                                |
+| `datanotes.schedule.released`                        | Why a waiting read started: `writes_done`, `max_wait` or `max_writes`                                                                |
+| `datanotes.read_cache`                               | For a read: `hit` (answered from memory), `miss` or `not_kept` ([read cache](https://datanotes.dev/develop/concurrency/#repeated-reads-come-from-memory)) |
+| `error.type`                                         | Why the call failed or was refused                                                                                                   |
+
+The SERVER span adds `http.request.method`, `http.route`, `http.response.status_code`, `client.address` and `user_agent.original`. It also names the caller: `datanotes.token.name` holds the token’s name, never the token. `datanotes.client.harness` and `datanotes.client.model` hold what the agent says of itself.
+
+### Calls to the vault
+
+Reading a table of thousands of notes means thousands of file reads. The engine doesn’t make a span of each one. It adds them up on the span that made them and on its operation span:
+
+* `datanotes.io.read_count`, `datanotes.io.read_chars` and `datanotes.io.read_ms`
+* The same for `write`, `create`, `list`, `exists`, `rename` and the other calls
+* `datanotes.cache.hits` and `datanotes.cache.misses` of the file cache
+* `datanotes.cache.text_reads` and `datanotes.cache.text_source`: note texts the file cache read again, or took from the [vault index](https://datanotes.dev/administration/limits/#start-faster-with-the-vault-index)
+* `datanotes.validate.rows` and `datanotes.validate.ms`, `datanotes.checks.rows` and `datanotes.checks.ms` for the checks run once per row
+
+`_chars` counts characters, not bytes: the length of the text as JavaScript counts it, in UTF-16 code units. In English Markdown, characters and bytes are about the same. Accents, other alphabets and emoji take more bytes than characters. For `list`, `_chars` counts the files and folders listed. The calls that move no text count 0.
+
+A single call that takes 20 ms or more gets a span of its own, `vault.<call>`, with its `file.path` and `datanotes.io.chars`.
+
+### Errors and refusals
+
+The engine tells failures and refusals apart:
+
+* **A failure** is a bug, a disk error, a write that failed partway or a 5xx answer. Its span has status ERROR and an `exception` event with the type, message and stack.
+* **A refusal** is a call the caller can fix: bad arguments, an unknown table, a broken CHECK. Its span keeps the status unset and sets `error.type`, such as `violation` or `TableNotFoundError`.
+
+An empty answer to a `list_changes` poll sends no trace and no log line: services poll every few seconds.
+
+## Follow one agent call across services
+
+One agent call is one trace across the MCP server and the engine:
+
+1. The MCP server makes a SERVER span, `tools/call <tool>`. It continues the client’s trace when the client sends `traceparent` in the call’s `_meta`.
+2. Each request it sends to the engine is a CLIENT span, child of the tool’s span. The request carries that span as its `traceparent` header.
+3. The engine’s SERVER span continues that trace, and its operation spans follow.
+
+The tool span carries `mcp.method.name`, `gen_ai.operation.name` (`execute_tool`), `gen_ai.tool.name` and `mcp.session.id`. Find the agent call by its tool or session, open the trace, and you see where the time went in both processes.
+
+The engine, the MCP server and the apps also log the same `session` and `tool` for each agent call. `session` is the first 16 characters of the MCP session ID.
+
+## Find the slow operations
+
+Your backend computes the latency of each span name from the traces. To find where the engine spends its time:
+
+1. List the `engine.op <operation>` spans by duration, or open your backend’s database calls view. Leave out the spans with `datanotes.long_poll`: they wait on purpose.
+2. Open a slow trace and compare its children. A long `engine.lock.acquire` means the call waited for another write. A long `engine.lock.hold` means the write itself was slow. A long `engine.schedule.wait` means a read waited for writes to end.
+3. In a read, compare `engine.query.scan` with the rest. A slow scan with many `datanotes.cache.misses` reads many notes from disk. A slow scan with few misses works on a large table: narrow it with `where`, or split the table.
+4. Look at the `datanotes.io.*_ms` attributes of the operation span. They show how much of its time went to the disk.
+5. Set `telemetrySlowSpanMs` to log the slow calls as WARN records, and alert on them.
+
+## Configure the MCP server and the apps
+
+The MCP server and the apps read the same variables, and each has its own alternative:
+
+* **`datanotes mcp`**: the `--otlp-endpoint your_endpoint` option, for its logs and traces
+* **Apps**: an `otlp` block in the app’s configuration, `{"endpoint": "http://127.0.0.1:4318", "headers": {}, "serviceName": "…"}`
+
+## Continue a trace from your own program
+
+Your program continues its trace in the engine when it sends the W3C `traceparent` header, and `tracestate` when it has one. `datanotes/client` does it for you with two options of `DatanotesClient`:
+
+* **`trace`** is the trace to continue: a `traceparent` value, a context, or a function that returns the current one.
+* **`tracer`** makes a CLIENT span for each request, child of `trace`, and sends that span as the engine’s parent.
+
+```ts
+import {
+  DatanotesClient,
+  OtlpTraceExporter,
+  Tracer,
+  otlpTracesFromEnv,
+} from "datanotes/client";
+
+
+const otlp = otlpTracesFromEnv(process.env, { serviceName: "my-app" });
+const exporter = otlp ? new OtlpTraceExporter(otlp) : null;
+const tracer = new Tracer({ export: (s) => exporter?.export(s) });
+
+
+const run = tracer.startSpan("import films", { kind: "server" });
+const client = new DatanotesClient({
+  url: "http://127.0.0.1:27150",
+  token: process.env.DATANOTES_TOKEN ?? "",
+  tracer,
+  trace: run.context,
+});
+await client.call("insert_row", {
+  table: "film",
+  values: { title: "Arrival" },
+});
+run.end();
+await exporter?.close();
+```
+
+Without a tracer, pass the incoming header as it is. A call can also name its own trace:
+
+```ts
+await client.call("select_rows", { table: "film" }, undefined, {
+  trace: request.headers.traceparent,
+});
+```
+
+Any HTTP client works too: send `traceparent: 00-<trace id>-<span id>-01` with the request.
+
+## Send logs from your own program
+
+`datanotes/client` exports the log exporter that Datanotes uses, so your own programs can log the same way:
+
+```ts
+import { OtlpLogExporter, otlpFromEnv } from "datanotes/client";
+
+
+const otlp = otlpFromEnv(process.env, { serviceName: "my-app" });
+const log = otlp ? new OtlpLogExporter(otlp) : null;
+log?.log("INFO", "imported films", { count: 8 });
+await log?.close();
+```
+
+`otlpFromEnv` reads the variables above and returns `null` when you set no endpoint or turn the export off. The program then logs nothing. Pass a span’s context as the fourth argument of `log` to link the record to that span. Call `close()` before the program exits to send the remaining records.

package/dist/docs/guides/agents/docs-offline.md

@@ -0,0 +1,99 @@
+# Read the docs offline
+
+> The datanotes package ships these docs as Markdown for its version: agents read and search them with list_docs, read_docs and search_docs, programs through the engine, people from the package.
+
+The `datanotes` package ships this documentation as Markdown: one file per page of the site in `dist/docs/`, with `llms.txt`. The pages match the installed version, and reading them needs no internet access.
+
+## The agent reads the docs of its version
+
+`datanotes mcp` serves these pages to agents. The agent looks up an operation or an example without internet access, and the pages always match the version it talks to. Three tools read them:
+
+* **`list_docs`**: lists the pages, each with its path, title and description. `prefix` keeps one folder, such as `guides/`.
+* **`read_docs`**: returns a page as Markdown, or one section of it by the anchor of its heading, such as `guides/read-and-write/read-and-filter#compare-values-with-operators`. It returns at most 20,000 characters at a time, and a longer text ends with how to read the rest.
+* **`search_docs`**: finds the sections that match a few words, best first. Each hit has the path to pass to `read_docs` and a short excerpt.
+
+The `Docs:` line of an error names the `read_docs` call for its section, so the agent reads it offline at once:
+
+```text
+Docs: read_docs("guides/read-and-write/read-and-filter#compare-values-with-operators")
+```
+
+The server leaves out the web address, since its agents always have the docs tools. A program that uses `DatanotesClient` gets both: the address, then the `read_docs` call in parentheses. Your own MCP server with the docs tools drops the address with `offlineDocs` from `datanotes/mcp`.
+
+An agent that asks for a page or section that doesn’t exist gets the closest ones in the hint. Reading a section instead of a whole page keeps the agent’s context small: `search_docs` finds the section, and `read_docs` returns only that.
+
+## Search with required and excluded words
+
+`search_docs` ranks the sections by the words of the query. A word matches in any form, so `rows` finds `row`. It also matches its synonyms, so `revert` finds `undo`. Words in a heading count most.
+
+Three marks narrow the search:
+
+* **`"a phrase"`**: the section must hold these words in a row, in any form, such as `"time travel"`
+* **`+word`**: the section must hold the word, such as `+view`
+* **`-word`**: the section must not hold the word, such as `-sql`; `-"a phrase"` leaves out a phrase
+
+The other words only rank the sections. This query finds the sections on time travel that mention views and never SQL. Here, `db` is a [client](https://datanotes.dev/develop/client/) connected to the engine, as in [Build your own MCP server](https://datanotes.dev/develop/build-on-datanotes/#build-your-own-mcp-server):
+
+```ts
+import { callDocsTool } from "datanotes/mcp";
+
+
+const hits = await callDocsTool(
+  "search_docs",
+  { query: '"time travel" +view -sql' },
+  { client: db },
+);
+```
+
+The marks filter the results, and never rank them. A section with every required word still ranks by all the words of the query. A word that starts with two dashes, such as `--vault`, is a plain word.
+
+## Search by words and by meaning
+
+When the engine has [semantic search](https://datanotes.dev/guides/search/semantic/) on, `search_docs` becomes a hybrid search. It ranks the sections twice:
+
+* **By words**: the ranking above, made by the MCP server from its own copy of the docs
+* **By meaning**: the engine embeds the query and ranks the sections by the similarity of their vectors
+
+Reciprocal Rank Fusion merges the two rankings. Each section scores the sum of 1 / (60 + its rank) over the rankings that hold it. Only the first 50 sections of each ranking count. A section that both rankings place high comes first, ahead of one that a single ranking finds. The required and excluded words filter both rankings.
+
+Each hit then ends with the rankings that found it: `(keyword, semantic)`, `(keyword)` or `(semantic)`. The last line of the answer says the search was hybrid.
+
+This is the hybrid search of vector databases such as ChromaDB, with no service to add. The engine already has the embedding model of semantic search. It keeps the vectors of the docs in its store, in `semantic/docs.json`, apart from the index of your notes.
+
+The engine builds that index in the background:
+
+* **When it starts**: after it updates the index of your notes
+* **When the docs change**: after an upgrade or a new docs build, the engine embeds again only the sections whose text changed
+* **When the model changes**: another provider, address or model starts a new index
+
+The version of the docs is a hash of their sections, so any change to a section counts. The docs are public: embedding them sends nothing from your vault. With a paid provider, the docs cost about as much to index as 700 short notes.
+
+`search_docs` ranks by words alone, as without semantic search, in these cases:
+
+* **No semantic search**: it’s off, or the engine is older than hybrid search
+* **The index is being built**: the first search after a change starts it
+* **The engine runs another version**: its docs differ from those of the MCP server
+* **The engine doesn’t answer**: `search_docs` asks it again a minute later
+
+## Read the docs from programs and editors
+
+Programs and people read the same pages:
+
+* **Programs**: the engine serves them, with a token like its other routes. `GET /datanotes/manual/` lists them, and `GET /datanotes/manual/guides/read-and-write/read-and-filter.md` returns one page. Add `?section=compare-values-with-operators` for one section.
+* **Search by meaning**: `POST /datanotes/manual/search/` takes a `query` and returns the sections closest in meaning, with `ready: false` while the engine builds the index. Programs that use `datanotes/mcp` pass a `client` to `callDocsTool` to search like `datanotes mcp`.
+* **People and editors**: the files themselves, in `node_modules/datanotes/dist/docs/`.
+
+```bash
+curl -H "Authorization: Bearer your_token" \
+  http://127.0.0.1:27150/datanotes/manual/start/quick-start.md
+```
+
+A program ranks the sections by meaning with the same [client](https://datanotes.dev/develop/client/):
+
+```ts
+const ranked = await db.post("/manual/search/", {
+  query: "undo what an agent did",
+});
+```
+
+In a checkout of the repository, build the docs first with `cd docs && npm ci && npm run build`, which copies the pages to `dist/docs/`. Until then, the tools answer with that command.

package/dist/docs/guides/agents/how-agents-use-datanotes.md

@@ -0,0 +1,124 @@
+# How agents work with your vault
+
+> What an agent reads first, how it finds tables and views, why the engine validates its writes, and how the journal records and undoes what each agent did.
+
+An agent connected through `datanotes mcp`, the Model Context Protocol (MCP) server of Datanotes, works with your vault the way a careful colleague works with a shared database. It reads the descriptions before it writes, writes only through operations the engine checks, and signs every change with its name in the journal. This page follows that path, from the instructions the agent receives to undoing what it did; the [prompts](https://datanotes.dev/guides/agents/prompts/) show it on concrete requests.
+
+## The agent starts with the server’s instructions
+
+When an MCP client connects, `datanotes mcp` sends instructions with its list of tools. The engine’s part is short:
+
+```text
+Database of Markdown notes: tables defined in metastore.yaml, a row is a note whose front matter key `table` names its table, and each front matter key is a column. Use list_tables / describe_table first, select_rows to read, insert_row / update_row / update_rows / batch to write (validated, journaled, undo with undo_migration). Change the schema only with its operations (create_table, add_column, alter_column, create_view, ...); never edit metastore.yaml. Every table and column needs a `description` of what it holds, or it is refused: descriptions are how agents and people understand the schema. Fill those describe_table lists in `missing_descriptions` with alter_table / alter_column. A body you write links the notes it mentions by name (a one-word name only with a capital, inside a sentence): when a write returns ambiguous_mentions, read the candidate notes and link the right one with update_row and autolink: false, or leave the text. Read the docs of this version with search_docs and read_docs.
+```
+
+The tools are every [operation](https://datanotes.dev/reference/operations/) of the engine, plus three that [read the docs of the installed version](#the-agent-reads-the-docs-of-its-version). To give agents rules and tools of your own, serve [your own MCP server](https://datanotes.dev/develop/build-on-datanotes/#build-your-own-mcp-server) next to it. Each tool’s description says when to call it: `list_tables` asks the agent to call it first, and `add_column` asks the agent to read `describe_table` before it adds a column.
+
+## The agent finds tables by their descriptions
+
+Before it reads or writes, the agent matches your words against what the tables say about themselves:
+
+* **`list_tables`**: every table and view, with its `description`, `aliases` and `tags`, so “my watchlist” leads to `film`, whose aliases include `watchlist`
+* **`describe_table`**: the columns, each with its type, `description`, `aliases`, limits and, for an enum, the meaning of each value, plus the table’s row count
+
+The meanings come from `value_descriptions`, which you give when you create the column. On the film diary, `describe_table` answers this for `status`:
+
+```json
+{
+  "name": "status",
+  "type": "enum",
+  "values": ["to watch", "watched"],
+  "value_descriptions": {
+    "to watch": "on my watchlist, not seen yet",
+    "watched": "I saw it at least once"
+  },
+  "ordered": true,
+  "default": "to watch"
+}
+```
+
+Every table and every column has a description, because the engine refuses one without. Agents and people understand the schema through them, so write each for a reader who never saw your notes. Good aliases help too: the better both are, the less the agent guesses. You set them with `create_table`, `add_column`, `alter_table`, `alter_column` and, for catalogs and schemas, `comment_on`.
+
+A table created before descriptions were required may lack some. `describe_table` lists them in `missing_descriptions`, and the agent fills them with `alter_table` and `alter_column`. [Describe every table and column](https://datanotes.dev/guides/database-objects/tables/#describe-every-table-and-column) shows the rule in use.
+
+## The agent reads views first
+
+A view is a stored query with a name, a description and aliases, such as `vw_to_watch`: the films whose status is `to watch`, shortest first. `list_tables` returns views next to tables, and `select_rows` reads a view like a table and can narrow it further. When a view answers the question, the agent reads it instead of rebuilding the filter, so the answer follows your definition of “to watch” and not the agent’s. [Views](https://datanotes.dev/guides/database-objects/views/) shows views with joins, groups and filters on the note they’re shown for.
+
+## The agent writes only through checked operations
+
+Rows change through `insert_row`, `update_row`, `update_rows`, `delete_rows` and `batch`; the schema changes through operations such as `create_table`, `add_column` and `add_constraint`. The engine checks every one against the table’s column types, limits, links and CHECK constraints, and writes nothing when a check fails. Three arguments make an agent’s writes safer:
+
+* **`dry_run`**: reports what would change, without writing
+* **`batch`**: a group of writes that succeed or fail together, and that you undo as one
+* **`expect`** on `update_row`: writes only if the row still has the values the agent read, so it doesn’t overwrite an edit you made in the meantime
+
+The agent never edits `metastore.yaml`: a hand edit skips validation and has no undo. [Concurrent writes](https://datanotes.dev/develop/concurrency/) explains what happens when agents, programs and people write at the same time.
+
+## A refusal tells the agent what to fix
+
+A refused write names the note, the column or rule, and the values the engine accepts. It ends with a hint on how to fix it and a link to the page that shows the fix. For a film with the status `seen`, the engine answers:
+
+```text
+insert_row refused — nothing was written:
+invalid datanotes/main/public/film/Sunrise.md: status must be one of: to watch (on my watchlist, not seen yet); watched (I saw it at least once)
+Hint: Use one of the allowed values listed in the message, written exactly; describe_table shows what each one means. To add a value, alter_column the enum. Then call insert_row again (dry_run: true checks without writing).
+Docs: read_docs("guides/database-objects/columns#describe-what-each-enum-value-means")
+```
+
+The agent can correct the value and try again, or ask you. Every other error follows the same pattern: a misspelled table or column gets the closest real name, a wrong filter operator gets the one meant, and a tool argument the tool doesn’t take gets the tool’s arguments. [Read the engine’s errors](https://datanotes.dev/guides/agents/reading-errors/) shows them all. Before it writes a link, the agent can call `list_link_targets`, which returns the rows the link column accepts, such as the genres a film may link to. Over HTTP, a refused write comes back with status 422 and the reasons in the body.
+
+## The agent reads the docs of its version
+
+`datanotes mcp` also serves this documentation, read from the installed `datanotes` package, with three tools: `list_docs`, `read_docs` and `search_docs`. The agent looks up an operation or an example without internet access, and the pages always match the version it talks to. [Read the docs offline](https://datanotes.dev/guides/agents/docs-offline/) shows the tools.
+
+## Every write names the agent that made it
+
+The engine records an actor with every change. Through `datanotes mcp`, the actor comes from the MCP connection:
+
+* **`harness`** and **`harness_version`**: the client’s name and version from the MCP handshake, such as `claude-code`
+* **`session`**: the MCP session, shortened to 16 characters
+* **`tool`**: the tool the agent called
+* **`auth`**: `owner`, or the token’s name for a [named token](https://datanotes.dev/start/connect-agents/#give-each-agent-its-own-token)
+* **`model`**: the model, when an MCP server reports it with `POST /datanotes/sessions/identify/`
+
+`describe_history` shows the actor of each commit:
+
+```json
+{
+  "table": "main.public.film",
+  "version": 10,
+  "operation": "UPDATE",
+  "migration_id": "20261009T215306-update_rows-update-film",
+  "actor": {
+    "session": "5d8c0456-3b15-40",
+    "harness": "claude-code",
+    "harness_version": "2.5.0",
+    "tool": "batch",
+    "auth": "owner"
+  },
+  "batch": "20261009T215306-batch-log-the-screening-of-metropoli"
+}
+```
+
+Three operations read this record:
+
+* **`list_migrations`**: the journal, newest first, each operation with `by`, the agent’s name
+* **`describe_history`**: the commits of one table or of every table, with the full actor; it needs history on
+* **`list_changes`**: every row change with its actor, and with `kinds: ["tool_call"]` every tool call the MCP server reported, reads included
+
+A script records its own actor with the `X-Datanotes-Actor` header or the client’s `actor` option, as the [HTTP API guide](https://datanotes.dev/develop/http-api/#record-who-makes-each-write) shows.
+
+## Undo what an agent did
+
+Because the journal records every write, you can take back one operation or a whole run:
+
+* **One operation**: `undo_migration` with its `migration_id`; for a `batch`, it reverts every write in it
+* **A whole run**: `restore_database` with `before_migration` set to the agent’s first operation or batch; every table with history goes back to its state before that operation (run it with `dry_run` first)
+* **One table**: `restore_table` to a version or a time
+
+The journal records an undo too, so you can undo the undo. The engine refuses an undo that would overwrite a note you edited since, and lists the conflicts. [Restore a table or the database](https://datanotes.dev/guides/history/restore/) shows each operation with the engine’s answers, and the [prompts](https://datanotes.dev/guides/agents/prompts/#safety-and-history-prompts) show how to ask for them.
+
+## Give each agent only what it needs
+
+The owner token can do everything. A [named token](https://datanotes.dev/guides/security/tokens/) limits an agent to the tables in `write`, to schema changes only with `ddl`, and to configuration, undo and restore only with `maintain`. The journal then names the token with each of its writes. [Connect agents](https://datanotes.dev/start/connect-agents/) shows how to create one and pass it to an MCP client.