datanotes 2.0.0 → 3.0.0

package/dist/docs/reference/settings.md

@@ -0,0 +1,31 @@
+# Settings
+
+> Every key of the engine's settings file and its default.
+
+The headless engine reads `settings.json` in its store, outside the vault (`datanotes store path` shows where), or the file given with `--settings`. Missing keys take their default.
+
+| Key                          | Default                   | Meaning                                                                                                                                                                                                                                                                                                                             |
+| ---------------------------- | ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `journalRetentionDays`       | `90`                      | Journal entries older than this many days are dropped (undo is no longer possible for them).                                                                                                                                                                                                                                        |
+| `fileLogEnabled`             | `true`                    | Write log records to `logs/<day>.<source>.jsonl` in the engine’s store.                                                                                                                                                                                                                                                             |
+| `logRetentionDays`           | `30`                      | Log files older than this many days are deleted.                                                                                                                                                                                                                                                                                    |
+| `tableKey`                   | `"table"`                 | Row property naming the catalog table a note belongs to (`table: personal.metrics.mood`).                                                                                                                                                                                                                                           |
+| `telemetryEnabled`           | `false`                   | Send the engine’s logs to an OpenTelemetry collector or OTLP backend (OTLP/HTTP).                                                                                                                                                                                                                                                   |
+| `telemetryEndpoint`          | `"http://127.0.0.1:4318"` | OTLP/HTTP base URL; logs are POSTed to `<endpoint>/v1/logs`.                                                                                                                                                                                                                                                                        |
+| `telemetryServiceName`       | `"datanotes"`             | `service.name` resource attribute.                                                                                                                                                                                                                                                                                                  |
+| `telemetryHeaders`           | `{}`                      | HTTP headers sent with every export: the API key or `Authorization` the backend asks for. Kept in the settings file; `OTEL_EXPORTER_OTLP_HEADERS` adds to them.                                                                                                                                                                     |
+| `telemetryProtocol`          | `"http/protobuf"`         | OTLP encoding: `http/protobuf` (the OpenTelemetry default) or `http/json`.                                                                                                                                                                                                                                                          |
+| `autoRecompute`              | `true`                    | After a row is edited by hand, recompute its generated / identity columns (a few seconds after the last change, as a journaled `recompute_columns` operation).                                                                                                                                                                      |
+| `repairLinksOnHandMove`      | `true`                    | Rewrite the links to a note renamed or moved by hand (in a file manager, with git or a sync tool), across the vault, once the vault has been quiet for 4 s and not while git pulls, rebases or merges. The repair is one journaled `repair_links` operation that undo takes back. Links an editor already rewrote stay as they are. |
+| `semanticEnabled`            | `false`                   | Semantic search: embed notes with `semanticProvider` (off by default: content is sent to it).                                                                                                                                                                                                                                       |
+| `semanticProvider`           | `"ollama"`                | Embedding provider: `ollama`, `lmstudio`, `openrouter`, `openai` or `custom` (any OpenAI-compatible API).                                                                                                                                                                                                                           |
+| `semanticBaseUrl`            | `""`                      | Empty: the provider’s default URL.                                                                                                                                                                                                                                                                                                  |
+| `semanticModel`              | `""`                      | Embedding model, e.g. `nomic-embed-text` (Ollama) or `openai/text-embedding-3-small` (OpenRouter).                                                                                                                                                                                                                                  |
+| `semanticScope`              | `"vault"`                 | “vault”: every note; “tables”: only table rows.                                                                                                                                                                                                                                                                                     |
+| `semanticExclude`            | `[]`                      | Folders left out of the index.                                                                                                                                                                                                                                                                                                      |
+| `semanticBatchSize`          | `32`                      | Passages per embedding request.                                                                                                                                                                                                                                                                                                     |
+| `httpServerPort`             | `27150`                   | Port of the HTTP API (on 127.0.0.1 unless the service is told otherwise): 1024 to 65535, or 0 for any free port. Any other value stops the engine from starting.                                                                                                                                                                    |
+| `httpServerToken`            | `""`                      | The owner’s bearer token (generated at the first start).                                                                                                                                                                                                                                                                            |
+| `storeBackupEnabled`         | `false`                   | Back up the engine’s store to a git repository: commits as `storeBackupIntervalSeconds` says, each pushed to `storeBackupRemote` (headless engine; `datanotes store backup` sets it).                                                                                                                                               |
+| `storeBackupRemote`          | `""`                      | The git remote the store backup pushes to, such as a private GitHub repository. Empty: commits in the store only, nothing pushed.                                                                                                                                                                                                   |
+| `storeBackupIntervalSeconds` | `60`                      | How often the store backup looks for changes, in seconds (at least 10): when something changed, it commits between two operations (a consistent snapshot), then pushes.                                                                                                                                                             |

package/dist/docs/release-notes.md

@@ -0,0 +1,67 @@
+# Release notes
+
+> What each version of Datanotes brings, newest first, starting with 3.0.0.
+
+This page lists each version of Datanotes, newest first. Datanotes follows [semantic versioning](https://semver.org): a major version can break what you rely on, and its notes list those changes first. The package is on [npm](https://www.npmjs.com/package/datanotes), and each version has a [GitHub release](https://github.com/davidefornelli/datanotes/releases).
+
+## 3.0.0
+
+Datanotes 3.0.0 is the first stable release. Versions 1.x and 2.x, published on 7 and 8 October 2026, were previews. Datanotes runs a folder of Markdown notes as typed database tables. You reach it through an HTTP API, a command line and an MCP server for agents.
+
+### Tables and the metastore
+
+* **One metastore per vault**: `metastore.yaml` defines the tables, views and bases. Only operations change it, and each change is validated and journaled. The engine refuses a vault with more than one `metastore.yaml`.
+* **The database folder**: a new database goes to `datanotes/`. The engine finds the folder by its `metastore.yaml`, whatever its name. See [Move the vault or the database folder](https://datanotes.dev/start/how-it-works/#move-the-vault-or-the-database-folder).
+* **Descriptions**: `create_table`, `add_column`, `clone_table` and `define_base` require a description for each table, base and column. A refusal lists every missing one. See [Describe every table and column](https://datanotes.dev/guides/database-objects/tables/#describe-every-table-and-column).
+* **CHECK constraints that follow links**: a CHECK such as `{"parent.kind": "project"}` tests the linked row.
+* **Filtered link targets**: `ref_where` and `ref_by` limit what a link may point to, and `list_link_targets` lists those rows.
+* **Hand edits are checked**: the engine checks notes edited outside it against the tables’ rules. The change feed, the transaction log and the log report what they break.
+* **Schema operations for hand-written metastores**: table `tags`, `alter_table` with `archive` and `fields`, and `comment_on` for catalogs and schemas.
+* **`select_rows` with `this_note`**: fills a view’s `{this.*}` tokens from a note, so one view serves every daily note.
+
+### The store and its backup
+
+* **The store is outside the vault**: settings, tokens, journal, transaction logs, change feed, semantic index, logs and lock live in `<data home>/<database id>/`. See [Where the engine keeps its store](https://datanotes.dev/administration/run-the-service/#where-the-engine-keeps-its-store).
+* **A database id**: the engine writes an `id` line at the top of `metastore.yaml` and finds the store by it.
+* **`datanotes store`**: `datanotes store path` shows where a vault’s store is. `datanotes store new-id` gives a copied database its own id and store.
+* **Store backup**: the engine keeps its store in a git repository of its own and pushes it to a private remote. Each commit is a consistent snapshot, and the secrets stay out. `datanotes store restore` brings the store back on a new machine. See [Back up the history](https://datanotes.dev/guides/history/back-up/).
+* **`--store` and `DATANOTES_DATA_HOME`**: place the store and move the data home. The Docker image keeps the store in `/data`.
+
+### History and undo
+
+* **The journal**: every write is journaled, and `undo_migration` reverts it. An undo of an undo is a redo, and a `batch` is one entry.
+* **History on in a new database**: a database the engine creates gets `history: true` in its `metastore.yaml`, so every table keeps its versions. A database without the setting keeps history off. See [The transaction log keeps every version](https://datanotes.dev/start/how-it-works/#the-transaction-log-keeps-every-version).
+* **History for the whole database**: `restore_database` brings every table back to a time, or to the moment before an operation.
+* **Time travel**: `describe_history` without `table` lists the commits of every table. `select_rows` with `timestamp` reads views, joins and bases as they were.
+* **Moves of the database folder**: a move by hand or with `configure_database` is one operation. Undo and time travel read what came before it with the new paths.
+
+### Notes and links
+
+* **Automatic links**: a write links the notes its text mentions by name, at their first mention. `links_added` lists them, and `ambiguous_mentions` returns the doubtful ones for the agent to decide. See [Automatic links](https://datanotes.dev/guides/read-and-write/automatic-links/).
+* **Safe on a large vault**: a one-word name links only when capitalized and not at the start of a sentence. Names inside paths, domains and snake_case words stay as they are.
+* **`autolink_target`**: a table option that says whether automatic links may point to its rows. See [Choose the tables a link may point to](https://datanotes.dev/guides/read-and-write/automatic-links/#choose-the-tables-a-link-may-point-to).
+* **Links follow a note renamed by hand**: the engine rewrites every link to its old path, in one `repair_links` operation that undo takes back. See [The engine repairs links after a hand move](https://datanotes.dev/start/how-it-works/#the-engine-repairs-links-after-a-hand-move).
+* **A note moved by hand is a rename**: a note gone and a new one with its text, seen at once, are one rename.
+
+### Agents
+
+* **`datanotes mcp`**: the MCP server ships in the `datanotes` package. [Connect your agents](https://datanotes.dev/start/connect-agents/) shows how to configure it.
+* **`datanotes/mcp`**: the engine’s operations as MCP tool definitions for an MCP server of your own. See [Build on Datanotes](https://datanotes.dev/develop/build-on-datanotes/).
+* **Docs offline**: the package ships these docs as Markdown. Agents read them with `list_docs`, `read_docs` and `search_docs`. The `Docs:` line of an error names the `read_docs` call, so the agent reads the section without internet access. Your own MCP server does the same with `offlineDocs`. See [Read the docs offline](https://datanotes.dev/guides/agents/docs-offline/).
+* **Hybrid search of the docs**: `search_docs` merges a ranking by words with one by meaning. See [Search by words and by meaning](https://datanotes.dev/guides/agents/docs-offline/#search-by-words-and-by-meaning).
+* **Errors that teach**: every error says what is wrong, how to fix it in `hint`, and where to read more in `docs`. See [Errors](https://datanotes.dev/reference/errors/).
+
+### Security
+
+* **Named tokens**: each program gets a token of its own, with the tables it may read and write. A token’s `autolink` sets the default of its writes. See [Turn automatic links off for a sync program](https://datanotes.dev/guides/security/tokens/#turn-automatic-links-off-for-a-sync-program).
+* **The actor**: every write records who made it, with `auth` set to `owner` or the token’s name.
+* **No secret in the change feed**: a recorded tool call loses webhook secrets and values under keys such as `token` or `password`.
+
+### Running the engine
+
+* **Node.js 22 or newer**: the engine and the command line need Node.js 22 or later.
+* **`datanotes serve --port`**: the port for one run, also as `DATANOTES_PORT`. With `--port 0`, the system picks a free port. See [Choose the port](https://datanotes.dev/administration/run-the-service/#choose-the-port).
+* **`datanotes status`**: the engine’s health at a glance. See [Check the engine’s status](https://datanotes.dev/administration/status/).
+* **`datanotes stop`**: a graceful stop. The engine finishes its requests, makes its last git sync and backup push, and releases its lock. See [Stop the engine](https://datanotes.dev/administration/run-the-service/#stop-the-engine).
+* **One engine per database**: the engine holds a lock in its store and checks the lock’s process on the same machine.
+* **Git never waits for a person**: git sync and the store backup run git without prompts, and with time limits. See [Git never waits for a person](https://datanotes.dev/administration/git-sync/#git-never-waits-for-a-person).

package/dist/docs/start/connect-agents.md

@@ -0,0 +1,204 @@
+# Connect your agents to a vault
+
+> Configure Claude, Cursor, Codex and other MCP clients with datanotes mcp, the Model Context Protocol server of Datanotes.
+
+Agents such as Claude, Cursor and Codex reach your vault through the Model Context Protocol (MCP) server of Datanotes, the `datanotes mcp` command. This page shows how to:
+
+* Start the server for an agent on your computer
+* Serve it over HTTP to more than one agent
+* Choose the tools each agent sees
+* Give each agent its own token and see what it changed
+* Protect your database from prompt injection
+
+## How the MCP server reaches the engine
+
+`datanotes mcp` ships in the `datanotes` package and runs as its own process, apart from the engine. It offers the [operations](https://datanotes.dev/reference/operations/) as MCP tools, all but the webhook tools [unless you add them](#choose-the-tools-an-agent-sees), plus `list_docs`, `read_docs` and `search_docs`, which [read the docs](#read-the-docs-offline) of the installed version. Tools of other programs come from their own MCP servers, as [Build on Datanotes](https://datanotes.dev/develop/build-on-datanotes/) explains. Each tool call becomes a request to the engine’s [HTTP API](https://datanotes.dev/develop/http-api/) with the agent’s token. The engine then checks what the token may do and records who did what.
+
+An MCP client starts the server in one of two ways:
+
+* **Standard input and output (stdio)**: the client starts `datanotes mcp` itself, on the computer that holds the vault
+* **HTTP**: the client calls `/mcp` on a running `datanotes mcp serve` and sends a bearer token; use it for more than one client at once, or for a hosted assistant that can send a custom `Authorization` header (the server offers no OAuth)
+
+## Connect an agent on your computer
+
+You need [Node.js](https://nodejs.org) 22 or newer. On Windows, use Node.js 24.16 or later, or 26.1 or later. Node.js 24.0 to 24.15 and 26.0 can crash at random on local connections. `datanotes mcp` warns at start on those versions.
+
+With `--database` or `--vault`, `datanotes mcp` uses the engine that runs on that folder. Give the same option as `datanotes serve`. When no engine runs, it starts one with `datanotes serve`. When the client closes, it asks that engine to stop, as `datanotes stop` does. The engine finishes its writes and releases the folder. Only an engine still running after 35 seconds is killed.
+
+It acts with the owner’s token from the engine’s settings, in its store, so the agent has full rights. To limit an agent, use one of these:
+
+* [Choose its tools](#choose-the-tools-an-agent-sees) with `--profile`, `--tools` or `--except`
+* Give it a [named token](#give-each-agent-its-own-token) with `--url` and `--token-file`
+* Connect it over HTTP with its own token
+
+The agent waits while the engine starts, for up to 60 seconds. A large database takes longer to start. Set `DATANOTES_ENGINE_START_TIMEOUT_S` to wait longer. Above about 20,000 notes, run `datanotes serve` as a service instead, as [Limits](https://datanotes.dev/administration/limits/#start-the-engine-before-the-agent) explains.
+
+Add the server to Claude Code with one command:
+
+```bash
+claude mcp add datanotes -- \
+  npx -y datanotes mcp --database your_database_folder
+```
+
+Claude Desktop, Cursor and other clients with an `mcpServers` file take the same command and arguments:
+
+```json
+{
+  "mcpServers": {
+    "datanotes": {
+      "command": "npx",
+      "args": [
+        "-y", "datanotes", "mcp", "--database", "your_database_folder"
+      ]
+    }
+  }
+}
+```
+
+Codex reads its MCP servers from `~/.codex/config.toml`:
+
+```toml
+[mcp_servers.datanotes]
+command = "npx"
+args = ["-y", "datanotes", "mcp", "--database", "your_database_folder"]
+```
+
+On Windows, escape the backslashes in JSON and TOML (`C:\\Notes\\Films`). Claude Code on Windows, outside Windows Subsystem for Linux (WSL), starts `npx` through `cmd`:
+
+```bash
+claude mcp add datanotes -- cmd /c npx -y datanotes mcp --database C:\Notes
+```
+
+To use an engine at an address instead of a local folder, pass the address and a file that contains the token:
+
+```bash
+npx -y datanotes mcp --url https://datanotes.example.com \
+  --token-file ~/.datanotes-token
+```
+
+## Serve agents over HTTP
+
+`datanotes mcp serve` serves MCP over Streamable HTTP at `/mcp`. Run it next to the engine:
+
+```bash
+npx -y datanotes mcp serve --url http://127.0.0.1:27150 --port 27151
+```
+
+Clients connect to `http://127.0.0.1:27151/mcp` and send their own token. Add it to Claude Code with:
+
+```bash
+claude mcp add --transport http datanotes http://127.0.0.1:27151/mcp \
+  --header "Authorization: Bearer your_agent_token"
+```
+
+Cursor and the `.mcp.json` file of Claude Code take the address and the header:
+
+```json
+{
+  "mcpServers": {
+    "datanotes": {
+      "type": "http",
+      "url": "http://127.0.0.1:27151/mcp",
+      "headers": { "Authorization": "Bearer your_agent_token" }
+    }
+  }
+}
+```
+
+The server keeps no credentials: it passes the token of each request to the engine, and refuses a token the engine doesn’t accept. `GET /` answers its status without a token. To reach the server from another device, publish only `/mcp` through a reverse proxy or a tunnel, over HTTPS, as [Security](https://datanotes.dev/guides/security/security-model/) explains.
+
+## Choose the tools an agent sees
+
+By default, `datanotes mcp` offers every tool except `create_webhook` and `delete_webhook`. A webhook sends your database’s changes, and with `since` its past ones too, to any web address. So an agent gets these two tools only when you add them. Both stdio and `serve` take the same options:
+
+| Option            | What the agent sees                                                                    |
+| ----------------- | -------------------------------------------------------------------------------------- |
+| None              | Every tool except `create_webhook` and `delete_webhook`                                |
+| `--profile read`  | Only the tools that read: `select_rows`, `describe_table`, the docs tools and the like |
+| `--profile write` | Every tool except the schema changes, `configure_database` and the webhook tools       |
+| `--profile all`   | Every tool, the webhook tools included                                                 |
+| `--tools a,b`     | Only the tools you name, separated by commas                                           |
+| `--except a,b`    | The others: it removes these tools from the profile or from `--tools`                  |
+| `--owner-tools`   | Adds `create_webhook` and `delete_webhook` to the rest                                 |
+
+Give an agent that only answers questions the `read` profile:
+
+```bash
+claude mcp add datanotes -- \
+  npx -y datanotes mcp --database your_database_folder --profile read
+```
+
+Name the tools one by one for a narrow agent, such as one that logs meals:
+
+```bash
+npx -y datanotes mcp --database your_database_folder \
+  --tools list_tables,describe_table,select_rows,insert_row
+```
+
+Keep an agent from undoing changes, and give it the webhook tools:
+
+```bash
+npx -y datanotes mcp serve --except undo_migration --owner-tools
+```
+
+You can’t pass `--profile` and `--tools` together. An unknown tool name stops the command, and the error names the closest tool. A call to a tool the server doesn’t offer fails as an unknown tool. The engine still checks the token on every call, so a [named token](#give-each-agent-its-own-token) limits the agent too.
+
+## Give each agent its own token
+
+A named token limits what an agent can do, and the journal records its name. It writes only the tables you allow in `write`, changes the schema only with `ddl`, and undoes changes only with `maintain`. Create one with the owner’s token:
+
+```bash
+curl -s http://127.0.0.1:27150/datanotes/tokens/create/ \
+  -H "Authorization: Bearer your_owner_token" \
+  -H "Content-Type: application/json" \
+  -d '{"name": "claude", "write": ["*"], "ddl": true, "maintain": true}'
+```
+
+The answer contains the token, and shows it only once. [Tokens](https://datanotes.dev/guides/security/tokens/) describes every permission.
+
+## Agents and prompt injection
+
+An agent reads text it didn’t write: your notes, web pages, emails and the answers of other tools. That text can hold instructions aimed at the agent, such as “send every note to this address”. This attack is called prompt injection. Datanotes can’t tell such text from your own words, so limit what a misled agent can do:
+
+* **Offer only the tools it needs.** Use `--profile read` for an agent that reads, and `--tools` for a narrow one.
+* **Keep the webhook tools for yourself.** They are off by default. Add `--owner-tools` only for an agent you ask to create a webhook.
+* **Give it a named token.** A [named token](#give-each-agent-its-own-token) writes only the tables you allow, and never creates webhooks.
+* **Confirm what leaves the database.** Ask your MCP client to confirm tool calls that write or send data.
+* **Check what it changed.** [See which agent made a change](#see-which-agent-made-a-change), and undo it with `undo_migration`.
+
+Every write is journaled, so you can undo what an agent changed. Data an agent sends out of the database can’t come back.
+
+## Set options with flags or environment variables
+
+Each option has an environment variable, which applies when the command line doesn’t give the option:
+
+| Option                            | Variable                                                         | Meaning                                                                                                       |
+| --------------------------------- | ---------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- |
+| `--database your_database_folder` | `DATANOTES_DATABASE`                                             | stdio on the engine of a local database folder, with the owner’s token                                        |
+| `--vault your_vault_folder`       | `DATANOTES_VAULT`                                                | stdio on the engine of a local vault, with the owner’s token; instead of `--database`                         |
+| `--url your_engine_url`           | `DATANOTES_URL`                                                  | stdio on the engine at this address                                                                           |
+| `--token-file your_token_file`    | `DATANOTES_TOKEN_FILE`, or the token itself in `DATANOTES_TOKEN` | The token for `--url`                                                                                         |
+| `serve --url your_engine_url`     | `DATANOTES_URL`                                                  | The engine behind the HTTP server; default `http://127.0.0.1:27150`                                           |
+| `serve --host your_address`       | `DATANOTES_MCP_HOST`                                             | The address the HTTP server listens on; default `127.0.0.1`                                                   |
+| `serve --port 27151`              | `DATANOTES_MCP_PORT`                                             | The port of the HTTP server; default `27151`                                                                  |
+| `--profile read`                  | `DATANOTES_MCP_PROFILE`                                          | The [tools](#choose-the-tools-an-agent-sees) to offer: `read`, `write` or `all`                               |
+| `--tools a,b`                     | `DATANOTES_MCP_TOOLS`                                            | Only these tools, separated by commas; instead of `--profile`                                                 |
+| `--except a,b`                    | `DATANOTES_MCP_EXCEPT`                                           | None of these tools, separated by commas                                                                      |
+| `--owner-tools`                   | `DATANOTES_MCP_OWNER_TOOLS=1`                                    | Also offer `create_webhook` and `delete_webhook`                                                              |
+| `--otlp-endpoint your_endpoint`   | `OTEL_EXPORTER_OTLP_ENDPOINT`                                    | The OpenTelemetry Protocol (OTLP) collector that receives the logs                                            |
+| None                              | `DATANOTES_DOCS_DIR`                                             | The folder of the docs that `list_docs`, `read_docs` and `search_docs` read; default: the ones in the package |
+| None                              | `DATANOTES_ENGINE_START_TIMEOUT_S`                               | Seconds `--database` waits for the engine it starts; default `60`                                             |
+
+`datanotes mcp --help` prints the same list, and [Command line](https://datanotes.dev/reference/cli/) shows every command of `datanotes`.
+
+## Read the docs offline
+
+The `datanotes` package ships this documentation as Markdown, for the installed version. Agents read it with `list_docs`, `read_docs` and `search_docs`, programs through the engine, and people in `node_modules/datanotes/dist/docs/`: [Read the docs offline](https://datanotes.dev/guides/agents/docs-offline/) shows each way.
+
+## Send the server’s logs to your backend
+
+`datanotes mcp` sends its logs with OpenTelemetry when `OTEL_EXPORTER_OTLP_ENDPOINT` or `--otlp-endpoint` names a collector. The records cover tool calls, refused requests and sessions, and never contain tokens or argument values. [Logs and telemetry](https://datanotes.dev/develop/telemetry/) shows how to set up the backend.
+
+## See which agent made a change
+
+Every write records its actor: the client’s name from the MCP handshake, its session, the tool, and the credential as `auth` (`owner`, or the token’s name). An MCP server of your own can also report the agent’s model with `POST /datanotes/sessions/identify/`. `list_migrations` shows who made each change in its `by` field, and `list_changes` shows the full actor of every change and tool call.