datanotes 2.0.1 → 3.1.0

package/dist/docs/guides/search/link-graph.md

@@ -0,0 +1,241 @@
+# Walk the link graph
+
+> Read the links between notes as a graph: neighbors, walks filtered by label and table, and the shortest path between two notes.
+
+The engine reads your notes and their links as a labelled graph. Edges from link columns have the column’s name as their label, such as `cast` or `genre`; other resolved links have the label `link`, and each row has its table as its label.
+
+The vault of the examples has three actors, two classic films, a viewing and two plain notes.
+
+## See what a note links to and what links to it
+
+`graph_neighbors` returns a note’s links in both directions. Each link has a label: the link column it comes from (`cast`), or `link` for a link in the text.
+
+**`graph_neighbors`**
+
+```json
+{ "from_nodes": { "table": "actor", "where": { "name": "Cary Grant" } } }
+```
+
+Answer:
+
+```json
+{
+  "nodes": [
+    {
+      "path": "datanotes/main/public/actor/Cary Grant.md",
+      "table": "main.public.actor",
+      "counts": { "in:cast": 2, "in:link": 1 },
+      "edges": [
+        {
+          "path": "datanotes/main/public/film/Charade.md",
+          "table": "main.public.film",
+          "label": "cast",
+          "direction": "in"
+        },
+        {
+          "path": "datanotes/main/public/film/His Girl Friday.md",
+          "table": "main.public.film",
+          "label": "cast",
+          "direction": "in"
+        },
+        { "path": "Notes/Film night.md", "label": "link", "direction": "in" }
+      ]
+    }
+  ]
+}
+```
+
+## Read the whole graph in one call
+
+`tables` starts from the rows of several tables at once. Name the tables, or use a pattern: `*` is every table, `main.*` every table of the `main` catalog. With `direction: "out"`, one call returns every link of every row: the whole graph, for an app that draws it. The start set holds at most 5,000 notes.
+
+**`graph_neighbors`**
+
+```json
+{ "from_nodes": { "tables": ["*"] }, "direction": "out" }
+```
+
+Answer (80 lines)
+
+```json
+{
+  "nodes": [
+    {
+      "path": "datanotes/main/public/actor/Audrey Hepburn.md",
+      "table": "main.public.actor",
+      "counts": {},
+      "edges": []
+    },
+    {
+      "path": "datanotes/main/public/actor/Cary Grant.md",
+      "table": "main.public.actor",
+      "counts": {},
+      "edges": []
+    },
+    {
+      "path": "datanotes/main/public/actor/Rosalind Russell.md",
+      "table": "main.public.actor",
+      "counts": {},
+      "edges": []
+    },
+    {
+      "path": "datanotes/main/public/film/Charade.md",
+      "table": "main.public.film",
+      "counts": { "out:cast": 2, "out:link": 1 },
+      "edges": [
+        {
+          "path": "datanotes/main/public/actor/Cary Grant.md",
+          "table": "main.public.actor",
+          "label": "cast",
+          "direction": "out"
+        },
+        {
+          "path": "datanotes/main/public/actor/Audrey Hepburn.md",
+          "table": "main.public.actor",
+          "label": "cast",
+          "direction": "out"
+        },
+        {
+          "path": "datanotes/main/public/film/His Girl Friday.md",
+          "table": "main.public.film",
+          "label": "link",
+          "direction": "out"
+        }
+      ]
+    },
+    {
+      "path": "datanotes/main/public/film/His Girl Friday.md",
+      "table": "main.public.film",
+      "counts": { "out:cast": 2 },
+      "edges": [
+        {
+          "path": "datanotes/main/public/actor/Cary Grant.md",
+          "table": "main.public.actor",
+          "label": "cast",
+          "direction": "out"
+        },
+        {
+          "path": "datanotes/main/public/actor/Rosalind Russell.md",
+          "table": "main.public.actor",
+          "label": "cast",
+          "direction": "out"
+        }
+      ]
+    },
+    {
+      "path": "datanotes/main/public/watch/2026-09-25T2100.md",
+      "table": "main.public.watch",
+      "counts": { "out:film": 1, "out:link": 1 },
+      "edges": [
+        {
+          "path": "datanotes/main/public/film/Charade.md",
+          "table": "main.public.film",
+          "label": "film",
+          "direction": "out"
+        },
+        { "path": "Notes/Film night.md", "label": "link", "direction": "out" }
+      ]
+    }
+  ]
+}
+```
+
+## Walk the links from a note
+
+`graph_traverse` follows links breadth-first, in the `direction` you choose: `out`, `in` or `both`. Here it starts from Audrey Hepburn, goes up to two steps in both directions and keeps only films. It finds Charade, the film she’s in, and His Girl Friday, which the Charade note links to. `via` shows the route to each one.
+
+**`graph_traverse`**
+
+```json
+{
+  "from_nodes": { "paths": ["datanotes/main/public/actor/Audrey Hepburn.md"] },
+  "direction": "both",
+  "tables": ["film"],
+  "max_depth": 2
+}
+```
+
+Answer:
+
+```json
+{
+  "start": ["datanotes/main/public/actor/Audrey Hepburn.md"],
+  "nodes": [
+    {
+      "path": "datanotes/main/public/film/Charade.md",
+      "table": "main.public.film",
+      "depth": 1,
+      "via": [
+        {
+          "from": "datanotes/main/public/actor/Audrey Hepburn.md",
+          "label": "cast",
+          "direction": "in",
+          "to": "datanotes/main/public/film/Charade.md"
+        }
+      ]
+    },
+    {
+      "path": "datanotes/main/public/film/His Girl Friday.md",
+      "table": "main.public.film",
+      "depth": 2,
+      "via": [
+        {
+          "from": "datanotes/main/public/actor/Audrey Hepburn.md",
+          "label": "cast",
+          "direction": "in",
+          "to": "datanotes/main/public/film/Charade.md"
+        },
+        {
+          "from": "datanotes/main/public/film/Charade.md",
+          "label": "link",
+          "direction": "out",
+          "to": "datanotes/main/public/film/His Girl Friday.md"
+        }
+      ]
+    }
+  ],
+  "truncated": false
+}
+```
+
+To read or total those rows, pass the paths to `select_rows` with `where: {"path": {"in": [...]}}`.
+
+## Find how two notes are connected
+
+`graph_path` finds the shortest chain of links between two notes, or between two sets of notes.
+
+**`graph_path`**
+
+```json
+{
+  "from_nodes": {
+    "paths": ["Notes/Rewatch ideas.md", "datanotes/main/public/actor/Audrey Hepburn.md"]
+  },
+  "to_nodes": { "paths": ["Notes/Film night.md"] }
+}
+```
+
+Answer:
+
+```json
+{
+  "found": true,
+  "length": 2,
+  "via": [
+    {
+      "from": "datanotes/main/public/actor/Audrey Hepburn.md",
+      "label": "cast",
+      "direction": "in",
+      "to": "datanotes/main/public/film/Charade.md"
+    },
+    {
+      "from": "datanotes/main/public/film/Charade.md",
+      "label": "link",
+      "direction": "in",
+      "to": "Notes/Film night.md"
+    }
+  ],
+  "from": "datanotes/main/public/actor/Audrey Hepburn.md",
+  "to": "Notes/Film night.md"
+}
+```

package/dist/docs/guides/search/semantic.md

@@ -0,0 +1,133 @@
+# Find notes by meaning
+
+> Turn on semantic search with a local or hosted embedding model, and search your notes by what they say.
+
+Semantic search finds notes by meaning instead of exact words. The engine splits each note into passages, sends them to an embedding model you choose and stores the resulting vectors in the vault. Agents and programs then search with the [`semantic_search`](https://datanotes.dev/reference/operations/queries/#semantic_search) operation. It’s off by default, because note text goes to the model’s provider.
+
+## Turn on semantic search
+
+Turn it on in the engine’s settings file, `settings.json` in its store (`datanotes store path` shows where):
+
+1. Stop the engine.
+2. Pick a provider and a model from the [embedding providers](#choose-an-embedding-provider) table.
+3. Set `semanticEnabled`, `semanticProvider` and `semanticModel` in the settings file.
+4. If the provider needs an API key, [pass the key to the engine](#pass-the-api-key-to-the-engine).
+5. Start the engine.
+
+The settings below use Ollama, which keeps your notes on your computer:
+
+```json
+{
+  "semanticEnabled": true,
+  "semanticProvider": "ollama",
+  "semanticModel": "nomic-embed-text"
+}
+```
+
+Leave `semanticBaseUrl` empty to use the provider’s default address. The [settings reference](https://datanotes.dev/reference/settings/) lists every key and its default.
+
+## Choose an embedding provider
+
+The provider decides where your note text goes. Local providers keep it on your computer; hosted ones send it over the network:
+
+| `semanticProvider` | Where notes go                                         | Default address                | Example model                                                |
+| ------------------ | ------------------------------------------------------ | ------------------------------ | ------------------------------------------------------------ |
+| `ollama`           | Stays on this computer                                 | `http://127.0.0.1:11434`       | `nomic-embed-text`, `qwen3-embedding:0.6b`                   |
+| `lmstudio`         | Stays on this computer                                 | `http://127.0.0.1:1234/v1`     | `text-embedding-nomic-embed-text-v1.5`                       |
+| `openrouter`       | OpenRouter and the model’s provider (needs an API key) | `https://openrouter.ai/api/v1` | `openai/text-embedding-3-small`, `qwen/qwen3-embedding-0.6b` |
+| `openai`           | OpenAI (needs an API key)                              | `https://api.openai.com/v1`    | `text-embedding-3-small`                                     |
+| `custom`           | Any OpenAI-compatible `/embeddings` API                | None: set `semanticBaseUrl`    | The model your API serves                                    |
+
+## Pass the API key to the engine
+
+The engine never stores API keys in the settings file. Set the `DATANOTES_EMBEDDING_KEY` environment variable before you start the engine, or pass a file that holds the key:
+
+```bash
+npx -y datanotes serve --database your_database_folder \
+  --embedding-key-file your_key_file
+```
+
+## Choose which notes to index
+
+By default the engine indexes every note in the vault. Two settings narrow the scope:
+
+* **`semanticScope`**: `"vault"` (the default) indexes every note; `"tables"` indexes only table rows
+* **`semanticExclude`**: folders to leave out
+
+The engine always skips `_archive` folders.
+
+## How the engine builds the index
+
+The engine splits each note into passages of at most about 1,500 characters. The splitting follows these rules:
+
+* **First passage**: holds the title and the front matter, up to 800 characters, without the keys `id` and `table`
+* **Heading sections**: each section starts a new passage
+* **Long sections**: the engine splits them at paragraph boundaries
+* **Limit**: 40 passages per note
+
+The engine sends the passages to the model and stores the vectors as int8 values in `semantic/`, in its store. It updates the index 10s after edits stop, and 5s after it starts. It embeds again only the notes whose content changed. If you change the provider, the address or the model, the engine starts a new index.
+
+## Search from agents and programs
+
+Agents and programs use three operations. All three work through the [HTTP API](https://datanotes.dev/develop/http-api/), the [JavaScript client](https://datanotes.dev/develop/client/) and the [Model Context Protocol (MCP) server](https://datanotes.dev/start/connect-agents/):
+
+* **[`semantic_search`](https://datanotes.dev/reference/operations/queries/#semantic_search)**: takes a `query`, with optional `tables`, `folder`, `min_score` and `limit`; returns one result per note, its best passage, best first
+* **[`describe_semantic_index`](https://datanotes.dev/reference/operations/auxiliary/#describe_semantic_index)**: shows the index status
+* **[`sync_semantic_index`](https://datanotes.dev/reference/operations/auxiliary/#sync_semantic_index)**: brings the index up to date, or rebuilds it with `rebuild: true`
+
+This example asks for the five notes closest in meaning to a question. `db` is a [client](https://datanotes.dev/develop/client/) connected to the engine:
+
+```ts
+const r = await db.call("semantic_search", {
+  query: "films that scare without a single spoken word",
+  limit: 5,
+});
+```
+
+## Search these docs by meaning
+
+With semantic search on, the engine also embeds the sections of these docs, in an index of their own. `search_docs` then merges its ranking by words with the engine’s ranking by meaning. [Search by words and by meaning](https://datanotes.dev/guides/agents/docs-offline/#search-by-words-and-by-meaning) explains how.
+
+## Search notes by meaning
+
+`semantic_search` finds notes by meaning rather than by words. It’s off by default, because it sends note text to an embedding provider. Until you turn it on in the engine’s settings, the call answers that it’s off; [Turn on semantic search](https://datanotes.dev/guides/search/semantic/#turn-on-semantic-search) explains how. `describe_semantic_index` reports the state of the index, and `sync_semantic_index` brings it up to date once it’s on.
+
+**`semantic_search`**
+
+```json
+{ "query": "the Friday we watched the Cary Grant film" }
+```
+
+Refused:
+
+```text
+semantic search is off: set semanticEnabled, semanticProvider and semanticModel in the engine settings
+Hint: Check the arguments of semantic_search: query (string, required), limit (integer), tables (list of strings), folder (string), min_score (number).
+Docs: https://datanotes.dev/reference/operations/queries.md#semantic_search (offline: read_docs("reference/operations/queries#semantic_search"))
+```
+
+**`describe_semantic_index`** (no arguments)
+
+Answer:
+
+```json
+{
+  "enabled": false,
+  "provider": "ollama",
+  "model": "",
+  "files": 0,
+  "passages": 0,
+  "pending": 6,
+  "running": false
+}
+```
+
+**`sync_semantic_index`** (no arguments)
+
+Refused:
+
+```text
+semantic search is off: set semanticEnabled, semanticProvider and semanticModel in the engine settings
+Hint: Check the arguments of sync_semantic_index: rebuild (boolean).
+Docs: https://datanotes.dev/reference/operations/auxiliary.md#sync_semantic_index (offline: read_docs("reference/operations/auxiliary#sync_semantic_index"))
+```

package/dist/docs/guides/search/text.md

@@ -0,0 +1,142 @@
+# Read and search notes
+
+> Read any note in the vault, list folders and search their text.
+
+Not every note is a row, so the operations on this page work on the whole vault: plain notes and rows alike. The vault has three actors, two classic films, a viewing and two plain notes.
+
+## Read a note
+
+`read_note` returns a note’s Markdown in `content`, with its front matter, tags, file dates and size, parsed.
+
+**`read_note`**
+
+```json
+{ "path": "Notes/Film night.md" }
+```
+
+Answer:
+
+```json
+{
+  "path": "Notes/Film night.md",
+  "content": "---\ntags: [films]\n---\n# Film night\n\nEvery other Friday. Next up: [[Charade]]. October’s theme is [[Cary Grant]].\n",
+  "front_matter": { "tags": ["films"] },
+  "tags": ["films"],
+  "modified_at": "<date-time>",
+  "created_at": "<date-time>",
+  "size": 115
+}
+```
+
+## List a folder
+
+`list_folder` lists a folder’s sub-folders and files, as vault paths, and `""` is the vault root. The listing hides the files and folders whose names start with a dot, such as `.obsidian`.
+
+**`list_folder`**
+
+```json
+{ "folder": "" }
+```
+
+Answer:
+
+```json
+{ "folder": "", "folders": ["Notes", "datanotes"], "files": [] }
+```
+
+**`list_folder`**
+
+```json
+{ "folder": "datanotes/main/public" }
+```
+
+Answer:
+
+```json
+{
+  "folder": "datanotes/main/public",
+  "folders": [
+    "datanotes/main/public/actor",
+    "datanotes/main/public/film",
+    "datanotes/main/public/watch"
+  ],
+  "files": []
+}
+```
+
+## Search text
+
+`search_notes` finds notes whose path or text contains every word of the query, ignoring case and accents. It returns the matching passages.
+
+**`search_notes`**
+
+```json
+{ "query": "grant", "context_length": 30 }
+```
+
+Answer (62 lines)
+
+```json
+{
+  "query": "grant",
+  "results": [
+    {
+      "path": "datanotes/main/public/actor/Cary Grant.md",
+      "score": 6,
+      "matches": [
+        {
+          "start": 81,
+          "end": 86,
+          "context": "-afa4-921743d3b309\nname: Cary Grant\n---\n\n"
+        }
+      ]
+    },
+    {
+      "path": "datanotes/main/public/film/Charade.md",
+      "score": 2,
+      "matches": [
+        {
+          "start": 102,
+          "end": 107,
+          "context": "le: Charade\ncast:\n  - \"[[Cary Grant]]\"\n  - \"[[Audrey Hepburn]]\"\n-"
+        },
+        {
+          "start": 147,
+          "end": 152,
+          "context": "[Audrey Hepburn]]\"\n---\n[[Cary Grant]] again, 23 years after [[His"
+        }
+      ]
+    },
+    {
+      "path": "datanotes/main/public/film/His Girl Friday.md",
+      "score": 1,
+      "matches": [
+        {
+          "start": 110,
+          "end": 115,
+          "context": "Girl Friday\ncast:\n  - \"[[Cary Grant]]\"\n  - \"[[Rosalind Russell]]\""
+        }
+      ]
+    },
+    {
+      "path": "Notes/Film night.md",
+      "score": 1,
+      "matches": [
+        {
+          "start": 104,
+          "end": 109,
+          "context": "]]. October’s theme is [[Cary Grant]].\n"
+        }
+      ]
+    },
+    {
+      "path": "Notes/Rewatch ideas.md",
+      "score": 1,
+      "matches": [
+        { "start": 7, "end": 12, "context": "A Cary Grant weekend: every film with him," }
+      ]
+    }
+  ],
+  "total": 5
+}
+```

package/dist/docs/guides/security/security-model.md

@@ -0,0 +1,110 @@
+# Security model
+
+> Security model, credentials, and how to report a vulnerability.
+
+Datanotes is an engine that reads and writes a folder of Markdown notes, the vault. It runs on your own machine, for you and the agents you choose. This page explains:
+
+* How to report a vulnerability
+* Which versions get security fixes
+* What the engine exposes on the network
+* What each token can do
+* How the engine guards your files and its store
+* How to limit what an agent can do
+* Which features send data off your machine
+
+## Report a vulnerability
+
+Report vulnerabilities only through GitHub Private Vulnerability Reporting. Open a [new private advisory](https://github.com/davidefornelli/datanotes/security/advisories/new) on the repository. It’s the one channel for security reports. Don’t report them in public issues or discussions. You’ll get an answer within a week.
+
+Include these details in your report:
+
+* The Datanotes version
+* How to reproduce the problem
+* What an attacker gains
+
+## Supported versions
+
+Security fixes go to the latest release of Datanotes 3.
+
+| Version | Supported            |
+| ------- | -------------------- |
+| 3.x     | Yes                  |
+| 2.x     | No: it was a preview |
+| 1.x     | No: it was a preview |
+
+Update to the latest 3.x release to get a fix.
+
+## What the engine exposes on the network
+
+The engine’s HTTP server listens on `127.0.0.1`, so only your machine can reach it, unless you start it with `--host`. It answers no Cross-Origin Resource Sharing (CORS) preflight, so web pages can’t call it.
+
+Three routes answer without a token:
+
+* **`GET /`**: the status and the version
+* **`GET /datanotes/openapi.json`**: the API’s description
+* **`/datanotes/docs`**: an interactive console over that description, with its script and stylesheet
+
+These routes contain no data from your vault: the website publishes the same description. The console’s page loads nothing from other hosts, because its Content Security Policy (CSP) allows the same origin only. Its calls need a token, which the browser keeps in local storage until you log out.
+
+To reach the engine from another device, such as a phone or a hosted assistant, publish it through a reverse proxy or a tunnel, over HTTPS. Give each device or agent its own named token. Publish the Model Context Protocol (MCP) server, `datanotes mcp serve`, the same way: it holds no credentials and passes each agent’s token on to the engine.
+
+## Credentials and what they can do
+
+The engine accepts two kinds of token, and the owner token can do everything a named token can. Prefer a named token per device, program or agent: it has only the permissions it needs, and the journal names it. MCP clients use the same tokens, because the MCP server passes them to the engine.
+
+### Owner token
+
+The owner token is `httpServerToken` in the settings file. It can do everything: read and write data, change the schema, manage tokens and webhooks, undo, and read the engine’s status. `datanotes status` reads it from the settings file. Keep it on the machine.
+
+### Named token
+
+The owner creates a named token with `POST /datanotes/tokens/create/`, and the engine stores only its hash, in `tokens.json` in its store, outside the vault. A named token has these permissions:
+
+* **Read**: everything
+* **Write rows**: only the tables in its `write` patterns
+* **Change the schema**: only with `ddl`
+* **Maintain**: with `maintain`, it may also undo, restore, vacuum, configure the database and rebuild the semantic index; use it for an agent’s own token
+
+A named token can’t create webhooks, manage tokens or read the engine’s status, and without `maintain` it can’t undo.
+
+### Who made each write
+
+The engine journals every write with its actor. The client sends the agent’s name, model and session, so they’re self-reported. The server sets the credential, `auth` (`owner`, or the named token’s name), and no client can claim another one.
+
+## File access inside the vault
+
+The engine resolves every path inside the vault. Both its operations and its file host refuse a path that contains `..`, a backslash, a drive letter or a NUL character.
+
+The engine follows symbolic links inside the vault. Don’t link to folders you wouldn’t give the engine.
+
+## A private store
+
+The engine’s store holds its secrets: the owner token in `settings.json` and the webhooks’ secrets in `webhooks.json`. It also holds the journal and the hashes of the named tokens. On Linux and macOS, the engine keeps the store private to the user that runs it:
+
+* **Folders**: the data home and each store are created with mode `0700`
+* **Files**: `settings.json`, `webhooks.json` and the other files of the store are created with mode `0600`
+* **At start**: the engine tightens a data home, a store, a settings file or a `webhooks.json` that an older version left readable by others
+
+On Windows, the store lives in your user profile, which other users can’t read. The Docker image runs the engine with `umask 077`, so every file it creates is private too.
+
+## Agents and prompt injection
+
+An agent reads your notes, and a note can hold text written to manipulate it: a prompt injection. A web page you clipped or an email you saved can tell the agent to delete rows or to send your data out. The engine can’t tell such text from your own notes.
+
+Limit what each agent can do:
+
+* **Give it a named token**: grant only the tables it writes, and leave out `ddl` and `maintain` unless it needs them
+* **Give it only the tools it needs**: start the MCP server with `datanotes mcp --profile`, `--tools` or `--except`
+* **Keep the webhook tools off**: they’re off by default, because a webhook sends change events to any URL
+
+Every write is journaled with the agent’s name, so you can review it and undo it.
+
+## Data that leaves your machine
+
+Nothing leaves your machine unless you turn it on. These features send data out:
+
+* **Semantic search**: sends note text to the embedding provider you configure. It’s off by default (`semanticEnabled`), and you choose the provider with `semanticProvider` and `semanticBaseUrl`. A local provider, such as Ollama or LM Studio, keeps the text on your machine.
+* **Telemetry**: sends logs to the OpenTelemetry Protocol (OTLP) endpoint you configure. It’s off by default (`telemetryEnabled`). The logs contain operations, routes, note paths, actors, tool names and error messages, which may quote a value, but not note bodies. The settings file stores `telemetryHeaders`, which may hold an API key; set `OTEL_EXPORTER_OTLP_HEADERS` instead to keep the key out of it.
+* **Webhooks**: the owner creates them, and the engine posts change events to the URLs they name.
+* **Git sync**: with `--git-sync`, the engine pushes the vault to its git remote.
+* **Store backup**: it’s off by default (`storeBackupEnabled`). When on, the engine pushes its store to the git remote you configure. The backup holds the journal, the transaction logs, the blobs, the change feed and the named tokens’ hashes. The journal, the logs and the feed quote your notes’ values. The backup leaves out `settings.json` and `webhooks.json`, which hold the secrets. Use a private repository.