datanotes 2.0.1 → 3.0.0

package/NOTICE

@@ -2,3 +2,12 @@ Datanotes
 Copyright 2026 Davide Fornelli
 
 This product is licensed under the Apache License, Version 2.0 (see LICENSE).
+
+The `datanotes` command (dist/datanotes.cjs) bundles third-party software, each under its own
+license: dist/THIRD-PARTY-NOTICES.txt lists every package with its license and copyright.
+
+It includes Swagger UI (https://github.com/swagger-api/swagger-ui), licensed under the Apache
+License, Version 2.0, whose NOTICE reads:
+
+  swagger-ui
+  Copyright 2020-2021 SmartBear Software Inc.

package/README.md

@@ -1,53 +1,232 @@
 # Datanotes
 
-The engine of your second brain: Markdown notes as database tables, for you and your AI agents.
+Datanotes is the open-source database engine for Markdown: it runs a folder of notes as typed, validated tables that your programs and AI agents query, write and undo over MCP or HTTP, while every row stays a plain file you own.
 
-A row is any note whose `table` property names a table; tables are defined in
-`Database/database.yaml`. Every write is checked against its table, journaled and undoable.
-Programs use its HTTP API or the `datanotes/client` library; agents use the separate MCP server
-`datanotes-mcp`.
+[![npm version](https://img.shields.io/npm/v/datanotes)](https://www.npmjs.com/package/datanotes)
+[![CI](https://github.com/davidefornelli/datanotes/actions/workflows/ci.yml/badge.svg)](https://github.com/davidefornelli/datanotes/actions/workflows/ci.yml)
+[![License: Apache-2.0](https://img.shields.io/badge/license-Apache--2.0-blue)](LICENSE)
+
+<!-- Hero GIF goes here: an agent writes a row over MCP, the engine refuses a bad value and explains the fix, then the note opens as plain Markdown. -->
 
 **Documentation: [datanotes.dev](https://datanotes.dev)**
 
+## Why Datanotes
+
+- **Agents can't write garbage.** Every write is checked against its table: types, links, unique values and CHECK constraints. A refused write changes nothing. Its answer says which value broke which rule and how to fix it.
+- **Every change can be undone.** The engine journals every write it makes. It records who made it: the agent, its session and its token. `undo_migration` takes any change back.
+- **It's still just Markdown.** A row is a note, and its front matter keys are the columns. Read it, edit it, grep it or commit it to Git. The engine reads your hand edits and checks them too.
+- **Local first.** Datanotes is Apache-2.0 and runs on your machine. Nothing leaves it unless you turn that feature on.
+
+## When not to use it
+
+Datanotes keeps every row as a file, so it isn't a replacement for SQLite or Postgres. Choose a classic database for:
+
+- Large analytical datasets
+- High write throughput
+- Many concurrent writers on one folder
+
+Datanotes is comfortable at 10,000 notes. At 50,000 notes, the engine takes 22 to 115 seconds to start.
+
 ## Quick start
 
-Requires Node.js 20 or newer. Start the engine on a vault:
+You need Node.js 22 or newer. On Windows, use Node.js 24.16 or newer, or 26.1 or newer.
+
+Start the engine on a folder, which becomes your database. It can be empty, or hold notes already:
 
 ```bash
-npx -y datanotes serve --vault /path/to/vault
+npx -y datanotes serve --database your_database_folder
 ```
 
-It serves the HTTP API on `http://127.0.0.1:27150` (routes under `/datanotes/*`, plus the
-Local-REST-API-style `/vault/*` and `/search/simple/`), with the owner's token in
-`<vault>/.datanotes/service.json`. From code:
+[Run Datanotes on your notes](https://datanotes.dev/start/quick-start/) walks you through your first table.
 
-```js
-import { DatanotesClient } from "datanotes/client";
+## Connect your agents
 
-const db = new DatanotesClient({ url: "http://127.0.0.1:27150", token });
-const { value } = await db.call("select_rows", { table: "book", where: { pages: { gt: 300 } } });
+Agents reach the engine through its Model Context Protocol (MCP) server, `datanotes mcp`. It uses the engine that runs on the folder, or starts one. Every client runs the same command:
+
+```bash
+npx -y datanotes mcp --database your_database_folder
 ```
 
-AI agents connect through [`datanotes-mcp`](https://www.npmjs.com/package/datanotes-mcp), the MCP
-server of Datanotes, a separate package built on this API.
+<details>
+<summary>Claude Code</summary>
 
-## Learn more
+```bash
+claude mcp add datanotes -- \
+  npx -y datanotes mcp --database your_database_folder
+```
+
+</details>
+
+<details>
+<summary>Claude Desktop</summary>
+
+Add the server to `claude_desktop_config.json`:
+
+```json
+{
+  "mcpServers": {
+    "datanotes": {
+      "command": "npx",
+      "args": [
+        "-y", "datanotes", "mcp", "--database", "your_database_folder"
+      ]
+    }
+  }
+}
+```
+
+</details>
+
+<details>
+<summary>Cursor</summary>
 
-- [How it works](https://datanotes.dev/concepts/how-it-works/)
-- [Operations](https://datanotes.dev/reference/tools/)
-- [Connect agents (datanotes-mcp)](https://datanotes.dev/start/connect-agents/)
-- [JavaScript client](https://datanotes.dev/guides/client/)
-- [Apps](https://datanotes.dev/guides/apps/)
-- [Obsidian client](https://datanotes.dev/guides/obsidian/)
+Add the server to `.cursor/mcp.json` in your project, or to `~/.cursor/mcp.json`:
 
-## Security
+```json
+{
+  "mcpServers": {
+    "datanotes": {
+      "command": "npx",
+      "args": [
+        "-y", "datanotes", "mcp", "--database", "your_database_folder"
+      ]
+    }
+  }
+}
+```
+
+</details>
+
+<details>
+<summary>VS Code</summary>
+
+Add the server to `.vscode/mcp.json` in your workspace:
+
+```json
+{
+  "servers": {
+    "datanotes": {
+      "type": "stdio",
+      "command": "npx",
+      "args": [
+        "-y", "datanotes", "mcp", "--database", "your_database_folder"
+      ]
+    }
+  }
+}
+```
 
-See [SECURITY.md](SECURITY.md): how to report a vulnerability, what each credential may do, and
-which data leaves the machine (only what you turn on).
+</details>
 
-## Contributing
+<details>
+<summary>Codex</summary>
+
+Add the server to `~/.codex/config.toml`:
+
+```toml
+[mcp_servers.datanotes]
+command = "npx"
+args = ["-y", "datanotes", "mcp", "--database", "your_database_folder"]
+```
+
+</details>
+
+<details>
+<summary>Windows</summary>
+
+On Windows, clients start `npx` through `cmd /c`. For Claude Code, outside Windows Subsystem for Linux (WSL):
+
+```bash
+claude mcp add datanotes -- cmd /c npx -y datanotes mcp --database C:\Notes
+```
+
+In a JSON file, the command is `cmd` and the backslashes are escaped:
+
+```json
+{
+  "mcpServers": {
+    "datanotes": {
+      "command": "cmd",
+      "args": [
+        "/c", "npx", "-y", "datanotes", "mcp", "--database", "C:\\Notes"
+      ]
+    }
+  }
+}
+```
+
+</details>
+
+[Connect your agents](https://datanotes.dev/start/connect-agents/) covers HTTP transport and named tokens.
+
+## Use it from TypeScript
+
+Install the package with `npm install datanotes`. This script finds the engine on your folder, creates a table and queries it:
+
+```ts
+import { DatanotesClient } from "datanotes/client";
+import { localEngine } from "datanotes/local";
+
+const engine = localEngine("your_database_folder");
+if (!engine) throw new Error("Start the engine on the folder first");
+const db = new DatanotesClient({ url: engine.url, token: engine.token });
+
+const created = await db.call("create_table", {
+  name: "film",
+  description: "Classic films I watched or want to watch",
+  filename: "{title}",
+  columns: [
+    {
+      name: "title",
+      type: "string",
+      required: true,
+      unique: true,
+      description: "The film's title, as on its poster",
+    },
+    {
+      name: "minutes",
+      type: "integer",
+      min: 1,
+      description: "Running time in minutes",
+    },
+  ],
+});
+if (!created.ok) throw new Error(String(created.value));
+
+await db.call("insert_row", {
+  table: "film",
+  values: { title: "Nosferatu", minutes: 94 },
+});
+
+// Refused: nothing is written, and the answer explains why
+const bad = await db.call("insert_row", {
+  table: "film",
+  values: { title: "Metropolis", minutes: 0 },
+});
+if (!bad.ok) console.log(bad.value);
+
+const films = await db.call("select_rows", {
+  table: "film",
+  where: { minutes: { gt: 90 } },
+});
+console.log(films.value);
+```
+
+Each film is now a note, `main/public/film/Nosferatu.md`, with the values in its front matter. The [JavaScript client](https://datanotes.dev/develop/client/) page covers the rest of the client.
+
+## Stability: Datanotes 3 stays 3
+
+Every 3.x version adds to what you use and never takes it away. Operations, arguments, answer fields, routes, commands and settings keep working the same way. [Compatibility](https://datanotes.dev/reference/compatibility/) explains the promise and how the tests enforce it.
+
+## Learn more
 
-See [CONTRIBUTING.md](CONTRIBUTING.md).
+- [Documentation](https://datanotes.dev)
+- [How it works](https://datanotes.dev/start/how-it-works/)
+- [Operations](https://datanotes.dev/reference/operations/)
+- [Use Datanotes with Obsidian](https://datanotes.dev/administration/use-with-obsidian/)
+- [Contributing](CONTRIBUTING.md)
+- [Security](SECURITY.md)
+- [License](LICENSE)
 
 ## License
 

package/SECURITY.md

@@ -1,61 +1,108 @@
 # Security
 
-## Reporting a vulnerability
+Datanotes is an engine that reads and writes a folder of Markdown notes, the vault. It runs on your own machine, for you and the agents you choose. This page explains:
 
-Please report vulnerabilities privately, by email to the maintainer listed on the npm package
-page ([npmjs.com/package/datanotes](https://www.npmjs.com/package/datanotes)), not in public. You will get an answer within a week.
-Include the version, how to reproduce it and what an attacker gains.
+- How to report a vulnerability
+- Which versions get security fixes
+- What the engine exposes on the network
+- What each token can do
+- How the engine guards your files and its store
+- How to limit what an agent can do
+- Which features send data off your machine
 
-## Security model
+## Report a vulnerability
 
-Datanotes is an engine that reads and writes a folder of Markdown notes (the vault). It is meant to
-run on your own machine, for you and the agents you choose.
+Report vulnerabilities only through GitHub Private Vulnerability Reporting. Open a [new private advisory](https://github.com/davidefornelli/datanotes/security/advisories/new) on the repository. It’s the one channel for security reports. Don’t report them in public issues or discussions. You’ll get an answer within a week.
 
-### Network
+Include these details in your report:
 
-- The HTTP server listens on `127.0.0.1` unless told otherwise (`--host`). It answers no CORS
-  preflight, so web pages cannot call it.
-- Only `GET /` (status and version), `GET /datanotes/openapi.json` (the API's description) and
-  `/datanotes/docs` (an interactive console over it, with its script and stylesheet) answer without
-  a token. They contain no data from the vault: the same description is published on the website.
-  The console's page loads nothing from other hosts (Content-Security-Policy: same origin only);
-  its calls need a token, which it keeps in the browser's local storage until you log out.
-- To reach it from outside (a phone, a hosted assistant), publish it through a reverse proxy or a
-  tunnel, over HTTPS, and give each device or agent its own app token. Publish the MCP server
-  (`datanotes-mcp serve`) the same way: it holds no credentials and passes each agent's token on.
+- The Datanotes version
+- How to reproduce the problem
+- What an attacker gains
 
-### Credentials
+## Supported versions
 
-| Credential | Where | What it can do |
-| --- | --- | --- |
-| Owner token | `httpServerToken` in the settings | Everything: data, schema, apps, webhooks, undo. Keep it on the machine. |
-| App token | created with `POST /datanotes/apps/create/` (owner only), stored hashed | Read everything; write only the tables in its `write` patterns; change the schema only with `ddl`. It cannot create webhooks, manage apps or undo; with `maintain` it may also undo, restore, vacuum, configure and rebuild the semantic index (an agent's own token). |
+Security fixes go to the latest release of Datanotes 3.
 
-Prefer an **app token per device or agent**: it has only the permissions it needs and the journal
-names it. MCP clients use the same tokens: the MCP server passes them to the engine.
+| Version | Supported |
+| --- | --- |
+| 3.x | Yes |
+| 2.x | No: it was a preview |
+| 1.x | No: it was a preview |
 
-Every write is journaled with its actor. The agent's name, model and session come from the client
-and are self-reported; the credential (`auth`: `owner` or `app`, and the app's name) is set
-by the server and cannot be claimed.
+Update to the latest 3.x release to get a fix.
 
-### Files
+## What the engine exposes on the network
 
-Every path is resolved inside the vault: a path with `..`, a backslash, a drive letter or a NUL is
-refused, both by the engine's operations and by the file host. Symbolic links inside the vault are
-followed: do not link to folders you would not give the engine.
+The engine’s HTTP server listens on `127.0.0.1`, so only your machine can reach it, unless you start it with `--host`. It answers no Cross-Origin Resource Sharing (CORS) preflight, so web pages can’t call it.
 
-### Data that leaves the machine
+Three routes answer without a token:
 
-Nothing leaves the machine unless you turn it on:
+- **`GET /`**: the status and the version
+- **`GET /datanotes/openapi.json`**: the API’s description
+- **`/datanotes/docs`**: an interactive console over that description, with its script and stylesheet
 
-- **Semantic search** sends note text to the embedding provider you configure (`semanticEnabled`,
-  off by default; provider `semanticProvider` / `semanticBaseUrl`). A local provider (Ollama,
-  LM Studio) keeps it on the machine.
-- **Telemetry** sends logs to the OTLP endpoint you configure (`telemetryEnabled`, off by default):
-  operations, routes, note paths, actors, tool names and error messages (which may quote a value),
-  not note bodies. `telemetryHeaders` (an API key) is
-  stored in the settings file; `OTEL_EXPORTER_OTLP_HEADERS` keeps it out.
-- **Webhooks** (created by the owner) post change events to the URLs they name.
-- **Apps' MCP endpoints** (registered by the owner): the engine sends each app's `mcp_token`, and
-  the arguments and caller of every call of its tools, to the app's `mcp_url`, which may be remote.
-- **Git sync** (`--git-sync`) pushes the vault to its git remote.
+These routes contain no data from your vault: the website publishes the same description. The console’s page loads nothing from other hosts, because its Content Security Policy (CSP) allows the same origin only. Its calls need a token, which the browser keeps in local storage until you log out.
+
+To reach the engine from another device, such as a phone or a hosted assistant, publish it through a reverse proxy or a tunnel, over HTTPS. Give each device or agent its own named token. Publish the Model Context Protocol (MCP) server, `datanotes mcp serve`, the same way: it holds no credentials and passes each agent’s token on to the engine.
+
+## Credentials and what they can do
+
+The engine accepts two kinds of token, and the owner token can do everything a named token can. Prefer a named token per device, program or agent: it has only the permissions it needs, and the journal names it. MCP clients use the same tokens, because the MCP server passes them to the engine.
+
+### Owner token
+
+The owner token is `httpServerToken` in the settings file. It can do everything: read and write data, change the schema, manage tokens and webhooks, undo, and read the engine’s status. `datanotes status` reads it from the settings file. Keep it on the machine.
+
+### Named token
+
+The owner creates a named token with `POST /datanotes/tokens/create/`, and the engine stores only its hash, in `tokens.json` in its store, outside the vault. A named token has these permissions:
+
+- **Read**: everything
+- **Write rows**: only the tables in its `write` patterns
+- **Change the schema**: only with `ddl`
+- **Maintain**: with `maintain`, it may also undo, restore, vacuum, configure the database and rebuild the semantic index; use it for an agent’s own token
+
+A named token can’t create webhooks, manage tokens or read the engine’s status, and without `maintain` it can’t undo.
+
+### Who made each write
+
+The engine journals every write with its actor. The client sends the agent’s name, model and session, so they’re self-reported. The server sets the credential, `auth` (`owner`, or the named token’s name), and no client can claim another one.
+
+## File access inside the vault
+
+The engine resolves every path inside the vault. Both its operations and its file host refuse a path that contains `..`, a backslash, a drive letter or a NUL character.
+
+The engine follows symbolic links inside the vault. Don’t link to folders you wouldn’t give the engine.
+
+## A private store
+
+The engine’s store holds its secrets: the owner token in `settings.json` and the webhooks’ secrets in `webhooks.json`. It also holds the journal and the hashes of the named tokens. On Linux and macOS, the engine keeps the store private to the user that runs it:
+
+- **Folders**: the data home and each store are created with mode `0700`
+- **Files**: `settings.json`, `webhooks.json` and the other files of the store are created with mode `0600`
+- **At start**: the engine tightens a data home, a store, a settings file or a `webhooks.json` that an older version left readable by others
+
+On Windows, the store lives in your user profile, which other users can’t read. The Docker image runs the engine with `umask 077`, so every file it creates is private too.
+
+## Agents and prompt injection
+
+An agent reads your notes, and a note can hold text written to manipulate it: a prompt injection. A web page you clipped or an email you saved can tell the agent to delete rows or to send your data out. The engine can’t tell such text from your own notes.
+
+Limit what each agent can do:
+
+- **Give it a named token**: grant only the tables it writes, and leave out `ddl` and `maintain` unless it needs them
+- **Give it only the tools it needs**: start the MCP server with `datanotes mcp --profile`, `--tools` or `--except`
+- **Keep the webhook tools off**: they’re off by default, because a webhook sends change events to any URL
+
+Every write is journaled with the agent’s name, so you can review it and undo it.
+
+## Data that leaves your machine
+
+Nothing leaves your machine unless you turn it on. These features send data out:
+
+- **Semantic search**: sends note text to the embedding provider you configure. It’s off by default (`semanticEnabled`), and you choose the provider with `semanticProvider` and `semanticBaseUrl`. A local provider, such as Ollama or LM Studio, keeps the text on your machine.
+- **Telemetry**: sends logs to the OpenTelemetry Protocol (OTLP) endpoint you configure. It’s off by default (`telemetryEnabled`). The logs contain operations, routes, note paths, actors, tool names and error messages, which may quote a value, but not note bodies. The settings file stores `telemetryHeaders`, which may hold an API key; set `OTEL_EXPORTER_OTLP_HEADERS` instead to keep the key out of it.
+- **Webhooks**: the owner creates them, and the engine posts change events to the URLs they name.
+- **Git sync**: with `--git-sync`, the engine pushes the vault to its git remote.
+- **Store backup**: it’s off by default (`storeBackupEnabled`). When on, the engine pushes its store to the git remote you configure. The backup holds the journal, the transaction logs, the blobs, the change feed and the named tokens’ hashes. The journal, the logs and the feed quote your notes’ values. The backup leaves out `settings.json` and `webhooks.json`, which hold the secrets. Use a private repository.