datanotes 2.0.1 → 3.0.0

package/dist/docs/guides/agents/reading-errors.md

@@ -0,0 +1,112 @@
+# Read the engine’s errors
+
+> Every error of the engine says what is wrong, how to fix it and where to read more: the shape of an error, the mistakes agents make most, and the operations that look up the right answer.
+
+Every error the engine returns answers three questions, so an agent can correct its call without guessing:
+
+1. **What is wrong**: the column, value, rule or argument the engine refused
+2. **How to fix it**: the allowed values, the closest real name, the right shape, or the operation that looks up the answer
+3. **Where to read more**: a page of this site in Markdown, with an anchor on the section that shows the fix
+
+This page shows how an error looks, how the engine finds the fix in your data, and the mistakes agents make most often.
+
+## Every error has a message, a hint and a docs link
+
+Through the Model Context Protocol (MCP) server, `datanotes mcp`, a failed call comes back as a tool error. Its text ends with two lines, `Hint:` and `Docs:`. An agent that writes a film with the status `seen` reads:
+
+```text
+insert_row refused — nothing was written:
+invalid datanotes/main/public/film/Nosferatu.md: status must be one of: to watch (on the list, not seen yet); watched (seen at least once)
+Hint: Use one of the allowed values listed in the message, written exactly; describe_table shows what each one means. To add a value, alter_column the enum. Then call insert_row again (dry_run: true checks without writing).
+Docs: read_docs("guides/database-objects/columns#describe-what-each-enum-value-means")
+```
+
+Over HTTP, the same error is JSON, with `hint` and `docs` next to `error`:
+
+```json
+{
+  "error": "Table not found: flim",
+  "hint": "Did you mean \"film\"? Tables: film. list_tables lists them …",
+  "docs": "https://datanotes.dev/guides/database-objects/tables.md#find-the-table-that-matches-a-request"
+}
+```
+
+A refused write, with status 422, keeps its usual body: the operation’s result with `ok: false` and the reasons. The engine adds `hint` and `docs` to it. In code, `DatanotesClient` appends both lines to two texts:
+
+* **`call`**: the `value` of a failed call
+* **`post`**: the message of the `DatanotesError` it throws, which also has `hint` and `docs` as fields
+
+The `docs` link always points to a `.md` page, the Markdown version of a page of this site. Agents read Markdown best. The reference of an operation is the page of its kind, followed by an anchor with the operation’s name, such as `https://datanotes.dev/reference/operations/rows.md#insert_row`.
+
+The `Docs:` line of the text names the page in two ways:
+
+* **`datanotes mcp`**: only the `read_docs` call, which reads the page from the docs of the installed version. Its agents always have the docs tools, so they need no internet access and no web address.
+* **`DatanotesClient` and `datanotes/mcp`**: the address, then the same `read_docs` call in parentheses. Your own MCP server may not serve the docs tools, so its agents keep the address. A server that serves them passes each result to `offlineDocs`, which keeps only the call. See [Build on Datanotes](https://datanotes.dev/develop/build-on-datanotes/).
+
+[Read the docs offline](https://datanotes.dev/guides/agents/docs-offline/) describes the docs tools.
+
+## The engine finds the fix in your data
+
+When the answer is data, the hint reads it from your database:
+
+* **A table or view that doesn’t exist**: the closest names, by spelling, by case and by the table’s `aliases`. `movies` leads to `film`, and a short database lists every table and view.
+* **A column that doesn’t exist**: the closest column and the table’s columns. This works wherever a call names a column: `values`, `where`, `columns`, `order_by`, `join`, `group_by` and `aggregates`.
+* **A value outside an enum**: the allowed values, each with its meaning from `value_descriptions`.
+* **A wrong operator in a filter**: the operator you meant, from a typo (`gtee`) or another language’s word (`>=`, `like`, `not_in`, `is_null`). Then every operator.
+* **A link to a row that doesn’t exist**: `list_link_targets`, which lists the rows the link column accepts.
+* **A row that doesn’t exist**: the row with the closest note name, and how to find a row’s `path` or `id`.
+* **A write to a view**: the view’s own table, since you can only read a view.
+
+## The MCP server checks a call before the engine runs it
+
+`datanotes mcp` checks a call against the tool’s input schema before it sends anything. It catches four mistakes:
+
+* A tool that doesn’t exist
+* An argument the tool doesn’t take
+* A required argument that is missing
+* A value of the wrong type
+
+Each comes back as a tool error that lists the tool’s arguments with their types and links the tool’s reference:
+
+```text
+select_rows: unknown argument "wehre"
+Hint: For "wehre" use "where". select_rows takes: table (string, required), columns (list of strings), where (object), …
+Docs: read_docs("reference/operations/queries#select_rows")
+```
+
+Your own MCP server gets the same checks from `callEngineTool` in `datanotes/mcp`. A program checks arguments with `checkArguments` from `datanotes/client`.
+
+## Fix the mistakes agents make most
+
+| Mistake                                    | What the agent reads                                 | The fix                                                |
+| ------------------------------------------ | ---------------------------------------------------- | ------------------------------------------------------ |
+| A misspelled table, `flim`                 | `Table not found: flim` and `Did you mean "film"?`   | Use the name given, or call `list_tables`              |
+| A misspelled column, `titel`               | `unknown column "titel"` and `Did you mean "title"?` | Use the name given; `describe_table` lists the columns |
+| A value outside an enum, `seen`            | The allowed values and their meanings                | Pick one, or ask the user which one they meant         |
+| A wrong operator, `">="`                   | `For ">=" use "gte"` and every operator              | Write `{"rating": {"gte": 4}}`                         |
+| A value of the wrong type                  | `minutes expected a number`                          | Send numbers unquoted and dates as `YYYY-MM-DD`        |
+| A row the table’s CHECK refuses            | `violates CHECK rated_when_watched` and the rule     | Change the values so the rule holds                    |
+| A row path that doesn’t exist              | `film has no row at …` and the closest row           | Use the path given, or find it with `select_rows`      |
+| An argument the tool doesn’t take, `wehre` | `For "wehre" use "where"` and the tool’s arguments   | Rename the argument                                    |
+| A write to a view                          | The view’s table                                     | Write to the table, and read the view                  |
+| An undo blocked by later changes           | Which notes changed since                            | Undo the later writes first, newest first              |
+
+## Look up the answer before you write
+
+The hints name these operations because each one answers a whole class of mistakes:
+
+* **`list_tables`**: every table and view, with descriptions and aliases
+* **`describe_table`**: a table’s columns, types, limits, enum values with their meanings, and CHECK constraints
+* **`describe_database`**: catalogs, schemas, bases and the database’s settings
+* **`list_link_targets`**: the rows a link column may point to, filtered by its `ref_where` and `ref_by`
+* **`check_table`**: the rows that break the schema, such as notes edited by hand
+* **`list_migrations`** and **`describe_history`**: the writes to undo, and a table’s versions
+
+A refused write changes nothing, so the agent can fix the call and send it again. To check a write without making it, pass `dry_run: true`. To make several writes succeed or fail together, send them in one `batch`.
+
+## Learn more about operations and refusals
+
+* [How agents work with your vault](https://datanotes.dev/guides/agents/how-agents-use-datanotes/): what an agent reads first, and how the journal records its writes
+* [Operations and MCP tools](https://datanotes.dev/reference/operations/): every operation with its arguments
+* [See how the engine refuses bad values](https://datanotes.dev/guides/data-quality/validation/#see-how-the-engine-refuses-bad-values): refusals on a real engine
+* [Read errors over HTTP](https://datanotes.dev/develop/http-api/#read-errors): the status codes of the HTTP API

package/dist/docs/guides/data-quality/check-constraints.md

@@ -0,0 +1,379 @@
+# CHECK constraints
+
+> Write rules that every row of a table must follow, including rules that follow links into other tables.
+
+A CHECK constraint is a rule written as a `where` filter, such as “a watched film has a rating”. Every row of the table must pass it: the engine refuses a write that breaks it, and refuses to add a constraint that rows already break. A rule can follow links into other tables.
+
+## Enforce rules with CHECK constraints
+
+`add_constraint` adds a CHECK constraint, a rule every row must satisfy, written as a `where` filter. The engine refuses it while existing rows break the rule and lists them; after that, it enforces the rule on every write. A condition on a column the row leaves empty passes, as in Structured Query Language (SQL). Inside `or`, `and` and `not`, the engine tests an empty value as it is, so a watched film without a `rating` breaks `rated_when_watched`.
+
+**`add_constraint`**
+
+```json
+{
+  "table": "film",
+  "name": "rated_when_watched",
+  "check": { "or": [{ "status": { "ne": "watched" } }, { "rating": { "exists": true } }] }
+}
+```
+
+Answer:
+
+```json
+{
+  "ok": true,
+  "op": "add_constraint",
+  "planned_texts": ["datanotes/metastore.yaml"],
+  "commits": [
+    { "table": "diary.classics.film", "version": 8, "operation": "ADD CONSTRAINT" }
+  ],
+  "migration_id": "20261012T072004-add_constraint-add-constraint-rated_when_watc"
+}
+```
+
+**`add_constraint`**
+
+```json
+{ "table": "film", "name": "at_most_three_hours", "check": { "minutes": { "lte": 180 } } }
+```
+
+**`insert_row`**
+
+```json
+{
+  "table": "film",
+  "values": {
+    "title": "His Girl Friday",
+    "year": 1940,
+    "minutes": 92,
+    "status": "watched"
+  }
+}
+```
+
+Refused:
+
+```text
+insert_row refused — nothing was written:
+invalid datanotes/diary/classics/film/His Girl Friday.md: rated_when_watched violates CHECK rated_when_watched: {"or":[{"status":{"ne":"watched"}},{"rating":{"exists":true}}]}
+Hint: The row breaks a CHECK constraint of the table (named in the message, with its rule): change the values so the rule holds. describe_table lists the table's constraints. Then call insert_row again (dry_run: true checks without writing).
+Docs: https://datanotes.dev/guides/data-quality/check-constraints.md#enforce-rules-with-check-constraints (offline: read_docs("guides/data-quality/check-constraints#enforce-rules-with-check-constraints"))
+```
+
+**`insert_row`**
+
+```json
+{
+  "table": "film",
+  "values": {
+    "title": "His Girl Friday",
+    "year": 1940,
+    "minutes": 92,
+    "status": "watched",
+    "rating": 5
+  }
+}
+```
+
+Answer:
+
+```json
+{
+  "ok": true,
+  "op": "insert_row",
+  "files_created": 1,
+  "planned_creates": ["datanotes/diary/classics/film/His Girl Friday.md"],
+  "commits": [{ "table": "diary.classics.film", "version": 10, "operation": "WRITE" }],
+  "migration_id": "20261012T072004-insert_row-insert-into-film"
+}
+```
+
+`drop_constraint` removes a constraint, here the three-hour limit, which a long restoration could break:
+
+**`drop_constraint`**
+
+```json
+{ "table": "film", "name": "at_most_three_hours" }
+```
+
+Answer:
+
+```json
+{
+  "ok": true,
+  "op": "drop_constraint",
+  "planned_texts": ["datanotes/metastore.yaml"],
+  "commits": [
+    { "table": "diary.classics.film", "version": 11, "operation": "DROP CONSTRAINT" }
+  ],
+  "migration_id": "20261012T072004-drop_constraint-drop-constraint-at_most_three_"
+}
+```
+
+## Write a CHECK rule that follows links
+
+A CHECK constraint is a rule every row must satisfy, and you add one with `add_constraint`. It can follow a link the way a query does: `parent.kind` is the `kind` of the row that `parent` names. Here a genre’s parent must be an era, an era’s parent must be Cinema, and Cinema has no parent.
+
+A rule treats links as follows:
+
+* **Empty link**: passes at the top level of a rule and fails inside `or`, `and` and `not`; on a longer path such as `parent.parent.kind`, the rule skips only an empty first link
+* **Link to a row that doesn’t exist**: fails, so it passes inside `not`
+* **List of links**: every linked row must match
+* **Resolution**: links resolve as in the vault, by path from the row’s note
+
+**`add_constraint`**
+
+```json
+{
+  "table": "genre",
+  "name": "parent_by_level",
+  "check": {
+    "or": [
+      { "kind": "cinema", "parent": { "exists": false } },
+      { "kind": "era", "parent.kind": "cinema" },
+      { "kind": "genre", "parent.kind": "era" }
+    ]
+  }
+}
+```
+
+Answer:
+
+```json
+{
+  "ok": true,
+  "op": "add_constraint",
+  "planned_texts": ["datanotes/metastore.yaml"],
+  "commits": [
+    { "table": "main.public.genre", "version": 4, "operation": "ADD CONSTRAINT" }
+  ],
+  "migration_id": "20261012T071949-add_constraint-add-constraint-parent_by_level"
+}
+```
+
+The engine refuses a genre straight under Cinema:
+
+**`insert_row`**
+
+```json
+{
+  "table": "genre",
+  "values": { "name": "Horror", "kind": "genre", "parent": "[[Cinema]]" }
+}
+```
+
+Refused:
+
+```text
+insert_row refused — nothing was written:
+invalid datanotes/main/public/genre/Horror.md: parent_by_level violates CHECK parent_by_level: {"or":[{"kind":"cinema","parent":{"exists":false}},{"kind":"era","parent.kind":"cinema"},{"kind":"genre","parent.kind":"era"}]}
+Hint: The row breaks a CHECK constraint of the table (named in the message, with its rule): change the values so the rule holds. describe_table lists the table's constraints. Then call insert_row again (dry_run: true checks without writing).
+Docs: https://datanotes.dev/guides/data-quality/check-constraints.md#enforce-rules-with-check-constraints (offline: read_docs("guides/data-quality/check-constraints#enforce-rules-with-check-constraints"))
+```
+
+When a write changes a linked row, the engine checks the rows that link to it again. Sound film can’t become a top level like Cinema while a genre hangs from it, and the refusal lists Screwball comedy, naming Sound film.
+
+**`update_row`**
+
+```json
+{
+  "table": "genre",
+  "path": "datanotes/main/public/genre/Sound film.md",
+  "set": { "kind": "cinema" },
+  "unset": ["parent"]
+}
+```
+
+Refused:
+
+```text
+update_row refused — nothing was written:
+invalid datanotes/main/public/genre/Screwball comedy.md: parent_by_level violates CHECK parent_by_level: {"or":[{"kind":"cinema","parent":{"exists":false}},{"kind":"era","parent.kind":"cinema"},{"kind":"genre","parent.kind":"era"}]} (after this change to [[Sound film]], a row it links to)
+Hint: The row breaks a CHECK constraint of the table (named in the message, with its rule): change the values so the rule holds. describe_table lists the table's constraints. Then call update_row again (dry_run: true checks without writing).
+Docs: https://datanotes.dev/guides/data-quality/check-constraints.md#enforce-rules-with-check-constraints (offline: read_docs("guides/data-quality/check-constraints#enforce-rules-with-check-constraints"))
+```
+
+In a `batch`, add the era first and then its genre. The engine checks each step against the vault as the earlier steps left it, so Slapstick comedy finds its parent.
+
+**`batch`**
+
+```json
+{
+  "summary": "add the silent era and slapstick comedy",
+  "operations": [
+    {
+      "op": "insert_row",
+      "table": "genre",
+      "values": { "name": "Silent cinema", "kind": "era", "parent": "[[Cinema]]" }
+    },
+    {
+      "op": "insert_row",
+      "table": "genre",
+      "values": {
+        "name": "Slapstick comedy",
+        "kind": "genre",
+        "parent": "[[Silent cinema]]"
+      }
+    }
+  ]
+}
+```
+
+Answer:
+
+```json
+{
+  "ok": true,
+  "op": "batch",
+  "dry_run": false,
+  "migration_id": "20261012T071949-batch-add-the-silent-era-and-slapsti",
+  "steps": [
+    {
+      "index": 0,
+      "op": "insert_row",
+      "ok": true,
+      "migration_id": "20261012T071949-insert_row-insert-into-genre-4",
+      "result": {
+        "ok": true,
+        "op": "insert_row",
+        "files_created": 1,
+        "planned_creates": ["datanotes/main/public/genre/Silent cinema.md"],
+        "commits": [{ "table": "main.public.genre", "version": 5, "operation": "WRITE" }],
+        "migration_id": "20261012T071949-insert_row-insert-into-genre-4"
+      }
+    },
+    {
+      "index": 1,
+      "op": "insert_row",
+      "ok": true,
+      "migration_id": "20261012T071949-insert_row-insert-into-genre-5",
+      "result": {
+        "ok": true,
+        "op": "insert_row",
+        "files_created": 1,
+        "planned_creates": ["datanotes/main/public/genre/Slapstick comedy.md"],
+        "commits": [{ "table": "main.public.genre", "version": 6, "operation": "WRITE" }],
+        "migration_id": "20261012T071949-insert_row-insert-into-genre-5"
+      }
+    }
+  ]
+}
+```
+
+**`select_rows`**
+
+```json
+{ "table": "genre", "columns": ["name", "kind", "parent"], "order_by": "name" }
+```
+
+Answer:
+
+```json
+{
+  "table": "main.public.genre",
+  "total": 5,
+  "rows": [
+    {
+      "path": "datanotes/main/public/genre/Cinema.md",
+      "name": "Cinema",
+      "kind": "cinema",
+      "parent": null
+    },
+    {
+      "path": "datanotes/main/public/genre/Screwball comedy.md",
+      "name": "Screwball comedy",
+      "kind": "genre",
+      "parent": "[[Sound film]]"
+    },
+    {
+      "path": "datanotes/main/public/genre/Silent cinema.md",
+      "name": "Silent cinema",
+      "kind": "era",
+      "parent": "[[Cinema]]"
+    },
+    {
+      "path": "datanotes/main/public/genre/Slapstick comedy.md",
+      "name": "Slapstick comedy",
+      "kind": "genre",
+      "parent": "[[Silent cinema]]"
+    },
+    {
+      "path": "datanotes/main/public/genre/Sound film.md",
+      "name": "Sound film",
+      "kind": "era",
+      "parent": "[[Cinema]]"
+    }
+  ]
+}
+```
+
+Renaming a row that other rows link to keeps the rule valid, because the links follow the row. Here Slapstick comedy follows the era to its new name, Silent film.
+
+**`update_row`**
+
+```json
+{
+  "table": "genre",
+  "path": "datanotes/main/public/genre/Silent cinema.md",
+  "set": { "name": "Silent film" }
+}
+```
+
+Answer:
+
+```json
+{
+  "ok": true,
+  "op": "update_row",
+  "notes_changed": 1,
+  "files_moved": 1,
+  "planned_edits": [
+    {
+      "path": "datanotes/main/public/genre/Silent cinema.md",
+      "set": { "name": "Silent film" },
+      "remove": []
+    }
+  ],
+  "planned_moves": [
+    {
+      "from": "datanotes/main/public/genre/Silent cinema.md",
+      "to": "datanotes/main/public/genre/Silent film.md"
+    }
+  ],
+  "commits": [{ "table": "main.public.genre", "version": 7, "operation": "UPDATE" }],
+  "migration_id": "20261012T071949-update_row-update-genre"
+}
+```
+
+**`select_rows`**
+
+```json
+{
+  "table": "genre",
+  "columns": ["name", "parent"],
+  "where": { "parent.kind": "era" },
+  "order_by": "name"
+}
+```
+
+Answer:
+
+```json
+{
+  "table": "main.public.genre",
+  "total": 2,
+  "rows": [
+    {
+      "path": "datanotes/main/public/genre/Screwball comedy.md",
+      "name": "Screwball comedy",
+      "parent": "[[Sound film]]"
+    },
+    {
+      "path": "datanotes/main/public/genre/Slapstick comedy.md",
+      "name": "Slapstick comedy",
+      "parent": "[[Silent film]]"
+    }
+  ]
+}
+```

package/dist/docs/guides/data-quality/check-table.md

@@ -0,0 +1,63 @@
+# Find rows that break the schema
+
+> List the rows that break their table's rules now, with check_table, for one table or the whole database.
+
+`check_table` checks the rows a table holds now against its columns and CHECK constraints, and lists the ones that break a rule. Use it after hand edits, after a sync, or before you tighten a rule. Without `table`, it checks every table, page by page. It writes nothing.
+
+## Find rows that break the schema
+
+The engine can’t refuse a hand edit, so a note you edit by hand can break the schema. `check_table` validates every row against these rules:
+
+* **Values**: types, ranges and enums
+* **Presence**: required columns and unknown keys
+* **Relations and rules**: links and constraints
+
+You edit `datanotes/main/public/film/His Girl Friday.md` by hand, outside the engine. The note now reads:
+
+```markdown
+---
+table: main.public.film
+id: 01a0f000-0000-7000-8000-0000000000bb
+title: His Girl Friday
+year: 1940
+status: finished
+rating: 6
+format: Blu-ray
+---
+```
+
+**`check_table`**
+
+```json
+{ "table": "film" }
+```
+
+Answer:
+
+```json
+{
+  "table": "film",
+  "rows_checked": 3,
+  "total": 1,
+  "violations": [
+    {
+      "path": "datanotes/main/public/film/His Girl Friday.md",
+      "violations": [
+        {
+          "column": "status",
+          "code": "enum",
+          "message": "must be one of: to watch; watched",
+          "value": "finished"
+        },
+        { "column": "rating", "code": "max", "message": "must be <= 5", "value": 6 },
+        {
+          "column": "format",
+          "code": "unknown_column",
+          "message": "is not a column of table \"film\"",
+          "value": "Blu-ray"
+        }
+      ]
+    }
+  ]
+}
+```