datanotes 2.0.0 → 3.0.0

package/dist/docs/guides/data-quality/hand-edits.md

@@ -0,0 +1,507 @@
+# Hand edits
+
+> Edit notes in an editor or with git: the engine checks the edit, reports what it breaks, recomputes what it owns and repairs the links to notes you rename.
+
+You can create, change, rename or remove a row outside the engine: in an editor, with git or through a sync tool. The engine checks the edit a moment later against the same rules as a write, but never refuses it: the note stays as you wrote it.
+
+The change feed reports what the edit breaks, and `check_table` lists the rows that are invalid now. The engine fixes what it owns: generated columns, two-way links, and the links to a note you rename or move.
+
+## Catch hand edits that break a rule
+
+The engine can’t refuse an edit made outside it, in an editor, through git or by a sync tool. Instead, it checks the edit as soon as it sees it, as `check_table` would:
+
+* **The row’s own rules**: types, enums, required columns, links, CHECKs and filtered link targets
+* **Rows that link to it**: when a rule follows links to the edited row
+* **Dangling links**: when you delete or rename a row by hand, the engine reports the links it leaves dangling
+
+The engine reports what the edit breaks in three places:
+
+* **Change feed**: the event lists the problems in `violations`, where `constraint` names a CHECK; the event leaves out rules the row already broke
+* **Log**: a WARN line
+* **History**: when history is on, the `EXTERNAL_EDIT` version counts them in `operation_metrics.violations`
+
+Here you turn Silent film into a top level by hand, with `kind: cinema`. It keeps a parent, and Slapstick comedy now hangs from a top level.
+
+First, wait until the change feed includes the rename’s rewrite of Slapstick comedy’s link, and keep the feed’s position:
+
+**`list_changes`**
+
+```json
+{ "tables": ["genre"], "wait_ms": 10000 }
+```
+
+You edit `datanotes/main/public/genre/Silent film.md` by hand, outside the engine. The note now reads:
+
+```markdown
+---
+table: main.public.genre
+id: 01a12efb-65b7-7feb-9066-1ef0e98cd245
+name: Silent film
+kind: cinema
+parent: "[[Cinema]]"
+---
+```
+
+Read the hand edit’s event. `list_changes` long-polls with `wait_ms` until the event arrives:
+
+**`list_changes`**
+
+```json
+{ "since": 39, "tables": ["genre"], "wait_ms": 10000 }
+```
+
+Answer:
+
+```json
+[
+  {
+    "seq": 40,
+    "kind": "update",
+    "path": "datanotes/main/public/genre/Silent film.md",
+    "source": "external",
+    "changes": { "kind": { "before": "era", "after": "cinema" } },
+    "violations": [
+      {
+        "path": "datanotes/main/public/genre/Silent film.md",
+        "violations": [
+          {
+            "code": "check",
+            "constraint": "parent_by_level",
+            "message": "violates CHECK parent_by_level"
+          }
+        ]
+      },
+      {
+        "path": "datanotes/main/public/genre/Slapstick comedy.md",
+        "violations": [
+          {
+            "code": "check",
+            "constraint": "parent_by_level",
+            "message": "violates CHECK parent_by_level (after the edit of [[Silent film]], a row it links to)"
+          }
+        ]
+      }
+    ]
+  }
+]
+```
+
+The rows stay as they are until you fix them. `check_table` lists the rows that are invalid now, and without `table` it checks every table:
+
+**`check_table`** (no arguments)
+
+Answer:
+
+```json
+{
+  "table": null,
+  "rows_checked": 16,
+  "tables_checked": 5,
+  "total": 2,
+  "violations": [
+    {
+      "path": "datanotes/main/public/genre/Silent film.md",
+      "table": "main.public.genre",
+      "violations": [
+        {
+          "constraint": "parent_by_level",
+          "code": "check",
+          "message": "violates CHECK parent_by_level: {\"or\":[{\"kind\":\"cinema\",\"parent\":{\"exists\":false}},{\"kind\":\"era\",\"parent.kind\":\"cinema\"},{\"kind\":\"genre\",\"parent.kind\":\"era\"}]}"
+        }
+      ]
+    },
+    {
+      "path": "datanotes/main/public/genre/Slapstick comedy.md",
+      "table": "main.public.genre",
+      "violations": [
+        {
+          "constraint": "parent_by_level",
+          "code": "check",
+          "message": "violates CHECK parent_by_level: {\"or\":[{\"kind\":\"cinema\",\"parent\":{\"exists\":false}},{\"kind\":\"era\",\"parent.kind\":\"cinema\"},{\"kind\":\"genre\",\"parent.kind\":\"era\"}]}"
+        }
+      ]
+    }
+  ]
+}
+```
+
+Fix the row through the engine:
+
+**`update_row`**
+
+```json
+{
+  "table": "genre",
+  "path": "datanotes/main/public/genre/Silent film.md",
+  "set": { "kind": "era" }
+}
+```
+
+Answer:
+
+```json
+{
+  "ok": true,
+  "op": "update_row",
+  "notes_changed": 1,
+  "planned_edits": [
+    {
+      "path": "datanotes/main/public/genre/Silent film.md",
+      "set": { "kind": "era" },
+      "remove": []
+    }
+  ],
+  "commits": [{ "table": "main.public.genre", "version": 9, "operation": "UPDATE" }],
+  "migration_id": "20261012T071951-update_row-update-genre"
+}
+```
+
+## Recompute generated columns after a hand edit
+
+An edit made outside the engine, in an editor, through git or by a sync tool, can leave generated columns stale. `check_table` reports them with code `generated`, and `recompute_columns` brings them up to date.
+
+The engine also recomputes them on its own about 3s after a hand edit, unless the `autoRecompute` setting is off. The examples on this page turn it off so you can see each step. Here you correct `where` by hand, because you saw His Girl Friday at the cinema, and `label` still says `home`.
+
+You edit `datanotes/main/public/watch/2026-10-03T2130.md` by hand, outside the engine. The note now reads:
+
+```markdown
+---
+table: main.public.watch
+id: 01a12efb-5b61-7abf-9cb9-45a1f5a8f839
+film: His Girl Friday
+occurred_at: 2026-10-03T21:30
+where: cinema
+day: 2026-10-03
+label: His Girl Friday (03/10 21:30, home)
+---
+```
+
+**`check_table`**
+
+```json
+{ "table": "watch" }
+```
+
+Answer:
+
+```json
+{
+  "table": "watch",
+  "rows_checked": 1,
+  "total": 1,
+  "violations": [
+    {
+      "path": "datanotes/main/public/watch/2026-10-03T2130.md",
+      "violations": [
+        {
+          "column": "label",
+          "code": "generated",
+          "message": "should be \"His Girl Friday (03/10 21:30, cinema)\" (computed)",
+          "value": "His Girl Friday (03/10 21:30, home)"
+        }
+      ]
+    }
+  ]
+}
+```
+
+**`recompute_columns`**
+
+```json
+{ "table": "watch" }
+```
+
+Answer:
+
+```json
+{
+  "ok": true,
+  "op": "recompute_columns",
+  "notes_changed": 1,
+  "planned_edits": [
+    {
+      "path": "datanotes/main/public/watch/2026-10-03T2130.md",
+      "set": { "label": "His Girl Friday (03/10 21:30, cinema)" },
+      "remove": []
+    }
+  ],
+  "commits": [{ "table": "main.public.watch", "version": 3, "operation": "RECOMPUTE" }],
+  "migration_id": "20261012T071948-recompute_columns-recompute-id-day-label-of-watc"
+}
+```
+
+**`select_rows`**
+
+```json
+{ "table": "watch", "columns": ["where", "label"] }
+```
+
+Answer:
+
+```json
+{
+  "table": "main.public.watch",
+  "total": 1,
+  "rows": [
+    {
+      "path": "datanotes/main/public/watch/2026-10-03T2130.md",
+      "where": "cinema",
+      "label": "His Girl Friday (03/10 21:30, cinema)"
+    }
+  ]
+}
+```
+
+## Rename a note by hand
+
+You can rename or move a note outside the engine: in a file manager, with git or with a sync tool. The engine then rewrites every link to the note’s old path, across the vault. It rewrites rows and plain notes alike, as it does when it renames a note itself. Here you rename the plain note `Notes/Film night.md`. A viewing of Charade, a row of the `watch` table, links to it in its text.
+
+You rename `Notes/Film night.md` to `Notes/Friday film night.md` by hand, outside the engine.
+
+The engine waits until the vault has been quiet for 4 seconds, so an editor or a sync tool finishes first. The journal then lists the repair as one operation:
+
+**`list_migrations`**
+
+```json
+{ "limit": 1 }
+```
+
+Answer:
+
+```json
+[
+  {
+    "op": "repair_links",
+    "summary": "repair the links after the hand move of Notes/Film night.md to Notes/Friday film night.md",
+    "notes_changed": 0,
+    "files": null
+  }
+]
+```
+
+The viewing now links to the new name:
+
+**`read_note`**
+
+```json
+{ "path": "datanotes/main/public/watch/2026-09-25T2100.md" }
+```
+
+Answer:
+
+```markdown
+---
+table: main.public.watch
+id: 01a12efb-c19b-751f-b73a-92c00d579131
+occurred_at: 2026-09-25T21:00
+film: "[[Charade]]"
+where: home
+---
+Part of [[Friday film night]]. Four of us on the sofa, and nobody guessed the ending.
+```
+
+`undo_migration` takes the repair back. The change feed lists its changes after the move, each with the move in `cause`. Links that an editor such as Obsidian already rewrote stay as they are: the engine doesn’t write them twice. To leave links as they are after a hand move, set `repairLinksOnHandMove` to `false` in the [engine’s settings](https://datanotes.dev/reference/settings/).
+
+## Find front matter keys the table doesn’t declare
+
+Suppose someone later adds a front matter key by hand. `infer_table` with a `table` lists the keys its rows use that the table doesn’t declare. Add them with `add_column`, or remove them from the rows.
+
+You edit `datanotes/main/public/film/Metropolis.md` by hand, outside the engine. The note now reads:
+
+```markdown
+---
+table: main.public.film
+id: 01a12efb-81ad-7b8f-aa3d-16ff4a7685be
+year: 1927
+minutes: 153
+status: to watch
+cast:
+  - "[[Brigitte Helm]]"
+director: Fritz Lang
+title: Metropolis
+---
+```
+
+**`infer_table`**
+
+```json
+{ "table": "film" }
+```
+
+Answer:
+
+```json
+{
+  "files": 6,
+  "paths": [
+    "datanotes/main/public/film/Battleship Potemkin.md",
+    "datanotes/main/public/film/Charade.md",
+    "datanotes/main/public/film/His Girl Friday.md",
+    "datanotes/main/public/film/Metropolis.md",
+    "datanotes/main/public/film/Nosferatu.md",
+    "datanotes/main/public/film/The Gold Rush.md"
+  ],
+  "skipped": [],
+  "columns": [
+    {
+      "name": "cast",
+      "type": "list",
+      "description": null,
+      "stats": {
+        "present": 1,
+        "filled": 1,
+        "distinct": 1,
+        "types": { "list": 1 },
+        "examples": [["[[Brigitte Helm]]"]]
+      },
+      "items": "link"
+    }
+  ],
+  "hints": [
+    "add these with add_column, each with a description of what it holds (null here; dense: false unless every row should carry the front matter key), or drop those front matter keys from the rows"
+  ]
+}
+```
+
+## Find rows whose front matter can’t be read
+
+A hand edit can break a note’s YAML, such as a title with a colon in it. The engine can’t read the row any more, so `select_rows` and every other read leave it out. The engine reports it in three places instead:
+
+* **Change feed**: an event of kind `unreadable`
+* **`check_table`**: a violation with code `unreadable`
+* **`describe_database`**: the table’s `unreadable_count`
+
+Here you give Sherlock Jr. a title with a colon by hand.
+
+**`list_changes`**
+
+```json
+{ "tables": ["film"] }
+```
+
+You edit `datanotes/main/public/film/Sherlock Jr..md` by hand, outside the engine. The note now reads:
+
+```markdown
+---
+table: main.public.film
+id: 01a12efb-6c40-791d-b62a-7b324b13333f
+title: Sherlock Jr.: the projectionist
+year: 1924
+status: watched
+tags:
+  - seen-2026
+rating: 4
+---
+```
+
+The change feed reports the broken note:
+
+**`list_changes`**
+
+```json
+{ "since": 37, "tables": ["film"], "kinds": ["unreadable"], "wait_ms": 10000 }
+```
+
+Answer:
+
+```json
+[
+  {
+    "kind": "unreadable",
+    "path": "datanotes/main/public/film/Sherlock Jr..md",
+    "violations": [
+      {
+        "path": "datanotes/main/public/film/Sherlock Jr..md",
+        "violations": [
+          {
+            "code": "unreadable",
+            "message": "the front matter cannot be read, so the row is left out of every read: Nested mappings are not allowed in compact mappings at line 3, column 8:"
+          }
+        ]
+      }
+    ]
+  }
+]
+```
+
+`check_table` lists it with the rows that break the schema:
+
+**`check_table`**
+
+```json
+{ "table": "film" }
+```
+
+Answer:
+
+```json
+[
+  {
+    "path": "datanotes/main/public/film/Sherlock Jr..md",
+    "violations": [
+      {
+        "code": "unreadable",
+        "message": "the front matter cannot be read, so the row is left out of every read: Nested mappings are not allowed in compact mappings at line 3, column 8:"
+      }
+    ]
+  }
+]
+```
+
+`describe_database` counts it apart from `row_count`:
+
+**`describe_database`** (no arguments)
+
+Answer:
+
+```json
+{
+  "name": "film",
+  "qualified": "main.public.film",
+  "row_count": 2,
+  "description": "Classic films I watched or want to watch",
+  "unreadable_count": 1
+}
+```
+
+Quote the title to fix the note. The row is back:
+
+You edit `datanotes/main/public/film/Sherlock Jr..md` by hand, outside the engine. The note now reads:
+
+```markdown
+---
+table: main.public.film
+id: 01a12efb-6c40-791d-b62a-7b324b13333f
+title: "Sherlock Jr.: the projectionist"
+year: 1924
+status: watched
+tags:
+  - seen-2026
+rating: 4
+---
+```
+
+**`select_rows`**
+
+```json
+{
+  "table": "film",
+  "columns": ["title"],
+  "where": { "title": { "contains": "Sherlock" } }
+}
+```
+
+Answer:
+
+```json
+{
+  "table": "main.public.film",
+  "total": 1,
+  "rows": [
+    {
+      "path": "datanotes/main/public/film/Sherlock Jr..md",
+      "title": "Sherlock Jr.: the projectionist"
+    }
+  ]
+}
+```

package/dist/docs/guides/data-quality/sync-between-devices.md

@@ -0,0 +1,51 @@
+# Sync the vault between devices
+
+> Keep a vault in Dropbox, iCloud Drive, Syncthing or Obsidian Sync: run one engine per synced folder, and find conflict copies and unreadable notes with check_table.
+
+You can keep the vault in a sync tool, such as Dropbox, iCloud Drive, Syncthing or Obsidian Sync, so your notes reach every device. The engine works on a synced vault when you follow one rule: one engine per synced folder.
+
+A sync tool can also leave conflict copies and notes it merged badly. `check_table` finds them.
+
+## Run one engine per synced folder
+
+The engine keeps its lock, `engine.lock`, in its store, outside the vault. The sync tool carries the vault but not the store. So an engine on another device doesn’t see the lock, and it starts. Two engines on one folder then lose each other’s writes.
+
+Follow these rules on a synced vault:
+
+* **One engine**: run the engine on one device only, such as a desktop or a server that stays on.
+* **No second engine on the same machine**: never run the engine on the host and the engine in Docker on the same folder. Each keeps a store of its own, so neither refuses the other.
+* **Other devices edit by hand**: on a laptop or a phone, edit notes in Obsidian or another editor. Programs and agents on those devices call the one engine over HTTP, with a [named token](https://datanotes.dev/guides/security/tokens/) each.
+
+To move the engine to another device, stop it with `datanotes stop` first. Then let the sync finish before you start the engine on the new device.
+
+The sync tool writes the other devices’ edits into the folder. The engine reads them as [hand edits](https://datanotes.dev/guides/data-quality/hand-edits/): it checks each one, but can’t refuse it.
+
+## Pick a sync tool
+
+Each tool handles a note changed on two devices at once in its own way:
+
+| Tool          | What it does with a conflict                                             |
+| ------------- | ------------------------------------------------------------------------ |
+| Dropbox       | keeps both: the second one gets `conflicted copy` and a date in its name |
+| iCloud Drive  | keeps both: the second one gets a number, such as `Metropolis 2.md`      |
+| Syncthing     | keeps both: the second one gets `.sync-conflict-` and a date in its name |
+| Obsidian Sync | merges the two versions into one note                                    |
+
+A conflict copy in a table’s folder has the same front matter, so it becomes a second row of the same table. A merge can leave front matter that no longer parses.
+
+## Find conflict copies and unreadable notes
+
+Run `check_table` without `table` after a sync, or whenever a device was offline for a while. It checks every table and reports what a sync left behind:
+
+* **Conflict copies**: a copy breaks each `unique` column, and an identity column that must match the file name. Its row lists the violations.
+* **Unreadable notes**: a table whose notes can’t be read is listed in `errors`, with the error that names the problem.
+
+To see the copies themselves, call `list_folder` on the table’s folder and look for the names in the table above.
+
+For each copy, keep one version:
+
+1. Compare the copy with the original with `read_note`.
+2. Move what you want to keep into the original with `update_row`.
+3. Delete the copy with `delete_rows`, or delete the file by hand.
+
+Run `check_table` again: the copies no longer show up.

package/dist/docs/guides/data-quality/validation.md

@@ -0,0 +1,90 @@
+# Validation
+
+> What the engine checks on every write, how it refuses a bad value, and how to try a write first.
+
+The engine checks every write against its table before it changes anything: required values, types, ranges, enums, links, unique values and CHECK constraints. When one check fails, it refuses the whole write and writes nothing. The refusal names the note, the column or rule, the values the engine accepts, and how to fix the call.
+
+People, programs and agents go through the same checks. Hand edits are checked too, but never refused: [Hand edits](https://datanotes.dev/guides/data-quality/hand-edits/) shows what happens to them.
+
+## See how the engine refuses bad values
+
+The engine checks every write against the columns. If one value is wrong, it writes nothing, and the answer names the column and the reason. For an enum, the answer also lists the allowed values.
+
+**`insert_row`**
+
+```json
+{
+  "table": "film",
+  "values": { "title": "Battleship Potemkin", "minutes": 0, "status": "finished" }
+}
+```
+
+Refused:
+
+```text
+insert_row refused — nothing was written:
+invalid datanotes/main/public/film/Battleship Potemkin.md: minutes must be >= 1
+invalid datanotes/main/public/film/Battleship Potemkin.md: status must be one of: to watch; watched
+Hint: Give a value inside the column's range (the message names the bound); describe_table shows its min and max. Use one of the allowed values listed in the message, written exactly; describe_table shows what each one means. To add a value, alter_column the enum. Then call insert_row again (dry_run: true checks without writing).
+Docs: https://datanotes.dev/guides/database-objects/tables.md#describe-a-table-before-writing-to-it (offline: read_docs("guides/database-objects/tables#describe-a-table-before-writing-to-it"))
+```
+
+A unique column refuses a second row with the same value:
+
+**`insert_row`**
+
+```json
+{ "table": "film", "values": { "title": "Metropolis" } }
+```
+
+Refused:
+
+```text
+insert_row refused — nothing was written:
+invalid datanotes/main/public/film/Metropolis-2.md: title must be unique (also used by datanotes/main/public/film/Metropolis.md)
+Hint: Another row already has this value: choose another one, or update that row instead (select_rows finds it). Then call insert_row again (dry_run: true checks without writing).
+Docs: https://datanotes.dev/guides/read-and-write/write-rows.md#update-one-row (offline: read_docs("guides/read-and-write/write-rows#update-one-row"))
+```
+
+## Try a write first with `dry_run`
+
+Every write accepts `dry_run: true`. The engine checks the write and reports what it would do, but writes nothing.
+
+**`insert_row`**
+
+```json
+{
+  "table": "film",
+  "values": { "title": "Battleship Potemkin", "year": 1925, "minutes": 75 },
+  "dry_run": true
+}
+```
+
+Answer:
+
+```json
+{
+  "ok": true,
+  "op": "insert_row",
+  "dry_run": true,
+  "planned_creates": ["datanotes/main/public/film/Battleship Potemkin.md"]
+}
+```
+
+The table still has one film:
+
+**`select_rows`**
+
+```json
+{ "table": "film", "columns": ["title"] }
+```
+
+Answer:
+
+```json
+{
+  "table": "main.public.film",
+  "total": 1,
+  "rows": [{ "path": "datanotes/main/public/film/Metropolis.md", "title": "Metropolis" }]
+}
+```