datanotes 2.0.1 → 3.0.0

package/dist/docs/guides/history/journal-and-undo.md

@@ -0,0 +1,262 @@
+# Journal and undo
+
+> Every write through the engine is journaled with who made it: list the writes, undo them, and see what undo can't take back.
+
+The journal records every write to rows and to the schema, with the data to undo it and who made it. `list_migrations` lists the writes, newest first, and `undo_migration` reverts one, batches included. An undo is itself a write, so you can undo it too.
+
+The journal works on every table, whatever its history setting. Reading a table as it was and restoring it need its transaction log too. A database the engine creates keeps one for every table: see [Table history and time travel](https://datanotes.dev/guides/history/table-history/).
+
+The examples work on a small diary of five classic films.
+
+## The engine rolls back a write it cannot finish
+
+A write can touch many notes, and the engine changes them one at a time. Sometimes it can’t change one of them halfway through:
+
+* Another program holds the file, such as an editor, an antivirus or a sync tool
+* Someone edits the note between the check and the change
+* Someone deletes or moves the note meanwhile
+
+The engine then puts back what it had already changed. It undoes the journal entry of the interrupted operation, as `undo_migration` would, so the database stays as it was. You don’t need to undo anything yourself.
+
+The answer has `ok: false`, the files it couldn’t change in `failed` and `rolled_back: true`. The HTTP API answers with status 503, which means nothing changed and you can retry. Its `hint` tells you to free the file and send the operation again:
+
+```json
+{
+  "ok": false,
+  "op": "update_rows",
+  "migration_id": "20261011T101500-update_rows-update-film",
+  "failed": [
+    {
+      "path": "datanotes/main/public/film/Nosferatu.md",
+      "error": "EBUSY: resource busy or locked"
+    }
+  ],
+  "rolled_back": true,
+  "notes_changed": 0
+}
+```
+
+Through the MCP server, the agent reads the same answer as text, which says that nothing is left changed.
+
+The journal keeps both steps. `list_migrations` shows the interrupted operation as undone, and a `rollback` entry that undid it. You can’t undo a rollback: send the operation again instead. The table history gets no version of the interrupted operation.
+
+Inside a [batch](https://datanotes.dev/guides/read-and-write/batch/), the engine rolls back the interrupted step first. The batch then undoes the steps before it, so the whole batch leaves nothing behind.
+
+On Windows, a file another program has open can’t be renamed for a moment. The engine retries a move for about three seconds before it gives up.
+
+### When the rollback can’t finish either
+
+The rollback can stop too, for example when the same program still holds a file. The answer then has `rolled_back: false`. `failed` lists the file that stopped the operation and each file the rollback left, with the error `not rolled back: …`.
+
+To finish the rollback:
+
+1. Close the program that holds the files.
+2. Call `undo_migration` with the operation’s `migration_id`.
+3. Send the operation again.
+
+`undo_migration` skips the files the rollback already put back, so you can call it again until it succeeds.
+
+## What undo can’t take back
+
+`undo_migration` reverts the writes the journal holds. Some changes never reach the journal, so no undo brings them back:
+
+* **`vacuum_table`**: it deletes old versions of a table’s history for good. Its `dry_run` is on by default.
+* **`configure_database`**: the database settings aren’t journaled. Set the old value again with `configure_database`.
+* **Webhooks**: `create_webhook` and `delete_webhook` aren’t journaled. Delete or create the webhook again.
+* **`sync_semantic_index`**: it rebuilds the semantic index, which the engine can build again at any time.
+* **Hand edits**: the journal doesn’t record a note you edit outside the engine. With history on, the table’s log records it, and `restore_table` brings back the version before it. Without history, nothing can.
+* **Old entries**: when the engine starts, it drops journal entries older than `journalRetentionDays`, 90 days by default. `list_migrations` no longer lists them, and they can’t be undone.
+* **A crash or `kill -9`**: a write cut short this way may leave some notes written and no journal entry. Find the rows it left with `check_table`, and repair them with a new write.
+
+`restore_database` doesn’t fill these gaps either: it goes back only through tables with history on. To revert one operation, use `undo_migration`.
+
+## Undo a write
+
+Every write returns a `migration_id`. `list_migrations` lists them, newest first, with who made each one, and `undo_migration` reverts one.
+
+**`list_migrations`**
+
+```json
+{ "table": "film", "limit": 3 }
+```
+
+Answer:
+
+```json
+{
+  "total": 13,
+  "migrations": [
+    {
+      "migration_id": "20261012T071951-delete_rows-archive-1-rows-of-film",
+      "op": "delete_rows",
+      "summary": "archive 1 rows of film",
+      "table": "main.public.film",
+      "executed_at": "2026-10-12T07:19:51.705Z",
+      "notes_changed": 0,
+      "files_changed": 1,
+      "undone": false,
+      "files_failed": 0,
+      "by": "docs-example"
+    },
+    {
+      "migration_id": "20261012T071951-merge_rows-merge-2-rows-into-film",
+      "op": "merge_rows",
+      "summary": "merge 2 rows into film",
+      "table": "main.public.film",
+      "executed_at": "2026-10-12T07:19:51.690Z",
+      "notes_changed": 1,
+      "files_changed": 1,
+      "undone": false,
+      "files_failed": 0,
+      "by": "docs-example"
+    },
+    {
+      "migration_id": "20261012T071951-update_rows-update-film",
+      "op": "update_rows",
+      "summary": "update film",
+      "table": "main.public.film",
+      "executed_at": "2026-10-12T07:19:51.674Z",
+      "notes_changed": 3,
+      "files_changed": 0,
+      "undone": false,
+      "files_failed": 0,
+      "by": "docs-example"
+    }
+  ]
+}
+```
+
+**`undo_migration`**
+
+```json
+{ "migration_id": "20261012T071951-delete_rows-archive-1-rows-of-film" }
+```
+
+Answer:
+
+```json
+{
+  "ok": true,
+  "op": "undo_migration",
+  "undoes": "20261012T071951-delete_rows-archive-1-rows-of-film",
+  "migration_id": "20261012T071951-undo_migration-undo-20261012t071951-delete_ro",
+  "conflicts": [],
+  "failed": []
+}
+```
+
+Battleship Potemkin is back:
+
+**`select_rows`**
+
+```json
+{ "table": "film", "columns": ["title"], "where": { "title": "Battleship Potemkin" } }
+```
+
+Answer:
+
+```json
+{
+  "table": "main.public.film",
+  "total": 1,
+  "rows": [
+    {
+      "path": "datanotes/main/public/film/Battleship Potemkin.md",
+      "title": "Battleship Potemkin"
+    }
+  ]
+}
+```
+
+## Undo refuses to overwrite later changes
+
+An undo never overwrites later work. The engine compares the values the operation wrote with the notes as they are now. If one of those values changed since, the engine refuses the undo and lists the changes. Changes to other columns don’t block the undo, and the engine keeps them.
+
+**`update_row`**
+
+```json
+{
+  "table": "film",
+  "path": "datanotes/main/public/film/Sherlock Jr..md",
+  "set": { "rating": 5 }
+}
+```
+
+Answer:
+
+```json
+{
+  "ok": true,
+  "op": "update_row",
+  "notes_changed": 1,
+  "planned_edits": [
+    {
+      "path": "datanotes/main/public/film/Sherlock Jr..md",
+      "set": { "rating": 5 },
+      "remove": []
+    }
+  ],
+  "commits": [{ "table": "main.public.film", "version": 14, "operation": "UPDATE" }],
+  "migration_id": "20261012T071951-update_row-update-film-5"
+}
+```
+
+Then someone changes the rating again:
+
+**`update_row`**
+
+```json
+{
+  "table": "film",
+  "path": "datanotes/main/public/film/Sherlock Jr..md",
+  "set": { "rating": 4 }
+}
+```
+
+**`undo_migration`**
+
+```json
+{ "migration_id": "20261012T071951-update_row-update-film-5" }
+```
+
+Refused:
+
+```text
+undo_migration refused — nothing was written:
+conflict datanotes/main/public/film/Sherlock Jr..md: "rating" changed since the operation
+Hint: Later changes touched the same notes: undo those first (list_migrations lists them, newest first), or change the rows with update_row.
+Docs: https://datanotes.dev/guides/history/journal-and-undo.md#undo-refuses-to-overwrite-later-changes (offline: read_docs("guides/history/journal-and-undo#undo-refuses-to-overwrite-later-changes"))
+```
+
+The undo of an insert trashes the new note only while it’s as the insert left it:
+
+**`insert_row`**
+
+```json
+{ "table": "film", "values": { "title": "The General", "year": 1926 } }
+```
+
+**`update_row`**
+
+```json
+{
+  "table": "film",
+  "path": "datanotes/main/public/film/The General.md",
+  "body": "Buster Keaton drives a locomotive.\n"
+}
+```
+
+**`undo_migration`**
+
+```json
+{ "migration_id": "20261012T071951-insert_row-insert-into-film-6" }
+```
+
+Refused:
+
+```text
+undo_migration refused — nothing was written:
+conflict datanotes/main/public/film/The General.md: created note changed since the operation
+Hint: Later changes touched the same notes: undo those first (list_migrations lists them, newest first), or change the rows with update_row.
+Docs: https://datanotes.dev/guides/history/journal-and-undo.md#undo-refuses-to-overwrite-later-changes (offline: read_docs("guides/history/journal-and-undo#undo-refuses-to-overwrite-later-changes"))
+```

package/dist/docs/guides/history/restore.md

@@ -0,0 +1,341 @@
+# Restore a table or the database
+
+> Bring a table back to an old version, or every table back to a time or to the moment before an operation.
+
+`restore_table` brings a table back to an old version, and `restore_database` brings every table that keeps a transaction log back to a time, or to the moment before an operation. Each restore is one write: it’s journaled, and `undo_migration` reverts it.
+
+The same history lets you time travel before you restore: `select_rows` with `timestamp` reads a table, a view, a join or a base as it was at that time.
+
+## Restore a table to an old version
+
+`restore_table` brings the rows and the definition back to a version:
+
+* it removes the rows added since
+* it gives changed rows their old values
+* it brings removed rows back
+
+The restore doesn’t rewrite history: it’s a new version, and you can undo it. Run it with `dry_run` first.
+
+The dry run names the version by its time, with a `timestamp` from the history read above:
+
+**`restore_table`**
+
+```json
+{ "table": "film", "timestamp": "2026-10-12T07:20:04.642Z", "dry_run": true }
+```
+
+Answer:
+
+```json
+{
+  "ok": true,
+  "op": "restore_table",
+  "dry_run": true,
+  "planned_creates": ["datanotes/main/public/film/Metropolis.md"]
+}
+```
+
+**`restore_table`**
+
+```json
+{ "table": "film", "version": 5 }
+```
+
+Answer:
+
+```json
+{
+  "ok": true,
+  "op": "restore_table",
+  "files_created": 1,
+  "planned_creates": ["datanotes/main/public/film/Metropolis.md"],
+  "commits": [{ "table": "main.public.film", "version": 7, "operation": "RESTORE" }],
+  "migration_id": "20261012T072004-restore_table-restore-main-public-film-to-ve"
+}
+```
+
+**`select_rows`**
+
+```json
+{ "table": "film", "columns": ["title", "status"] }
+```
+
+Answer:
+
+```json
+{
+  "table": "main.public.film",
+  "total": 2,
+  "rows": [
+    {
+      "path": "datanotes/main/public/film/Metropolis.md",
+      "title": "Metropolis",
+      "status": "watched"
+    },
+    {
+      "path": "datanotes/main/public/film/Nosferatu.md",
+      "title": "Nosferatu",
+      "status": "watched"
+    }
+  ]
+}
+```
+
+## Time travel and restore the whole database
+
+An agent can change several tables in one call: here one batch adds an actor and rates a film. `describe_history` without `table` lists the commits of every table, newest first, each with its `table`. In that list:
+
+* the commits of one operation share its `migration_id`
+* the commits of a batch carry the batch’s id as `batch`
+* `tables` names every table the change touched
+
+**`batch`**
+
+```json
+{
+  "summary": "add Cary Grant, rate His Girl Friday",
+  "operations": [
+    {
+      "op": "insert_row",
+      "table": "actor",
+      "values": { "name": "Cary Grant", "born": "1904-01-18" }
+    },
+    {
+      "op": "update_row",
+      "table": "film",
+      "path": "datanotes/main/public/film/His Girl Friday.md",
+      "set": { "rating": 5, "watched_on": "2026-10-08" }
+    }
+  ]
+}
+```
+
+Answer:
+
+```json
+{ "ok": true, "migration_id": "20261012T072013-batch-add-cary-grant-rate-his-girl-f" }
+```
+
+**`describe_history`**
+
+```json
+{ "limit": 3 }
+```
+
+Answer (62 lines)
+
+```json
+{
+  "tables": [
+    "main.public.actor",
+    "main.public.film",
+    "main.public.film_at",
+    "main.public.film_before"
+  ],
+  "history": [
+    {
+      "table": "main.public.film",
+      "version": 11,
+      "timestamp": "2026-10-12T07:20:13.321Z",
+      "operation": "UPDATE",
+      "operation_parameters": { "summary": "update film" },
+      "migration_id": "20261012T072013-update_row-update-film",
+      "operation_metrics": {
+        "num_added_files": 1,
+        "num_removed_files": 1,
+        "metadata_changed": false
+      },
+      "engine_info": "datanotes",
+      "user_name": "docs-example",
+      "actor": { "session": "history", "harness": "docs-example", "auth": "owner" },
+      "batch": "20261012T072013-batch-add-cary-grant-rate-his-girl-f",
+      "tables": ["main.public.actor", "main.public.film"]
+    },
+    {
+      "table": "main.public.actor",
+      "version": 2,
+      "timestamp": "2026-10-12T07:20:13.314Z",
+      "operation": "WRITE",
+      "operation_parameters": { "summary": "insert into actor" },
+      "migration_id": "20261012T072013-insert_row-insert-into-actor-2",
+      "operation_metrics": {
+        "num_added_files": 1,
+        "num_removed_files": 0,
+        "metadata_changed": false
+      },
+      "engine_info": "datanotes",
+      "user_name": "docs-example",
+      "actor": { "session": "history", "harness": "docs-example", "auth": "owner" },
+      "batch": "20261012T072013-batch-add-cary-grant-rate-his-girl-f",
+      "tables": ["main.public.actor", "main.public.film"]
+    },
+    {
+      "table": "main.public.film_at",
+      "version": 1,
+      "timestamp": "2026-10-12T07:20:13.304Z",
+      "operation": "ALTER TABLE",
+      "operation_parameters": { "summary": "alter table main.public.film_at" },
+      "migration_id": "20261012T072013-alter_table-alter-table-main-public-film_a",
+      "operation_metrics": {
+        "num_added_files": 0,
+        "num_removed_files": 0,
+        "metadata_changed": true
+      },
+      "engine_info": "datanotes",
+      "user_name": "docs-example",
+      "actor": { "session": "history", "harness": "docs-example", "auth": "owner" }
+    }
+  ]
+}
+```
+
+`restore_database` brings every table that keeps a log back to a moment, as one operation. `before_migration` names the moment by an operation, here the batch, and means the state immediately before it. The dry run lists what would change in each table, and shows a table without history, like `film_before`, as `skipped` with the reason.
+
+**`restore_database`**
+
+```json
+{
+  "before_migration": "20261012T072013-batch-add-cary-grant-rate-his-girl-f",
+  "dry_run": true
+}
+```
+
+Answer:
+
+```json
+{
+  "ok": true,
+  "op": "restore_database",
+  "dry_run": true,
+  "planned_deletes": ["datanotes/main/public/actor/Cary Grant.md"],
+  "planned_texts": ["datanotes/main/public/film/His Girl Friday.md"],
+  "restored_to": "2026-10-12T07:20:13.313Z",
+  "tables": [
+    {
+      "table": "main.public.actor",
+      "action": "restore",
+      "version": 1,
+      "definition_changed": false,
+      "creates": [],
+      "texts": [],
+      "deletes": ["datanotes/main/public/actor/Cary Grant.md"],
+      "archived": []
+    },
+    {
+      "table": "main.public.film",
+      "action": "restore",
+      "version": 10,
+      "definition_changed": false,
+      "creates": [],
+      "texts": ["datanotes/main/public/film/His Girl Friday.md"],
+      "deletes": [],
+      "archived": []
+    }
+  ],
+  "unchanged": ["main.public.film_at"],
+  "skipped": [
+    {
+      "table": "main.public.film_before",
+      "reason": "history is off for it: it keeps no transaction log to go back with (alter_table history=true, or configure_database history=true, keeps one from then on)"
+    }
+  ]
+}
+```
+
+**`restore_database`**
+
+```json
+{ "before_migration": "20261012T072013-batch-add-cary-grant-rate-his-girl-f" }
+```
+
+Answer:
+
+```json
+{
+  "ok": true,
+  "migration_id": "20261012T072013-restore_database-restore-database-to-before-202",
+  "restored_to": "2026-10-12T07:20:13.313Z",
+  "tables": ["main.public.actor: restore", "main.public.film: restore"]
+}
+```
+
+**`select_rows`**
+
+```json
+{ "table": "actor", "columns": ["name"] }
+```
+
+Answer:
+
+```json
+{
+  "table": "main.public.actor",
+  "total": 1,
+  "rows": [
+    {
+      "path": "datanotes/main/public/actor/Rosalind Russell.md",
+      "name": "Rosalind Russell"
+    }
+  ]
+}
+```
+
+The restore has one `migration_id` for every table, so `undo_migration` puts them all back as they were.
+
+A `timestamp` names the moment by its time, as it does for a table. `restored_to` is that time for the restore above: the last millisecond before the batch, so passing it as `timestamp` names the same moment. A restore to that time would drop a table created after it and archive its rows. It would also bring back a table dropped since.
+
+**`undo_migration`**
+
+```json
+{ "migration_id": "20261012T072013-restore_database-restore-database-to-before-202" }
+```
+
+**`restore_database`**
+
+```json
+{ "timestamp": "2026-10-12T07:20:13.313Z", "dry_run": true }
+```
+
+Answer:
+
+```json
+{
+  "ok": true,
+  "dry_run": true,
+  "tables": [
+    "main.public.actor: restore to version 1",
+    "main.public.film: restore to version 10"
+  ],
+  "skipped": [
+    {
+      "table": "main.public.film_before",
+      "reason": "history is off for it: it keeps no transaction log to go back with (alter_table history=true, or configure_database history=true, keeps one from then on)"
+    }
+  ]
+}
+```
+
+Time travel reads several tables the same way. `select_rows` with `timestamp` on a join, a view or `base:<name>` reads each table as of that time, and `versions` says which version of each it read.
+
+**`select_rows`**
+
+```json
+{ "table": "actor", "columns": ["name"], "timestamp": "2026-10-12T07:20:13.313Z" }
+```
+
+Answer:
+
+```json
+{
+  "table": "main.public.actor",
+  "total": 1,
+  "rows": [
+    {
+      "path": "datanotes/main/public/actor/Rosalind Russell.md",
+      "name": "Rosalind Russell"
+    }
+  ],
+  "version": 1,
+  "timestamp": "2026-10-12T07:20:13.272Z"
+}
+```