datanotes 3.4.2 → 3.5.0

package/dist/docs/guides/database-objects/packs.md

@@ -0,0 +1,563 @@
+# Packs of rows
+
+> Keep the rows of a closed period in one note, a section per row: fewer files on disk and in git, the same rows for the engine.
+
+A table can hold thousands of small rows a year, such as a reading log or a sleep log. Each row is a note, so the table folder holds thousands of files. A pack is one note that holds the rows of one period, such as a month, one `##` section per row.
+
+The engine still sees one row at a time. `select_rows`, links, the link graph, the change feed and the Model Context Protocol (MCP) tools give each row at its own path, with its values, text and modification time. Only the files on disk change.
+
+The examples on this page keep a log of reading sessions.
+
+## How a pack reads
+
+A pack is a Markdown note in the table’s folder, named `<table> <key>.md`, such as `reading 2025-03.md`. Its front matter names the table and the period. Each row is a `##` section:
+
+```markdown
+---
+pack: main.public.reading
+key: 2025-03
+description: "Rows of main.public.reading for 2025-03. ..."
+---
+
+
+## 2025-03-02T2100
+id:: 019a3c1e-7f3b-7c2a-9d41-6c0f1e2a3b4c
+started_at:: 2025-03-02T21:00
+book:: Moby-Dick
+minutes:: 40
+
+
+## 2025-03-16T2200
+id:: 019a3c1e-8a10-7d55-b2e3-1f4a5b6c7d8e
+started_at:: 2025-03-16T22:00
+book:: Persuasion
+minutes:: 30
+
+
+Chapter 4, the walk to Winthrop.
+```
+
+* The heading is the row’s file name.
+* A field is `key:: value`, its value YAML on one line, as in front matter.
+* The row’s own text starts after the first empty line, with headings from `###` down.
+
+The engine lists each row at the path it would have as a note of its own, such as `datanotes/main/public/reading/2025/2025-03-02T2100.md`. That path doesn’t exist on disk, but every operation finds the row there.
+
+## Know which rows stay notes of their own
+
+`pack_rows` leaves a row as a note of its own when:
+
+* its period isn’t over yet, or ended less than `pack_after_days` days ago
+* it changed in the last 24 hours
+* its text has a `#` or `##` heading, which would end its section
+* it isn’t at the path its values give, such as a name with a `-2` suffix
+* its note isn’t in the form the engine writes, such as another YAML style or a comment
+
+A dry run lists these rows in `excluded`, with the reason. A row the engine can’t pack now waits for the next run.
+
+## Link to a packed row
+
+A link to a packed row names the pack and the row’s heading: `[[reading 2025-03#2025-03-02T2100]]`. Obsidian follows it to the section. `pack_rows` rewrites the links other notes have to the rows in this form, and `unpack_rows` writes them back.
+
+A link by the row’s name, such as `[[2025-03-02T2100]]`, still leads to the row for the engine. A link column takes both forms.
+
+Links between rows of the same pack stay as they are written.
+
+## Know who reads the files
+
+Every reader that goes through the engine sees the rows: the HTTP API, the MCP tools, `datanotes/client` and the change feed. Programs that read the vault’s files directly see the packs instead:
+
+* Obsidian, which shows each pack as one note
+* scripts and services that count or open the files of a table folder, with `find`, `grep` or fixed paths
+* agents with a shell, and git hosts such as GitHub
+
+To count the rows of a table, call `select_rows`. On disk, count the headings of its packs with `grep -c '^## '`.
+
+Before you set `pack` on a table, check what reads its files without the engine.
+
+## Packs need the engine service
+
+The engine service, `datanotes serve`, keeps the packs. Obsidian’s plugin doesn’t, so it refuses the `pack` option and `pack_rows`.
+
+The service also packs on its own. Once a day, after 10 minutes without writes, it runs `pack_rows` on every table with a `pack` option. The setting `autoPack: false` turns this off.
+
+## Merge packs changed on two devices
+
+Two devices that add or change different rows of one pack shouldn’t get a git conflict. The engine has a git merge driver that merges a pack row by row, by `id`:
+
+1. A row changed on one side only takes that side.
+2. A row added on one side is added.
+3. A row removed on one side, and unchanged on the other, is removed.
+4. A row changed differently on both sides is a conflict in its section only.
+
+[Git sync](https://datanotes.dev/administration/git-sync/) registers the driver in the vault’s repository. To register it by hand, see that page.
+
+## Choose the period of a pack
+
+A table’s `pack` option names a date or date-time column and a period, such as `{started_at:YYYY-MM}`. Once a month is over, one note holds all its rows. `pack_after_days` sets how many days after the end of a period its rows wait, 7 by default.
+
+**`create_table`**
+
+```json
+{
+  "name": "reading",
+  "description": "Reading sessions: when I read, which book and for how long",
+  "time_column": "started_at",
+  "partition": "{started_at:YYYY}",
+  "pack": "{started_at:YYYY-MM}",
+  "pack_after_days": 3,
+  "columns": [
+    {
+      "name": "started_at",
+      "type": "datetime",
+      "required": true,
+      "description": "When the session started"
+    },
+    {
+      "name": "book",
+      "type": "string",
+      "required": true,
+      "description": "The book I read"
+    },
+    {
+      "name": "minutes",
+      "type": "integer",
+      "min": 1,
+      "description": "How long I read, in minutes"
+    }
+  ]
+}
+```
+
+Answer:
+
+```json
+{
+  "ok": true,
+  "migration_id": "20261026T080043-create_table-create-table-main-public-readi"
+}
+```
+
+`datanotes/metastore.yaml` now reads:
+
+```yaml
+# Metastore (Datanotes): catalogs, schemas, tables, columns, views.
+id: 01a17739-dd7c-7689-914e-27b8f67b394c
+config:
+  history: true
+
+
+catalogs:
+  main:
+    schemas:
+      public:
+        tables:
+          reading:
+            description: "Reading sessions: when I read, which book and for how long"
+            options:
+              partition: "{started_at:YYYY}"
+              filename: "{started_at:YYYY-MM-DD[T]HHmm}"
+              archive: _archive
+              pack: "{started_at:YYYY-MM}"
+              pack_after_days: 3
+            columns:
+              - name: id
+                type: string
+                required: true
+                description: "Unique id of the row (UUIDv7): set when the row is created, never changes"
+                unique: true
+                identity: uuid
+                dense: true
+              - name: started_at
+                type: datetime
+                required: true
+                description: When the session started
+              - name: book
+                type: string
+                required: true
+                description: The book I read
+              - name: minutes
+                type: integer
+                min: 1
+                description: How long I read, in minutes
+```
+
+`alter_table` changes them while no row is in a pack. Here the rows wait one day:
+
+**`alter_table`**
+
+```json
+{ "table": "reading", "pack": "{started_at:YYYY-MM}", "pack_after_days": 1 }
+```
+
+Answer:
+
+```json
+{ "ok": true, "planned_texts": ["datanotes/metastore.yaml"] }
+```
+
+The template takes one date or date-time column and a format made of `YYYY`, `MM` and `DD`. Anything else is refused, with the table’s date columns in the hint:
+
+**`alter_table`**
+
+```json
+{ "table": "reading", "pack": "{minutes:YYYY-MM}" }
+```
+
+Refused:
+
+```text
+main.public.reading: "pack": "minutes" is a integer column, not a date
+Hint: A pack holds the rows of a period: name a date or date-time column. The date columns of the table: started_at. For example: "{started_at:YYYY-MM}".
+Docs: https://datanotes.dev/guides/database-objects/packs.md#choose-the-period-of-a-pack (offline: read_docs("guides/database-objects/packs#choose-the-period-of-a-pack"))
+```
+
+## Pack and unpack rows
+
+`pack_rows` puts the rows of every closed period into their pack. A row changed in the last 24 hours waits for the next run. A dry run shows the packs first, and the rows that stay notes of their own, with the reason.
+
+Four sessions of March and April 2025. The last one has a heading of its own:
+
+**`batch`**
+
+```json
+{
+  "operations": [
+    {
+      "op": "insert_row",
+      "table": "reading",
+      "values": { "started_at": "2025-03-02T21:00", "book": "Moby-Dick", "minutes": 40 }
+    },
+    {
+      "op": "insert_row",
+      "table": "reading",
+      "values": { "started_at": "2025-03-09T21:30", "book": "Moby-Dick", "minutes": 25 }
+    },
+    {
+      "op": "insert_row",
+      "table": "reading",
+      "values": { "started_at": "2025-03-16T22:00", "book": "Persuasion", "minutes": 30 },
+      "body": "Chapter 4, the walk to Winthrop.\n"
+    },
+    {
+      "op": "insert_row",
+      "table": "reading",
+      "values": { "started_at": "2025-04-06T21:00", "book": "Persuasion", "minutes": 35 },
+      "body": "# Notes\n\nAnne and Wentworth meet again.\n"
+    }
+  ]
+}
+```
+
+The notes in `datanotes/main/public/reading` haven’t changed since 2025-05-01.
+
+**`pack_rows`**
+
+```json
+{ "table": "reading", "dry_run": true }
+```
+
+Answer:
+
+```json
+{
+  "ok": true,
+  "dry_run": true,
+  "packs": [
+    {
+      "pack": "datanotes/main/public/reading/reading 2025-03.md",
+      "key": "2025-03",
+      "rows": [
+        "datanotes/main/public/reading/2025/2025-03-02T2100.md",
+        "datanotes/main/public/reading/2025/2025-03-09T2130.md",
+        "datanotes/main/public/reading/2025/2025-03-16T2200.md"
+      ]
+    }
+  ],
+  "excluded": [
+    {
+      "path": "datanotes/main/public/reading/2025/2025-04-06T2100.md",
+      "reason": "its text has a # or ## heading: \"# Notes\""
+    }
+  ],
+  "open_rows": 0
+}
+```
+
+The same call without `dry_run` writes the pack. The three notes of March go, and one note holds them:
+
+**`pack_rows`**
+
+```json
+{ "table": "reading" }
+```
+
+Answer:
+
+```json
+{
+  "ok": true,
+  "migration_id": "20261026T080044-pack_rows-pack-3-rows-of-reading",
+  "packs": [
+    {
+      "pack": "datanotes/main/public/reading/reading 2025-03.md",
+      "key": "2025-03",
+      "rows": [
+        "datanotes/main/public/reading/2025/2025-03-02T2100.md",
+        "datanotes/main/public/reading/2025/2025-03-09T2130.md",
+        "datanotes/main/public/reading/2025/2025-03-16T2200.md"
+      ]
+    }
+  ]
+}
+```
+
+`datanotes/main/public/reading/reading 2025-03.md` now reads:
+
+```markdown
+---
+pack: main.public.reading
+key: 2025-03
+description: "Rows of main.public.reading for 2025-03. Each ## section is a row, its fields are key:: value. Read them with select_rows."
+---
+
+
+## 2025-03-02T2100
+id:: 01a17739-dd92-7321-9568-708a1a72dffe
+started_at:: 2025-03-02T21:00
+book:: Moby-Dick
+minutes:: 40
+
+
+
+
+
+
+## 2025-03-09T2130
+id:: 01a17739-dd97-79de-b866-231408f32289
+started_at:: 2025-03-09T21:30
+book:: Moby-Dick
+minutes:: 25
+
+
+
+
+
+
+## 2025-03-16T2200
+id:: 01a17739-dd9b-780a-9860-0eb45495b7d9
+started_at:: 2025-03-16T22:00
+book:: Persuasion
+minutes:: 30
+
+
+Chapter 4, the walk to Winthrop.
+```
+
+Nothing changes for the engine and its readers. `select_rows` gives each row at its own path, with its values and text:
+
+**`select_rows`**
+
+```json
+{
+  "table": "reading",
+  "columns": ["started_at", "book", "minutes", "file_modified"],
+  "order_by": "started_at"
+}
+```
+
+Answer:
+
+```json
+{
+  "table": "main.public.reading",
+  "total": 4,
+  "rows": [
+    {
+      "path": "datanotes/main/public/reading/2025/2025-03-02T2100.md",
+      "started_at": "2025-03-02T21:00",
+      "book": "Moby-Dick",
+      "minutes": 40,
+      "file_modified": "2025-05-01T00:00:00Z"
+    },
+    {
+      "path": "datanotes/main/public/reading/2025/2025-03-09T2130.md",
+      "started_at": "2025-03-09T21:30",
+      "book": "Moby-Dick",
+      "minutes": 25,
+      "file_modified": "2025-05-01T00:00:00Z"
+    },
+    {
+      "path": "datanotes/main/public/reading/2025/2025-03-16T2200.md",
+      "started_at": "2025-03-16T22:00",
+      "book": "Persuasion",
+      "minutes": 30,
+      "file_modified": "2025-05-01T00:00:00Z"
+    },
+    {
+      "path": "datanotes/main/public/reading/2025/2025-04-06T2100.md",
+      "started_at": "2025-04-06T21:00",
+      "book": "Persuasion",
+      "minutes": 35,
+      "file_modified": "2025-05-01T00:00:00Z"
+    }
+  ]
+}
+```
+
+`unpack_rows` takes rows out again, as notes of their own with the text and modification time they had. `where` picks some rows, `key` one pack:
+
+**`unpack_rows`**
+
+```json
+{ "table": "reading", "where": { "book": "Persuasion" } }
+```
+
+Answer:
+
+```json
+{
+  "ok": true,
+  "packs": [
+    {
+      "pack": "datanotes/main/public/reading/reading 2025-03.md",
+      "key": "2025-03",
+      "rows": ["datanotes/main/public/reading/2025/2025-03-16T2200.md"]
+    }
+  ]
+}
+```
+
+Both operations are journaled: `undo_migration` takes a pack apart, or packs the rows again. The change feed and the webhooks see no change of the rows, and no new version of the table is written.
+
+## Read and write a packed row
+
+A packed row is still a row. A write changes its section of the pack. When the new values move it to another period or path, the row comes out as a note of its own.
+
+**`update_row`**
+
+```json
+{
+  "table": "reading",
+  "path": "datanotes/main/public/reading/2025/2025-03-09T2130.md",
+  "set": { "minutes": 45 }
+}
+```
+
+Answer:
+
+```json
+{
+  "ok": true,
+  "migration_id": "20261026T080044-update_row-update-reading",
+  "planned_edits": [
+    {
+      "path": "datanotes/main/public/reading/2025/2025-03-09T2130.md",
+      "set": { "minutes": 45 },
+      "remove": []
+    }
+  ]
+}
+```
+
+`datanotes/main/public/reading/reading 2025-03.md` now reads:
+
+```markdown
+---
+pack: main.public.reading
+key: 2025-03
+description: "Rows of main.public.reading for 2025-03. Each ## section is a row, its fields are key:: value. Read them with select_rows."
+---
+
+
+## 2025-03-02T2100
+id:: 01a17739-dd92-7321-9568-708a1a72dffe
+started_at:: 2025-03-02T21:00
+book:: Moby-Dick
+minutes:: 40
+
+
+
+
+
+
+## 2025-03-09T2130
+id:: 01a17739-dd97-79de-b866-231408f32289
+started_at:: 2025-03-09T21:30
+book:: Moby-Dick
+minutes:: 45
+```
+
+## Unpack before you change the table’s layout
+
+A packed row keeps the path its values give it. So the rows must be notes of their own before the table moves, before its rows change folder or name, and before its `pack` option changes:
+
+**`alter_table`**
+
+```json
+{ "table": "reading", "partition": "{started_at:YYYY/MM}" }
+```
+
+Refused:
+
+```text
+main.public.reading has 2 rows in 1 pack: the table moves or its rows change folder or name only while every row is a note of its own
+Hint: Run unpack_rows on main.public.reading first, then alter the table, then pack_rows again.
+Docs: https://datanotes.dev/guides/database-objects/packs.md#unpack-before-you-change-the-tables-layout (offline: read_docs("guides/database-objects/packs#unpack-before-you-change-the-tables-layout"))
+```
+
+**`unpack_rows`**
+
+```json
+{ "table": "reading" }
+```
+
+Answer:
+
+```json
+{
+  "ok": true,
+  "packs": [
+    {
+      "pack": "datanotes/main/public/reading/reading 2025-03.md",
+      "key": "2025-03",
+      "rows": [
+        "datanotes/main/public/reading/2025/2025-03-02T2100.md",
+        "datanotes/main/public/reading/2025/2025-03-09T2130.md"
+      ]
+    }
+  ]
+}
+```
+
+**`alter_table`**
+
+```json
+{ "table": "reading", "partition": "{started_at:YYYY/MM}" }
+```
+
+Answer:
+
+```json
+{
+  "ok": true,
+  "planned_moves": [
+    {
+      "from": "datanotes/main/public/reading/2025/2025-03-02T2100.md",
+      "to": "datanotes/main/public/reading/2025/03/2025-03-02T2100.md"
+    },
+    {
+      "from": "datanotes/main/public/reading/2025/2025-03-09T2130.md",
+      "to": "datanotes/main/public/reading/2025/03/2025-03-09T2130.md"
+    },
+    {
+      "from": "datanotes/main/public/reading/2025/2025-03-16T2200.md",
+      "to": "datanotes/main/public/reading/2025/03/2025-03-16T2200.md"
+    },
+    {
+      "from": "datanotes/main/public/reading/2025/2025-04-06T2100.md",
+      "to": "datanotes/main/public/reading/2025/04/2025-04-06T2100.md"
+    }
+  ]
+}
+```