roam-research-mcp 3.1.0 → 4.0.1

package/README.md

@@ -20,6 +20,24 @@ Whether you want to give Claude superpowers over your knowledge base or just wan
 
 ![Before and after: copy-pasting notes into Roam by hand, versus Claude and your terminal reading and writing the graph directly — install with npm i -g roam-research-mcp, then `roam save "idea"` or pipe with `echo "Buy milk" | roam save --todo`](./roam-research-mcp-sketchnote.png)
 
+## What's New in v4.0
+
+**In one line:** a block containing a soft line break (Shift+Enter) now survives a page rewrite. Read a page, write it back, and nothing moves.
+
+Until now, a multi-line block rendered as two physical lines, the second at column 0. That reset the parser's indentation baseline, so every block after it collapsed toward the root and `roam_update_page_markdown` dutifully generated the moves to make your real page match. Reading a page and writing back a revision, the documented purpose of the tool, was enough to trigger it. Callout bodies and fenced code blocks are exactly the blocks that carry soft breaks.
+
+- **Soft breaks render as `⏎`.** A page containing one gains a leading `<!-- roam:escaped-newlines -->` marker line; keep it if you write the markdown back. Pages with no multi-line block render byte-identical to 3.x, with no marker and no encoding.
+- **Backslashes are never special.** The earlier design escaped newlines as `\n`, which is also a common prefix in authored text: `\nabla`, `\neq`, `C:\newdir`. The sentinel needs no such rule, so all of those are written exactly as typed, everywhere.
+- **Verbatim round-trips are no-ops.** Renderer output submitted back unchanged, title header included, produces zero actions. The CLI shares the fix: `roam get` piped into `roam save --update` leaves the page as it was.
+- **Two more guards on page rewrites.** Non-empty markdown that parses to zero blocks is now refused rather than deleting every block on the page (genuinely empty markdown still clears a page, as documented). And a hand-authored first block that happens to be an H1 echoing the page title is no longer stripped on an ordinary update.
+- **Linked references are encoded too.** `roam_fetch_page_full_view` escapes soft breaks in referring blocks and breadcrumbs, not just the page's own content.
+- **Browser clients pass CORS preflight.** The HTTP transport now allows `MCP-Protocol-Version` and `Last-Event-ID`, both of which a client must send after initialization. Non-browser clients were never affected.
+- **The MCP SDK is pinned** to the version the test suite runs against, so a fresh install gets the protocol surface that was tested rather than whatever npm serves that day.
+
+**Why a major.** Four read surfaces return different bytes for any page containing a multi-line block: `roam_fetch_page_by_title` (`format: "markdown"`), `roam_fetch_page_full_view`, `roam_get_subpages`, and `roam get`. If you use the server through an AI assistant, nothing is required of you. A script that parses markdown output of multi-line pages sees the new encoding. If you pinned `roam-research-mcp@3`, you keep 3.2.0's fixes and its documented multi-line limitation until you re-pin.
+
+Full detail, including the corner cases and how each fix was verified against the prior state, is in the [changelog](CHANGELOG.md).
+
 ## How this differs from Roam's official MCP server
 
 Roam Research ships its own MCP server and CLI ([`@roam-research/roam-mcp`](https://github.com/Roam-Research/roam-tools)). It is a good tool, and this project is not trying to replace it. **They talk to two different Roam APIs, which is the difference everything else follows from.**
@@ -162,6 +180,8 @@ Three things worth knowing:
 - **Read tools deliberately have neither.** They already serialise their whole result into the text channel, so a schema would just double the payload.
 - **These fields are additive-only.** Some clients validate live responses against a cached tool list, so a field will be added or deprecated — never renamed or removed outside a major version.
 
+> **Upgrading to 4.0.0:** markdown reads of a page containing a soft line break (Shift+Enter) now render that break as `⏎` and carry a leading `<!-- roam:escaped-newlines -->` marker, so the page survives a write-back intact. Pages without multi-line blocks are byte-identical to 3.x. AI-assistant users need do nothing; scripts parsing markdown output of multi-line pages see the new encoding. See the [changelog](CHANGELOG.md).
+
 > **Upgrading from 2.x:** three write-result fields were renamed — `uid` → `page_uid` (`roam_create_page`), `created_uids` → `created_blocks` (`roam_create_outline`, `roam_import_markdown`) and `preservedUids` → `preserved_uids` (`roam_update_page_markdown`). This only affects code that reads those names; if you use the server through an AI assistant, nothing changes. See the [changelog](CHANGELOG.md) for why.
 
 ---
@@ -183,6 +203,14 @@ This is distinct from `CUSTOM_INSTRUCTIONS_PATH`, and the two compose:
 
 If the page doesn't exist, the tool returns `exists: false` rather than failing, so it is always safe to call.
 
+### It also returns the rules that aren't yours to set
+
+Alongside your conventions, every `roam_get_guidelines` response carries a `roamSyntax` field: the short list of things that *destroy* content — `roam_update_page_markdown` deleting every block your markdown omits, truncated `structure` previews written back as if they were content, block references retyped as plain text — plus a caution that reads silently exclude `#.rm-hide` subtrees, and the handful of places Roam's markdown inverts standard markdown.
+
+Two reasons it rides here rather than in the cheatsheet. It reaches **every** client, including one that never calls `roam_markdown_cheatsheet`; and it is returned even when a graph has **no** guidelines page, which is exactly the case where an agent has least context. The layering is deliberate: **your conventions win on style, `roamSyntax` wins on data safety.** No convention can make a truncated preview complete.
+
+The full syntax reference — components, queries, embeds, tool selection — stays in `roam_markdown_cheatsheet`. `roamSyntax` is ~800 tokens and deliberately capped.
+
 Each graph can point at a different page, or turn it off:
 
 ```bash
@@ -200,6 +228,14 @@ Results are cached for 30 seconds — an edit to the page takes effect without a
 
 Note that guidelines are read through the normal page path, so blocks tagged `#.rm-hide` / `#.rm-private` are withheld from them too — see below.
 
+Reads of a page containing a soft line break render it as `⏎` so each block
+stays on one line — an unescaped newline lands at column 0 and reparents
+everything after it on write-back. Such payloads carry a leading
+`<!-- roam:escaped-newlines -->` marker; keep it if you write the markdown
+back. Markdown you author is never decoded: backslashes are not special, and
+only `⏎` inside a marked payload is interpreted. `roam_get_guidelines` output
+is plain prose — no sentinel, no marker.
+
 ---
 
 ## Hiding content from the AI
@@ -210,6 +246,8 @@ This follows the same convention as Roam's official MCP server, so a block tagge
 
 Applied to: `roam_fetch_page_by_title`, `roam_fetch_block`, `roam_fetch_page_full_view`, `roam_get_subpages`, `roam_search_by_text`, `roam_search_for_tag`, `roam_search_by_status`, `roam_search_block_refs`, `roam_search_hierarchy`, `roam_search_by_date`.
 
+**Hidden blocks are also excluded from the page-rewrite diff**, which is what stops them being *deleted* for being absent from markdown the agent could not have written. `roam_update_page_markdown` (and `roam save --update`) replaces a page with what you give it, deleting whatever your markdown omits — so its baseline is pruned by this same filter, on the rule that **the baseline a diff deletes from must be the same page the caller was allowed to read.** It reports `preserved_hidden` when it protected anything. Content is preserved; exact ordering relative to visible siblings may shift. This was a real data-loss bug before the fix — see the [changelog](CHANGELOG.md).
+
 **This is a convenience filter, not a security guarantee.** `roam_datomic_query` reads the database directly and deliberately does **not** apply it, so a capable agent can still surface hidden blocks through raw Datalog. Treat these tags as "keep it out of the AI's way," not "keep it secret."
 
 Tag matching is case-insensitive, and only exact tags match — `#.rm-hidden` and `#.rm-highlight` are left alone. The set of hidden UIDs is cached for 30 seconds, so a block tagged just now may remain visible for up to that long.

package/build/Roam_Markdown_Cheatsheet.md

@@ -1,16 +1,22 @@
-# Roam Markdown Cheatsheet v2.3.0
+# Roam Markdown Cheatsheet v2.8.0
 
 ## Core Syntax
 
 ### Formatting
 `**bold**` · `__italic__` · `^^highlight^^` · `~~strike~~` · `` `code` `` · `$$LaTeX$$`
 
+⚠️ `*italic*` and `_italic_` are normalized to `__italic__` by the markdown tools, but `roam_process_batch_actions` writes strings literally and Roam does not render single markers. Write `__italic__` everywhere so reads and writes agree.
+
+### Headings
+`# H1` · `## H2` · `### H3` at the start of a block. Roam has H1–H3 only; `####` and deeper are not headings and stay literal text. A heading is a property of its own block: it does not nest the blocks after it. Indent to nest.
+
 ### Links & References
 - **Page ref:** `[[Page Name]]` — creates/links to page
 - **Block ref:** `((block-uid))` — embeds block content inline
 - **Block embed:** `{{[[embed]]: ((block-uid))}}` — full block with children
 - **Embed children:** `{{[[embed-children]]: ((block-uid))}}` — children only (not the parent block)
 - **Embed path:** `{{[[embed-path]]: ((block-uid))}}` — block with its ancestor path
+- **Page embed:** `{{[[embed]]: [[Page Name]]}}` — the whole page inline
 - **External:** `[text](URL)`
 - **Aliased page:** `[display text]([[Actual Page]])`
 - **Aliased block:** `[display text](<((block-uid))>)` — note the angle brackets
@@ -32,6 +38,50 @@ Always ordinal format: `[[January 1st, 2025]]`, `[[December 23rd, 2024]]`
 - Todo: `{{[[TODO]]}} task`
 - Done: `{{[[DONE]]}} task`
 
+⚠️ `{{[[TODO]]}}` must lead the block. Anywhere else it renders a checkbox that does not toggle to DONE.
+
+### Callouts
+Styled blockquotes with an icon and colour. Two page refs open the block: `[[>]]` marks it a callout, `[[!TYPE]]` picks the style.
+
+```
+[[>]] [[!TIP]] Title text
+Body on the next line
+```
+
+The body is a **soft line break inside the same block** (Shift+Enter in the UI, `\n` in the block string) — not a child block. A child block renders as a nested bullet inside the callout instead, which is usually not what you want.
+
+Types: `NOTE` `INFO` `SUMMARY` `TIP` `SUCCESS` `QUESTION` `WARNING` `FAILURE` `DANGER` `BUG` `EXAMPLE` `QUOTE`
+
+Append `+` or `-` to make it foldable — `[[!TIP]]+` starts expanded, `[[!TIP]]-` starts collapsed.
+
+⚠️ `[[>]]` and `[[!TIP]]` are real page references, so every callout backlinks to those pages. That is normal and how the feature works — don't "clean it up."
+⚠️ A plain `> quote` is an ordinary blockquote, not a callout. The two are unrelated.
+
+### Soft line breaks
+
+A block can hold more than one line — a **soft line break** (Shift+Enter in the
+UI). It stays *one* block, which is what callout bodies and fenced code blocks
+rely on.
+
+**You cannot write one through the markdown tools.** `roam_create_page`,
+`roam_import_markdown`, `roam_create_outline` and `roam_update_page_markdown`
+all treat a line break as a block break. Use `roam_process_batch_actions`, which
+writes block strings literally — put a real newline in the `string`.
+
+The one exception: inside a payload carrying the `<!-- roam:escaped-newlines -->`
+marker (below), a `⏎` decodes to a real soft line break on write. So keeping —
+or adding — a `⏎` in a marker-carrying payload handed to
+`roam_update_page_markdown` *is* the one markdown-tools path that writes one.
+Without the marker, `⏎` is just the character; it does not decode.
+
+⚠️ Reads render a soft line break as `⏎` so the block stays on one line, and a
+payload containing any is marked with a leading `<!-- roam:escaped-newlines -->`
+comment. If you edit that markdown and pass it back to
+`roam_update_page_markdown`, **keep the marker line** — it is what tells the
+server `⏎` means a line break there. Content you add yourself is safe either
+way: backslashes are never special, and `$$\nabla f$$` or `C:\newdir` are
+written exactly as typed.
+
 ### Attributes
 ```
 Type:: Book
@@ -54,6 +104,9 @@ Rating:: 4/5
         - Grandchild
 ```
 
+### Numbered & document lists
+A numbered list is a **view type on the parent**, not `1.` markers in the text. Set `children-view-type: "numbered"` (or `"document"` for no bullets) on the parent through `roam_process_batch_actions` (`create-block` / `update-block`) or a `roam_create_page` content item. `1.` markers written into block text stay literal.
+
 ### Code Blocks
 ````
 ```javascript
@@ -61,14 +114,40 @@ const x = 1;
 ```
 ````
 
+⚠️ Through `roam_process_batch_actions` (literal), do not leave a trailing newline before the closing fence. Roam stores it verbatim and renders it wrong. The markdown tools normalize it away.
+
 ### Queries
 ```
 {{[[query]]: {and: [[tag1]] [[tag2]]}}}
 {{[[query]]: {or: [[A]] [[B]]}}}
 {{[[query]]: {not: [[exclude]]}}}
 {{[[query]]: {between: [[January 1st, 2025]] [[January 31st, 2025]]}}}
+{{[[query]]: {and: [[Project]] {search: mobile}}}}
+```
+
+Clauses nest: `{and: [[Project]] {not: [[DONE]]}}`.
+
+**Queries match REFERENCES, not text.** Operands must be `[[Page]]` or `((block-uid))` — bare or quoted words do not match. `{and: TODO}` and `{and: "project alpha"}` both find nothing; write `{and: [[TODO]]}` and `{and: [[project alpha]]}`. For free text use `roam_search_by_text`, or a `{search:}` clause.
+
+**`{search:}` only works nested inside `{and:}` or `{or:}`** — never on its own, and it is the one clause that takes plain text rather than a reference.
+
+**`{between:}` only works on Daily Notes pages.** It filters by the daily page a block lives on, so it does nothing for content on ordinary pages. It accepts shorthands: `[[today]]`, `[[yesterday]]`, `[[last week]]`, `[[next month]]`.
+
+Also available: `{created-by: [[User]]}`, `{edited-by: [[User]]}`, `{by: [[User]]}`.
+
+#### Page-ref inheritance — the non-obvious one
+**A block inherits its parent's page refs for query matching.** So this TODO matches `{and: [[TODO]] [[Project Alpha]]}` even though it contains no reference to Project Alpha:
+
+```
+- Notes on [[Project Alpha]]
+    - {{[[TODO]]}} Ship the thing
 ```
 
+Three consequences:
+- **Don't tag every child with the parent's ref** — it is already inherited, and the duplication just clutters the backlinks.
+- **Do tag a child explicitly** if you want it to match *independently* of where it sits. Move it later and inherited matching goes with the old parent.
+- **Reading results:** a returned block may not visibly contain the thing you queried for. The matching ref can be on an ancestor. Don't report the result as wrong, and don't "fix" the block by adding the tag.
+
 ### Calculator
 `{{[[calc]]: 2 + 2}}`
 
@@ -115,13 +194,15 @@ Diagram definition via nested bullets or a code block child:
         - A[Start] --> B{Decision}
         - B -->|Yes| C[Action]
 ```
-Theme via CSS: `:root { --mermaidjs-theme: dark; }` (in `roam/css`)
+Per-diagram theme: first child line `%%{init: {"theme":"forest"}}%%`. Graph-wide via CSS: `:root { --mermaidjs-theme: dark; }` (in `roam/css`)
 
 ### Hiccup
 `:hiccup [:iframe {:width "600" :height "400" :src "URL"}]`
 
 ## Advanced Components
 
+Write the `{{[[name]]: arg}}` form. The `/name` slash commands are UI-only and do nothing in a written block string.
+
 ### Dropdowns & Tooltips
 - **Dropdown:** `{{or: option A|option B|option C}}` — select from options, display chosen one
 - **Tooltip:** `{{=:text|hidden content}}` — click to reveal/hide content
@@ -134,7 +215,28 @@ Theme via CSS: `:root { --mermaidjs-theme: dark; }` (in `roam/css`)
 - **Datalog block query:** `{{datalog-block-query: [:find ?b :where [?b :block/string "text"]]}}` — renders results like native queries
 - **Datalog table:** `:q [:find ?title :where [?p :node/title ?title]]` — renders results in sortable table
   - Supports column transforms, date arithmetic, resizable columns, pagination
-  - Built-in rules: `(created-by ?user ?block)`, `(edited-by ?user ?block)`, `(by ?user ?block)`, `(refs-page ?title ?b)`, `(block-or-parent-refs-page ?title ?b)`, `(created-between ?t1 ?t2 ?b)`, `(edited-between ?t1 ?t2 ?b)`
+  - `:q` extensions, usable **only inside a `:q` block in the graph**: date symbols `ms/today-start`, `ms/this-week-start`, `ms/+1D-start`, `dnp/today`, `dnp/-1D` (a daily-page title), `current/page-title`; inbuilt rules `(created-by ?user ?b)`, `(edited-by ?user ?b)`, `(by ?user ?b)`, `(refs-page ?title ?b)`, `(block-or-parent-refs-page ?title ?b)`, `(created-between ?t1 ?t2 ?b)`, `(edited-between ?t1 ?t2 ?b)`, `(in-dnp ?dnp ?b)`, `(refs-dnp ?dnp ?b)`, `(in-dnp-between ?start ?end ?b)`
+
+#### `roam_datomic_query` runs plain DataScript only
+⚠️ The `:q` extensions above do **not** work through `roam_datomic_query`. A rule such as `(refs-page "X" ?b)` fails with `Missing rules var '%'`. Write the raw clauses instead, `[?p :node/title "X"] [?b :block/refs ?p]`, and use epoch-millisecond literals for time bounds.
+
+**Schema** (what `:where` clauses match):
+- Pages: `:node/title`
+- Blocks: `:block/string` (raw stored text), `:block/uid`, `:block/order`, `:block/page`, `:block/children` (immediate), `:block/parents` (all ancestors), `:block/refs` (outgoing), `:block/heading` (1–3), `:children/view-type`
+- Both: `:create/time`, `:edit/time` (epoch ms); `:create/user`, `:edit/user` point at a user entity with `:user/uid` (every user) and `:user/display-page`
+
+Three gotchas:
+- **`:user/email` exists only on human users.** API-token and AI writers have none, so a join on `:user/email` silently drops their blocks. Match on `:user/uid`.
+- **Results are unordered.** Sort client-side; `:create/time` descending for newest-first.
+- **Recursive child pulls cap at 1000 per level** unless written `{[:block/children :limit nil] ...}`. `[*]` alone is immediate children only.
+
+```clojure
+;; blocks referencing a page (the portable form of refs-page)
+[:find ?uid :where [?p :node/title "Project Alpha"] [?b :block/refs ?p] [?b :block/uid ?uid]]
+
+;; a page and its full block tree, uncapped
+[:find (pull ?e [* {[:block/children :limit nil] ...}]) :where [?e :node/title "Page Name"]]
+```
 
 ### Document Mode
 `:document` — opens inline WYSIWYG text editor in the block
@@ -145,6 +247,15 @@ Theme via CSS: `:root { --mermaidjs-theme: dark; }` (in `roam/css`)
 - `{{word-count}}` — displays word count for the block
 - `{{chart: ATTR_PAGE_TO_CHART}}` — chart component
 - `{{a}}` — anonymous slider (shared graphs)
+- `{{[[video]]: URL}}` — YouTube / Vimeo / Loom
+- `{{[[mentions]]: [[Page]]}}` — inline a page's linked and unlinked references
+- `{{date}}` — date picker that inserts a date-page ref
+- `{{[[slider]]}}` — inline rating control, e.g. `certainty:: {{[[slider]]}}`
+- `{{[[streak]]: [[Goal]]}}` — heatmap of how often a ref appears in daily notes
+- `{{roam/render: ((codeUid))}}` — render a referenced code block as a component
+- `{{[[roam/css]]}}` — a child fenced `css` block applies graph-wide styling
+- `{{diagram: Title}}` — 2D canvas; each node is a real block
+- `{{encrypt}}` — write it bare; Roam converts it once the user supplies content and a password
 
 ### CSS Tags
 - `#.classname` — applies CSS class `.classname` to the block
@@ -153,10 +264,13 @@ Theme via CSS: `:root { --mermaidjs-theme: dark; }` (in `roam/css`)
 | Tag | Effect |
 |-----|--------|
 | `#.rm-E` | Display children horizontally |
-| `#.rm-g` | Hide block when children expanded |
-| `#.rm-hide` | Hide block when collapsed (clickable bar to reveal) |
+| `#.rm-g` | Promote children up a level; `[[.rm-g]]` keeps the container visible |
+| `#.rm-hide` | Collapse to a clickable bar in the UI; **withheld from every AI read** |
+| `#.rm-private` | Roam's hidden-from-other-users tag; **also withheld from every AI read** |
 | `#.rm-hide-for-readers` | Hide block for read-only users |
 
+⚠️ `#.rm-hide` and `#.rm-private` remove the block **and its whole subtree** from what the read tools return, with no marker in the output. A page can look complete when it is not.
+
 ## Anti-Patterns
 
 | ❌ Wrong | ✅ Correct |
@@ -170,6 +284,16 @@ Theme via CSS: `:root { --mermaidjs-theme: dark; }` (in `roam/css`)
 | `- *bullet` | `- bullet` |
 | `* bullet` | `- bullet` |
 | `**Attr**:: val` | `Attr:: val` |
+| `{and: TODO}` | `{and: [[TODO]]}` (queries match refs, not words) |
+| `{and: "project alpha"}` | `{and: [[project alpha]]}` |
+| `{{[[query]]: {search: text}}}` | `{{[[query]]: {and: {search: text}}}}` (never standalone) |
+| `> [[!TIP]] Title` | `[[>]] [[!TIP]] Title` |
+| callout body as a child block | body as `\n` in the same block |
+| `*italic*` in batch actions | `__italic__` |
+| `#### Heading` | `### Heading` (H1–H3 only) |
+| `1. item` as a numbered list | `children-view-type: "numbered"` on the parent |
+| `- [ ] task` | `{{[[TODO]]}} task` |
+| `(refs-page "X" ?b)` in `roam_datomic_query` | `[?p :node/title "X"] [?b :block/refs ?p]` |
 
 ## Tool Selection
 
@@ -229,222 +353,62 @@ Use `{{uid:name}}` for parent refs in batch actions:
 ```
 Server returns `{"uid_map": {"parent": "Xk7mN2pQ9"}}`.
 
+⚠️ `delete-block` breaks every `((uid))` that pointed at the block, and Roam has no undo for API writes. Check `roam_search_block_refs` before deleting a block others may reference.
+
 ## Structural Defaults
 
 - **Hierarchy:** 2-4 levels preferred, rarely exceed 5
 - **Blocks:** One idea per block
-- **Page refs vs tags:** `[[Page]]` for expandable concepts, `#tag` for filtering
 - **Embed vs ref:** `((uid))` inline, `{{[[embed]]: ((uid))}}` with children, `{{[[embed-children]]: ((uid))}}` children only, `{{[[embed-path]]: ((uid))}}` with ancestors, `[text](<((uid))>)` link only
-- **No empty blocks or `---` dividers** — use hierarchy for visual separation
+- **No empty blocks** — use hierarchy for visual separation
+- **Never invent a `((uid))`:** use only uids a tool actually returned. A fabricated ref is a broken link.
 
-## Output Conventions
+## Conventions
 
-**Quote:** `<text> —[[Author]] #quote`
-**Definition:** `Term:: definition #definition`
-**Open question:** `{{[[TODO]]}} Research: <question> #[[open questions]]`
+This cheatsheet is syntax only. How to tag, when to use a page ref versus a hashtag, and the shape of a quote, definition, TODO, or footnote are conventions, and they belong to the graph's `[[roam/agent guidelines]]` page (returned by `roam_get_guidelines`) and the personalization layer appended below. Where a convention and this sheet appear to disagree, the convention wins on style; this sheet wins on what Roam will actually render.
 
 ---
-# Roam Preferences — Personalization Layer
-
-> This section contains YOUR specific conventions, tagging philosophy, and graph-specific rules. Customize to match your workflow.
-
----
-
-## Graph-Level Behaviors
-
-### On Creating New Pages
-<!-- CUSTOMIZE: What should happen when a new page is created? -->
-- After creating a new page, add a reference block on today's daily page: `Created page: [[New Page Name]]`
-- <!-- Add any naming conventions, required metadata, etc. -->
-
-### On Adding Content
-<!-- CUSTOMIZE: Any rules about where/how content gets added? -->
-- Default location for quick captures: Daily page
-- Long-form content: Create dedicated page, link from daily page
-- <!-- Your preferences here -->
-
----
-
-## Tagging Philosophy
-
-### Core Principle
-> Tag for **intellectual collision** and **future discovery**, not just categorization. Every tag should maximize potential for unexpected connections.
-
-### The Serendipity Test
-Before tagging, ask: *"Could this concept surprise me by connecting to something completely unrelated?"*
-
-### What To Tag — Decision Framework
-
-```
-ASK YOURSELF:
-┌─ How will Future Me find this?
-│   └─ Tag by retrieval context, not just content
-│
-├─ What domain does this belong to?
-│   └─ Use broad category tags: #[[knowledge management]], #[[decision-making]]
-│
-├─ Is this a proper noun?
-│   └─ YES → Wrap name (no titles): [[Werner Erhard]], [[NASA]]
-│   └─ For abbreviations: [NASA]([[National Aeronautics and Space Administration (NASA)]])
-│
-├─ Could this alias to existing page?
-│   └─ YES → [displayed phrase]([[existing page name]])
-│   └─ Example: [frameworks for decisions]([[decision-making frameworks]])
-│
-└─ Parent block with children?
-    └─ Tag parent when category applies to all children
-    └─ Tag individual children for specific categorization
-```
-
-### Tag Type Selection
-
-| Use This | When |
-|----------|------|
-| `[[Page Reference]]` | Concept deserves its own page, will be expanded |
-| `#[[hashtag]]` | Categorization, filtering, won't be a standalone page |
-| `#single-word` | Simple, unambiguous category |
-| Attribute `Type::` | Structured metadata for queries |
-
-### WHEN creating Endnotes/Footnotes:
-- Find/Create the block with heading "Footnotes::" and nest footnote item below. (Footnotes do not need to be on the same page as the block to which it references. Typically on the same page unless instructed otherwise.)
-- If not known, retrieve the block_uid reference for this footnote item.
-- In the block referencing the footnote, append the reference with footnote-item-block_id, example: "- <block_text> #ref ((block_uid))"
-
-### Structural Tagging (Beyond Content)
-
-Tag by **patterns and mechanisms**, not just subjects:
-
-| Structural Tag | Connects |
-|----------------|----------|
-| `#[[has feedback loops]]` | Systems, habits, markets, conversations |
-| `#[[requires calibration]]` | Instruments, relationships, AI prompts |
-| `#[[exhibits emergence]]` | Complexity, culture, creativity |
-| `#[[perspective switching]]` | Photography, negotiation, analysis |
-| `#[[flow dynamics]]` | Fluids, music, conversation, sequences |
-
-### Problem-Oriented Tagging
-
-Tag by problems solved, not methods used:
-
-- `#[[breaking cognitive constraints]]`
-- `#[[expanding solution spaces]]`
-- `#[[preventing expert blindness]]`
-
-### Temporal & State-Based Tags
-
-| Tag Type | Examples |
-|----------|----------|
-| Future relevance | `#[[will be relevant in 5 years]]`, `#[[connects to unborn projects]]` |
-| Mental state triggers | `#[[feeling stuck in patterns]]`, `#[[needing fresh perspective]]` |
-| Review scheduling | `[[For review]]: [[August 12th, 2026]]` |
-
----
-
-## Formatting Conventions
-
-### Quotes
-```
-<quote text> —[[Author Name]] #quote #[[topic1]] #[[topic2]]
-```
-Always include 2-3 relevant hashtags after quotes.
-
-### TODOs and Follow-ups
-```
-{{[[TODO]]}} <action needed>
-{{[[TODO]]}} #researchThis : <topic to investigate> 
-```
-
-### Scheduled Reviews
-
-- Any block tagged with a date will show on that respective daily page.
-
-```
-[[For review]]: [[Date in ordinal format]]
-```
-Optional labels: "Deadline", "Approved", "Pending", "Deferred", "Postponed until"
-
-### Aliasing for Case Sensitivity
-When a tag would awkwardly affect sentence capitalization:
-```
-[Cognitive biases]([[cognitive biases]]) affect decision-making...
-```
-
-### Definitions (OVERRIDE)
-```
-#def [[<term>]] : <definition>
-```
-
----
-
-## Constraints & Guardrails
-
-### DON'T
-- **Overtag** — Quality over quantity; each tag should earn its place
-- **Tag obvious/redundant** — If parent block is tagged, children inherit context
-- **Use inconsistent capitalization** — Tags are lowercase unless proper nouns
-- **Create orphan tags** — Check if existing page/tag serves the purpose
-- **Bold Attributes** - ❌ `**Attribute**::`, ✅ `Attribute::` (Roam auto-formats)
-- **Separators** - `---` Don't use them.
-
-### DO
-- **Think retrieval-first** — How will you search for this later?
-- **Cross-pollinate domains** — Force unlikely intellectual meetings
-- **Update aging tags** — As interests evolve, so should tag vocabulary
-- **Track surprise discoveries** — When unexpected connections yield insights, engineer more of those patterns
-
----
-
-## Custom Rules
-
-<!-- 
-CUSTOMIZE THIS SECTION with your specific conventions:
-- Naming patterns for certain page types
-- Required attributes for books/articles/people
-- Project-specific tagging schemes
-- Integration rules with other tools
-- etc.
--->
-
-### Example Custom Rules (modify as needed):
-
-**Books:**
-```
-[[Book/<title> | <author>]]
-Type:: Book
-Author:: [[Author Name]]
-Status:: Reading | Completed | Abandoned
-Rating:: X/5
-```
-
-**People:**
-```
-[[Person Name]]
-Type:: Person
-Context:: How I know them
-```
-- When linking bibliographic references —>
-  Example: `McAdams, D.P. (2001) [The Psychology of Life Stories](https://journals.sagepub.com/doi/10.1037/1089-2680.5.2.100) — foundational paper`
-  - [McAdams, D.P.]([[Dan McAdams]]) - author's name in the graph
-  - If source URL, link to source: [The Psychology of Life Stories](https://journals.sagepub.com/doi/10.1037/1089-2680.5.2.100)
-  - If notes page exists or will exist in Roam: append ` | [Notes]([[Article/The Psychology of Life Stories]]), if not, just leave it without link.
-  
-**Projects:**
-```
-[[Project/<project anme>]]
-Status:: Active | Paused | Completed
-Start:: [[Date]]
-```
----
-
-## Integration Notes
-
-<!-- CUSTOMIZE: Any rules about how Roam integrates with your other tools/systems -->
-
-- Daily pages serve as: <!-- inbox / journal / task list / etc. -->
-- Weekly reviews occur on: <!-- day of week -->
-- Content flows from: <!-- capture tools, read-later apps, etc. -->
-- Content flows to: <!-- publishing, archives, etc. -->
-
----
-
-*End of Personalization Layer*
+<personalization_layer>
+# Roam: Nova operating notes (server-side supplement)
+
+Style conventions (voice, `#Nova` attribution, tagging, namespacing, formatting) live on the `[[roam/agent guidelines]]` page and arrive through `roam_get_guidelines`. This file carries only what that page deliberately leaves out: the mechanics behind the `#@Nova` handshake, so an interactive session produces the same artifacts as the nightly job and neither one answers a block twice.
+
+## The `#@Nova` handshake: mechanics
+
+Ownership rules are on the guidelines page (`#@Nova` is Ian's, `#@Nova-DONE` is Nova's, never self-tag). What follows is the output shape the nightly job writes and checks for.
+
+### Reading the request
+- The request is the tagged block's text with `#@Nova` removed, plus every child block that is not a `[NOVA]` reply, read as further instructions.
+- Context is the page title, the ancestor chain, and the sibling blocks under the same parent. Fetch any URL present before answering.
+- Empty tag line and no children means "review and respond to the surrounding context."
+
+### Writing the reply
+- First child of the tagged block: `[NOVA] <one-line summary, at most 150 characters>`.
+- The rest of the reply nests under that `[NOVA]` block as ordinary blocks. Headings and bullets become nesting.
+- Then flip the tag in place: `#@Nova` becomes `#@Nova-DONE`. Leave the rest of the block text untouched. A `{{[[TODO]]}}` on the block stays as-is; that checkbox is Ian's.
+- Then embed on today's daily page under a `## Nova Responses` heading: a block `Nova response on [[<origin page>]]` with the child `{{[[embed]]: ((<uid of the [NOVA] block>))}}`. The guidelines' root-level rule for daily pages applies to memories; Nova responses go under this heading.
+
+### Idempotency
+- A block that already has a `[NOVA]` child is answered. Do not answer it again, even if the tag still reads `#@Nova`. The nightly cleanup pass removes stray duplicates; do not create them.
+- `#@Nova-DONE` is never a work queue. Search `@Nova` for open items and skip anything containing `-DONE`.
+
+### Who runs when (America/Los_Angeles)
+| When | Job | Scope |
+|---|---|---|
+| 03:00 nightly | AI Intel sweep, `Skills/AIIntel/Tools/ProcessTriage.ts` | `#@Nova` and `#Status/*` on the `AI Intel Report` and `PAI-Anthropic Report` pages |
+| 04:30 nightly | `Skills/Productivity/scripts/ProcessNovaTodos.ts`, EventBridge job `nova_todo_processing` | `#@Nova` on every other page |
+| Interactive | This session, when Ian points at a block or asks for a sweep | Same output shape as above |
+
+- Both nightly jobs post their own Telegram summary. An interactive answer does not need one.
+- Dry run by hand: `bun $PAI_DIR/Skills/Productivity/scripts/ProcessNovaTodos.ts --dry-run`. Add `--cleanup` to only remove duplicate `[NOVA]` blocks.
+- On AI Intel pages, `#Status/ExecSum`, `#Status/Explore`, `#Status/Backburner`, and `#Status/Reject` run unattended. `#Status/Implement`, `#Status/PRD`, and `#@Nova` wait for an interactive session. Processed items end as `#Status/Processed`.
+
+## Retired conventions
+- `#PAI/do`, `#PAI/done`, `#PAI/hold`, `#PAI/ignore`: replaced by `#@Nova` and `#@Nova-DONE`. Nothing processes them and no block carried them as of September 6th, 2026. Never create them. If one turns up, treat it as `#@Nova`.
+- The `| [[Date]]` suffix once used to surface nested results on the daily page: replaced by the embed above.
+
+## Ian's capture tags (not Nova's queue)
+- `{{[[TODO]]}} #task <action>` on the daily page is Ian's capture inbox. ProcessFlow routes it to OmniFocus weekly. Do not convert `#task` to `#@Nova` unprompted; Ian decides what to delegate.
+- `#researchThis` means look into this later. It is not a task and not a delegation.
+- `#suggested-next-action` is a follow-up Nova proposes under a reply. Ian adds `#@Nova` to delegate it or checks it `{{[[DONE]]}}` to close it himself.
+</personalization_layer>

package/build/cli/utils/output.js

@@ -1,26 +1,49 @@
+import { escapeBlockString, needsNewlineEscaping, ESCAPED_NEWLINES_MARKER } from '../../shared/block-escaping.js';
 /**
- * Convert RoamBlock hierarchy to markdown with proper indentation
+ * Convert RoamBlock hierarchy to markdown with proper indentation.
+ *
+ * `escape` mirrors `PageOperations.fetchPageByTitle`'s markdown branch: a
+ * block may contain a soft line break (Shift+Enter), and this renderer emits
+ * one `- ` line per block, so an unescaped newline spills onto a second
+ * physical line with no bullet, resetting the parser's indentation baseline
+ * and reparenting everything after it. Callers that render a full page
+ * compute whether escaping is needed and set this flag; callers that never
+ * feed their output back through `updatePageMarkdown` can leave it off.
  */
-export function blocksToMarkdown(blocks, level = 0) {
+export function blocksToMarkdown(blocks, level = 0, escape = false) {
     return blocks
         .map(block => {
         const indent = '  '.repeat(level);
         let md;
+        const text = escape ? escapeBlockString(block.string) : block.string;
         // Check block heading level and format accordingly
         if (block.heading && block.heading > 0) {
             const hashtags = '#'.repeat(block.heading);
-            md = `${indent}${hashtags} ${block.string}`;
+            md = `${indent}${hashtags} ${text}`;
         }
         else {
-            md = `${indent}- ${block.string}`;
+            md = `${indent}- ${text}`;
         }
         if (block.children && block.children.length > 0) {
-            md += '\n' + blocksToMarkdown(block.children, level + 1);
+            md += '\n' + blocksToMarkdown(block.children, level + 1, escape);
         }
         return md;
     })
         .join('\n');
 }
+/**
+ * Collect every block string in a tree, for deciding whether a page needs
+ * newline escaping at all (see `blocksToMarkdown`).
+ */
+function collectBlockStrings(blocks, out = []) {
+    for (const block of blocks) {
+        out.push(block.string);
+        if (block.children && block.children.length > 0) {
+            collectBlockStrings(block.children, out);
+        }
+    }
+    return out;
+}
 /**
  * Flatten block hierarchy to single-level list
  */
@@ -42,7 +65,11 @@ export function formatPageOutput(title, blocks, options) {
         return JSON.stringify({ title, children: data }, null, 2);
     }
     const displayBlocks = options.flat ? flattenBlocks(blocks) : blocks;
-    return `# ${title}\n\n${blocksToMarkdown(displayBlocks)}`;
+    const escape = needsNewlineEscaping(collectBlockStrings(displayBlocks));
+    const body = blocksToMarkdown(displayBlocks, 0, escape);
+    return escape
+        ? `# ${title}\n${ESCAPED_NEWLINES_MARKER}\n\n${body}`
+        : `# ${title}\n\n${body}`;
 }
 /**
  * Format block content for output
@@ -53,7 +80,9 @@ export function formatBlockOutput(block, options) {
         return JSON.stringify(data, null, 2);
     }
     const displayBlocks = options.flat ? flattenBlocks([block]) : [block];
-    return blocksToMarkdown(displayBlocks);
+    const escape = needsNewlineEscaping(collectBlockStrings(displayBlocks));
+    const body = blocksToMarkdown(displayBlocks, 0, escape);
+    return escape ? `${ESCAPED_NEWLINES_MARKER}\n\n${body}` : body;
 }
 /**
  * Format search results for output