roam-research-mcp 3.2.0 → 4.0.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -20,6 +20,24 @@ Whether you want to give Claude superpowers over your knowledge base or just wan
20
20
 
21
21
  ![Before and after: copy-pasting notes into Roam by hand, versus Claude and your terminal reading and writing the graph directly — install with npm i -g roam-research-mcp, then `roam save "idea"` or pipe with `echo "Buy milk" | roam save --todo`](./roam-research-mcp-sketchnote.png)
22
22
 
23
+ ## What's New in v4.0
24
+
25
+ **In one line:** a block containing a soft line break (Shift+Enter) now survives a page rewrite. Read a page, write it back, and nothing moves.
26
+
27
+ Until now, a multi-line block rendered as two physical lines, the second at column 0. That reset the parser's indentation baseline, so every block after it collapsed toward the root and `roam_update_page_markdown` dutifully generated the moves to make your real page match. Reading a page and writing back a revision, the documented purpose of the tool, was enough to trigger it. Callout bodies and fenced code blocks are exactly the blocks that carry soft breaks.
28
+
29
+ - **Soft breaks render as `⏎`.** A page containing one gains a leading `<!-- roam:escaped-newlines -->` marker line; keep it if you write the markdown back. Pages with no multi-line block render byte-identical to 3.x, with no marker and no encoding.
30
+ - **Backslashes are never special.** The earlier design escaped newlines as `\n`, which is also a common prefix in authored text: `\nabla`, `\neq`, `C:\newdir`. The sentinel needs no such rule, so all of those are written exactly as typed, everywhere.
31
+ - **Verbatim round-trips are no-ops.** Renderer output submitted back unchanged, title header included, produces zero actions. The CLI shares the fix: `roam get` piped into `roam save --update` leaves the page as it was.
32
+ - **Two more guards on page rewrites.** Non-empty markdown that parses to zero blocks is now refused rather than deleting every block on the page (genuinely empty markdown still clears a page, as documented). And a hand-authored first block that happens to be an H1 echoing the page title is no longer stripped on an ordinary update.
33
+ - **Linked references are encoded too.** `roam_fetch_page_full_view` escapes soft breaks in referring blocks and breadcrumbs, not just the page's own content.
34
+ - **Browser clients pass CORS preflight.** The HTTP transport now allows `MCP-Protocol-Version` and `Last-Event-ID`, both of which a client must send after initialization. Non-browser clients were never affected.
35
+ - **The MCP SDK is pinned** to the version the test suite runs against, so a fresh install gets the protocol surface that was tested rather than whatever npm serves that day.
36
+
37
+ **Why a major.** Four read surfaces return different bytes for any page containing a multi-line block: `roam_fetch_page_by_title` (`format: "markdown"`), `roam_fetch_page_full_view`, `roam_get_subpages`, and `roam get`. If you use the server through an AI assistant, nothing is required of you. A script that parses markdown output of multi-line pages sees the new encoding. If you pinned `roam-research-mcp@3`, you keep 3.2.0's fixes and its documented multi-line limitation until you re-pin.
38
+
39
+ Full detail, including the corner cases and how each fix was verified against the prior state, is in the [changelog](CHANGELOG.md).
40
+
23
41
  ## How this differs from Roam's official MCP server
24
42
 
25
43
  Roam Research ships its own MCP server and CLI ([`@roam-research/roam-mcp`](https://github.com/Roam-Research/roam-tools)). It is a good tool, and this project is not trying to replace it. **They talk to two different Roam APIs, which is the difference everything else follows from.**
@@ -162,6 +180,8 @@ Three things worth knowing:
162
180
  - **Read tools deliberately have neither.** They already serialise their whole result into the text channel, so a schema would just double the payload.
163
181
  - **These fields are additive-only.** Some clients validate live responses against a cached tool list, so a field will be added or deprecated — never renamed or removed outside a major version.
164
182
 
183
+ > **Upgrading to 4.0.0:** markdown reads of a page containing a soft line break (Shift+Enter) now render that break as `⏎` and carry a leading `<!-- roam:escaped-newlines -->` marker, so the page survives a write-back intact. Pages without multi-line blocks are byte-identical to 3.x. AI-assistant users need do nothing; scripts parsing markdown output of multi-line pages see the new encoding. See the [changelog](CHANGELOG.md).
184
+
165
185
  > **Upgrading from 2.x:** three write-result fields were renamed — `uid` → `page_uid` (`roam_create_page`), `created_uids` → `created_blocks` (`roam_create_outline`, `roam_import_markdown`) and `preservedUids` → `preserved_uids` (`roam_update_page_markdown`). This only affects code that reads those names; if you use the server through an AI assistant, nothing changes. See the [changelog](CHANGELOG.md) for why.
166
186
 
167
187
  ---
@@ -208,6 +228,14 @@ Results are cached for 30 seconds — an edit to the page takes effect without a
208
228
 
209
229
  Note that guidelines are read through the normal page path, so blocks tagged `#.rm-hide` / `#.rm-private` are withheld from them too — see below.
210
230
 
231
+ Reads of a page containing a soft line break render it as `⏎` so each block
232
+ stays on one line — an unescaped newline lands at column 0 and reparents
233
+ everything after it on write-back. Such payloads carry a leading
234
+ `<!-- roam:escaped-newlines -->` marker; keep it if you write the markdown
235
+ back. Markdown you author is never decoded: backslashes are not special, and
236
+ only `⏎` inside a marked payload is interpreted. `roam_get_guidelines` output
237
+ is plain prose — no sentinel, no marker.
238
+
211
239
  ---
212
240
 
213
241
  ## Hiding content from the AI
@@ -1,16 +1,22 @@
1
- # Roam Markdown Cheatsheet v2.4.0
1
+ # Roam Markdown Cheatsheet v2.8.0
2
2
 
3
3
  ## Core Syntax
4
4
 
5
5
  ### Formatting
6
6
  `**bold**` · `__italic__` · `^^highlight^^` · `~~strike~~` · `` `code` `` · `$$LaTeX$$`
7
7
 
8
+ ⚠️ `*italic*` and `_italic_` are normalized to `__italic__` by the markdown tools, but `roam_process_batch_actions` writes strings literally and Roam does not render single markers. Write `__italic__` everywhere so reads and writes agree.
9
+
10
+ ### Headings
11
+ `# H1` · `## H2` · `### H3` at the start of a block. Roam has H1–H3 only; `####` and deeper are not headings and stay literal text. A heading is a property of its own block: it does not nest the blocks after it. Indent to nest.
12
+
8
13
  ### Links & References
9
14
  - **Page ref:** `[[Page Name]]` — creates/links to page
10
15
  - **Block ref:** `((block-uid))` — embeds block content inline
11
16
  - **Block embed:** `{{[[embed]]: ((block-uid))}}` — full block with children
12
17
  - **Embed children:** `{{[[embed-children]]: ((block-uid))}}` — children only (not the parent block)
13
18
  - **Embed path:** `{{[[embed-path]]: ((block-uid))}}` — block with its ancestor path
19
+ - **Page embed:** `{{[[embed]]: [[Page Name]]}}` — the whole page inline
14
20
  - **External:** `[text](URL)`
15
21
  - **Aliased page:** `[display text]([[Actual Page]])`
16
22
  - **Aliased block:** `[display text](<((block-uid))>)` — note the angle brackets
@@ -32,6 +38,8 @@ Always ordinal format: `[[January 1st, 2025]]`, `[[December 23rd, 2024]]`
32
38
  - Todo: `{{[[TODO]]}} task`
33
39
  - Done: `{{[[DONE]]}} task`
34
40
 
41
+ ⚠️ `{{[[TODO]]}}` must lead the block. Anywhere else it renders a checkbox that does not toggle to DONE.
42
+
35
43
  ### Callouts
36
44
  Styled blockquotes with an icon and colour. Two page refs open the block: `[[>]]` marks it a callout, `[[!TYPE]]` picks the style.
37
45
 
@@ -49,18 +57,30 @@ Append `+` or `-` to make it foldable — `[[!TIP]]+` starts expanded, `[[!TIP]]
49
57
  ⚠️ `[[>]]` and `[[!TIP]]` are real page references, so every callout backlinks to those pages. That is normal and how the feature works — don't "clean it up."
50
58
  ⚠️ A plain `> quote` is an ordinary blockquote, not a callout. The two are unrelated.
51
59
 
52
- ### ⚠️ Multi-line blocks and page rewrites
60
+ ### Soft line breaks
53
61
 
54
- A block can hold a **soft line break** (Shift+Enter, stored as `\n` in the block
55
- string). Callout bodies require one; fenced code blocks are full of them.
62
+ A block can hold more than one line — a **soft line break** (Shift+Enter in the
63
+ UI). It stays *one* block, which is what callout bodies and fenced code blocks
64
+ rely on.
56
65
 
57
- **Do not run `roam_update_page_markdown` or `roam save --update` — on a page
58
- containing a callout, a fenced code block, or any Shift+Enter line break.** It
59
- will split those blocks and **flatten the hierarchy of everything after them**,
60
- reparenting blocks under the wrong ancestors.
66
+ **You cannot write one through the markdown tools.** `roam_create_page`,
67
+ `roam_import_markdown`, `roam_create_outline` and `roam_update_page_markdown`
68
+ all treat a line break as a block break. Use `roam_process_batch_actions`, which
69
+ writes block strings literally — put a real newline in the `string`.
61
70
 
62
- Use `roam_process_batch_actions` for those pages. It writes block strings
63
- literally, and is currently the only way to create a soft line break at all.
71
+ The one exception: inside a payload carrying the `<!-- roam:escaped-newlines -->`
72
+ marker (below), a `⏎` decodes to a real soft line break on write. So keeping —
73
+ or adding — a `⏎` in a marker-carrying payload handed to
74
+ `roam_update_page_markdown` *is* the one markdown-tools path that writes one.
75
+ Without the marker, `⏎` is just the character; it does not decode.
76
+
77
+ ⚠️ Reads render a soft line break as `⏎` so the block stays on one line, and a
78
+ payload containing any is marked with a leading `<!-- roam:escaped-newlines -->`
79
+ comment. If you edit that markdown and pass it back to
80
+ `roam_update_page_markdown`, **keep the marker line** — it is what tells the
81
+ server `⏎` means a line break there. Content you add yourself is safe either
82
+ way: backslashes are never special, and `$$\nabla f$$` or `C:\newdir` are
83
+ written exactly as typed.
64
84
 
65
85
  ### Attributes
66
86
  ```
@@ -84,6 +104,9 @@ Rating:: 4/5
84
104
  - Grandchild
85
105
  ```
86
106
 
107
+ ### Numbered & document lists
108
+ A numbered list is a **view type on the parent**, not `1.` markers in the text. Set `children-view-type: "numbered"` (or `"document"` for no bullets) on the parent through `roam_process_batch_actions` (`create-block` / `update-block`) or a `roam_create_page` content item. `1.` markers written into block text stay literal.
109
+
87
110
  ### Code Blocks
88
111
  ````
89
112
  ```javascript
@@ -91,6 +114,8 @@ const x = 1;
91
114
  ```
92
115
  ````
93
116
 
117
+ ⚠️ Through `roam_process_batch_actions` (literal), do not leave a trailing newline before the closing fence. Roam stores it verbatim and renders it wrong. The markdown tools normalize it away.
118
+
94
119
  ### Queries
95
120
  ```
96
121
  {{[[query]]: {and: [[tag1]] [[tag2]]}}}
@@ -169,13 +194,15 @@ Diagram definition via nested bullets or a code block child:
169
194
  - A[Start] --> B{Decision}
170
195
  - B -->|Yes| C[Action]
171
196
  ```
172
- Theme via CSS: `:root { --mermaidjs-theme: dark; }` (in `roam/css`)
197
+ Per-diagram theme: first child line `%%{init: {"theme":"forest"}}%%`. Graph-wide via CSS: `:root { --mermaidjs-theme: dark; }` (in `roam/css`)
173
198
 
174
199
  ### Hiccup
175
200
  `:hiccup [:iframe {:width "600" :height "400" :src "URL"}]`
176
201
 
177
202
  ## Advanced Components
178
203
 
204
+ Write the `{{[[name]]: arg}}` form. The `/name` slash commands are UI-only and do nothing in a written block string.
205
+
179
206
  ### Dropdowns & Tooltips
180
207
  - **Dropdown:** `{{or: option A|option B|option C}}` — select from options, display chosen one
181
208
  - **Tooltip:** `{{=:text|hidden content}}` — click to reveal/hide content
@@ -188,7 +215,28 @@ Theme via CSS: `:root { --mermaidjs-theme: dark; }` (in `roam/css`)
188
215
  - **Datalog block query:** `{{datalog-block-query: [:find ?b :where [?b :block/string "text"]]}}` — renders results like native queries
189
216
  - **Datalog table:** `:q [:find ?title :where [?p :node/title ?title]]` — renders results in sortable table
190
217
  - Supports column transforms, date arithmetic, resizable columns, pagination
191
- - Built-in rules: `(created-by ?user ?block)`, `(edited-by ?user ?block)`, `(by ?user ?block)`, `(refs-page ?title ?b)`, `(block-or-parent-refs-page ?title ?b)`, `(created-between ?t1 ?t2 ?b)`, `(edited-between ?t1 ?t2 ?b)`
218
+ - `:q` extensions, usable **only inside a `:q` block in the graph**: date symbols `ms/today-start`, `ms/this-week-start`, `ms/+1D-start`, `dnp/today`, `dnp/-1D` (a daily-page title), `current/page-title`; inbuilt rules `(created-by ?user ?b)`, `(edited-by ?user ?b)`, `(by ?user ?b)`, `(refs-page ?title ?b)`, `(block-or-parent-refs-page ?title ?b)`, `(created-between ?t1 ?t2 ?b)`, `(edited-between ?t1 ?t2 ?b)`, `(in-dnp ?dnp ?b)`, `(refs-dnp ?dnp ?b)`, `(in-dnp-between ?start ?end ?b)`
219
+
220
+ #### `roam_datomic_query` runs plain DataScript only
221
+ ⚠️ The `:q` extensions above do **not** work through `roam_datomic_query`. A rule such as `(refs-page "X" ?b)` fails with `Missing rules var '%'`. Write the raw clauses instead, `[?p :node/title "X"] [?b :block/refs ?p]`, and use epoch-millisecond literals for time bounds.
222
+
223
+ **Schema** (what `:where` clauses match):
224
+ - Pages: `:node/title`
225
+ - Blocks: `:block/string` (raw stored text), `:block/uid`, `:block/order`, `:block/page`, `:block/children` (immediate), `:block/parents` (all ancestors), `:block/refs` (outgoing), `:block/heading` (1–3), `:children/view-type`
226
+ - Both: `:create/time`, `:edit/time` (epoch ms); `:create/user`, `:edit/user` point at a user entity with `:user/uid` (every user) and `:user/display-page`
227
+
228
+ Three gotchas:
229
+ - **`:user/email` exists only on human users.** API-token and AI writers have none, so a join on `:user/email` silently drops their blocks. Match on `:user/uid`.
230
+ - **Results are unordered.** Sort client-side; `:create/time` descending for newest-first.
231
+ - **Recursive child pulls cap at 1000 per level** unless written `{[:block/children :limit nil] ...}`. `[*]` alone is immediate children only.
232
+
233
+ ```clojure
234
+ ;; blocks referencing a page (the portable form of refs-page)
235
+ [:find ?uid :where [?p :node/title "Project Alpha"] [?b :block/refs ?p] [?b :block/uid ?uid]]
236
+
237
+ ;; a page and its full block tree, uncapped
238
+ [:find (pull ?e [* {[:block/children :limit nil] ...}]) :where [?e :node/title "Page Name"]]
239
+ ```
192
240
 
193
241
  ### Document Mode
194
242
  `:document` — opens inline WYSIWYG text editor in the block
@@ -199,6 +247,15 @@ Theme via CSS: `:root { --mermaidjs-theme: dark; }` (in `roam/css`)
199
247
  - `{{word-count}}` — displays word count for the block
200
248
  - `{{chart: ATTR_PAGE_TO_CHART}}` — chart component
201
249
  - `{{a}}` — anonymous slider (shared graphs)
250
+ - `{{[[video]]: URL}}` — YouTube / Vimeo / Loom
251
+ - `{{[[mentions]]: [[Page]]}}` — inline a page's linked and unlinked references
252
+ - `{{date}}` — date picker that inserts a date-page ref
253
+ - `{{[[slider]]}}` — inline rating control, e.g. `certainty:: {{[[slider]]}}`
254
+ - `{{[[streak]]: [[Goal]]}}` — heatmap of how often a ref appears in daily notes
255
+ - `{{roam/render: ((codeUid))}}` — render a referenced code block as a component
256
+ - `{{[[roam/css]]}}` — a child fenced `css` block applies graph-wide styling
257
+ - `{{diagram: Title}}` — 2D canvas; each node is a real block
258
+ - `{{encrypt}}` — write it bare; Roam converts it once the user supplies content and a password
202
259
 
203
260
  ### CSS Tags
204
261
  - `#.classname` — applies CSS class `.classname` to the block
@@ -207,10 +264,13 @@ Theme via CSS: `:root { --mermaidjs-theme: dark; }` (in `roam/css`)
207
264
  | Tag | Effect |
208
265
  |-----|--------|
209
266
  | `#.rm-E` | Display children horizontally |
210
- | `#.rm-g` | Hide block when children expanded |
211
- | `#.rm-hide` | Hide block when collapsed (clickable bar to reveal) |
267
+ | `#.rm-g` | Promote children up a level; `[[.rm-g]]` keeps the container visible |
268
+ | `#.rm-hide` | Collapse to a clickable bar in the UI; **withheld from every AI read** |
269
+ | `#.rm-private` | Roam's hidden-from-other-users tag; **also withheld from every AI read** |
212
270
  | `#.rm-hide-for-readers` | Hide block for read-only users |
213
271
 
272
+ ⚠️ `#.rm-hide` and `#.rm-private` remove the block **and its whole subtree** from what the read tools return, with no marker in the output. A page can look complete when it is not.
273
+
214
274
  ## Anti-Patterns
215
275
 
216
276
  | ❌ Wrong | ✅ Correct |
@@ -229,6 +289,11 @@ Theme via CSS: `:root { --mermaidjs-theme: dark; }` (in `roam/css`)
229
289
  | `{{[[query]]: {search: text}}}` | `{{[[query]]: {and: {search: text}}}}` (never standalone) |
230
290
  | `> [[!TIP]] Title` | `[[>]] [[!TIP]] Title` |
231
291
  | callout body as a child block | body as `\n` in the same block |
292
+ | `*italic*` in batch actions | `__italic__` |
293
+ | `#### Heading` | `### Heading` (H1–H3 only) |
294
+ | `1. item` as a numbered list | `children-view-type: "numbered"` on the parent |
295
+ | `- [ ] task` | `{{[[TODO]]}} task` |
296
+ | `(refs-page "X" ?b)` in `roam_datomic_query` | `[?p :node/title "X"] [?b :block/refs ?p]` |
232
297
 
233
298
  ## Tool Selection
234
299
 
@@ -288,222 +353,62 @@ Use `{{uid:name}}` for parent refs in batch actions:
288
353
  ```
289
354
  Server returns `{"uid_map": {"parent": "Xk7mN2pQ9"}}`.
290
355
 
356
+ ⚠️ `delete-block` breaks every `((uid))` that pointed at the block, and Roam has no undo for API writes. Check `roam_search_block_refs` before deleting a block others may reference.
357
+
291
358
  ## Structural Defaults
292
359
 
293
360
  - **Hierarchy:** 2-4 levels preferred, rarely exceed 5
294
361
  - **Blocks:** One idea per block
295
- - **Page refs vs tags:** `[[Page]]` for expandable concepts, `#tag` for filtering
296
362
  - **Embed vs ref:** `((uid))` inline, `{{[[embed]]: ((uid))}}` with children, `{{[[embed-children]]: ((uid))}}` children only, `{{[[embed-path]]: ((uid))}}` with ancestors, `[text](<((uid))>)` link only
297
- - **No empty blocks or `---` dividers** — use hierarchy for visual separation
298
-
299
- ## Output Conventions
300
-
301
- **Quote:** `<text> —[[Author]] #quote`
302
- **Definition:** `Term:: definition #definition`
303
- **Open question:** `{{[[TODO]]}} Research: <question> #[[open questions]]`
304
-
305
- ---
306
- # Roam Preferences — Personalization Layer
307
-
308
- > This section contains YOUR specific conventions, tagging philosophy, and graph-specific rules. Customize to match your workflow.
309
-
310
- ---
363
+ - **No empty blocks** — use hierarchy for visual separation
364
+ - **Never invent a `((uid))`:** use only uids a tool actually returned. A fabricated ref is a broken link.
311
365
 
312
- ## Graph-Level Behaviors
366
+ ## Conventions
313
367
 
314
- ### On Creating New Pages
315
- <!-- CUSTOMIZE: What should happen when a new page is created? -->
316
- - After creating a new page, add a reference block on today's daily page: `Created page: [[New Page Name]]`
317
- - <!-- Add any naming conventions, required metadata, etc. -->
318
-
319
- ### On Adding Content
320
- <!-- CUSTOMIZE: Any rules about where/how content gets added? -->
321
- - Default location for quick captures: Daily page
322
- - Long-form content: Create dedicated page, link from daily page
323
- - <!-- Your preferences here -->
324
-
325
- ---
326
-
327
- ## Tagging Philosophy
328
-
329
- ### Core Principle
330
- > Tag for **intellectual collision** and **future discovery**, not just categorization. Every tag should maximize potential for unexpected connections.
331
-
332
- ### The Serendipity Test
333
- Before tagging, ask: *"Could this concept surprise me by connecting to something completely unrelated?"*
334
-
335
- ### What To Tag — Decision Framework
336
-
337
- ```
338
- ASK YOURSELF:
339
- ┌─ How will Future Me find this?
340
- │ └─ Tag by retrieval context, not just content
341
-
342
- ├─ What domain does this belong to?
343
- │ └─ Use broad category tags: #[[knowledge management]], #[[decision-making]]
344
-
345
- ├─ Is this a proper noun?
346
- │ └─ YES → Wrap name (no titles): [[Werner Erhard]], [[NASA]]
347
- │ └─ For abbreviations: [NASA]([[National Aeronautics and Space Administration (NASA)]])
348
-
349
- ├─ Could this alias to existing page?
350
- │ └─ YES → [displayed phrase]([[existing page name]])
351
- │ └─ Example: [frameworks for decisions]([[decision-making frameworks]])
352
-
353
- └─ Parent block with children?
354
- └─ Tag parent when category applies to all children
355
- └─ Tag individual children for specific categorization
356
- ```
357
-
358
- ### Tag Type Selection
359
-
360
- | Use This | When |
361
- |----------|------|
362
- | `[[Page Reference]]` | Concept deserves its own page, will be expanded |
363
- | `#[[hashtag]]` | Categorization, filtering, won't be a standalone page |
364
- | `#single-word` | Simple, unambiguous category |
365
- | Attribute `Type::` | Structured metadata for queries |
366
-
367
- ### WHEN creating Endnotes/Footnotes:
368
- - Find/Create the block with heading "Footnotes::" and nest footnote item below. (Footnotes do not need to be on the same page as the block to which it references. Typically on the same page unless instructed otherwise.)
369
- - If not known, retrieve the block_uid reference for this footnote item.
370
- - In the block referencing the footnote, append the reference with footnote-item-block_id, example: "- <block_text> #ref ((block_uid))"
371
-
372
- ### Structural Tagging (Beyond Content)
373
-
374
- Tag by **patterns and mechanisms**, not just subjects:
375
-
376
- | Structural Tag | Connects |
377
- |----------------|----------|
378
- | `#[[has feedback loops]]` | Systems, habits, markets, conversations |
379
- | `#[[requires calibration]]` | Instruments, relationships, AI prompts |
380
- | `#[[exhibits emergence]]` | Complexity, culture, creativity |
381
- | `#[[perspective switching]]` | Photography, negotiation, analysis |
382
- | `#[[flow dynamics]]` | Fluids, music, conversation, sequences |
383
-
384
- ### Problem-Oriented Tagging
385
-
386
- Tag by problems solved, not methods used:
387
-
388
- - `#[[breaking cognitive constraints]]`
389
- - `#[[expanding solution spaces]]`
390
- - `#[[preventing expert blindness]]`
391
-
392
- ### Temporal & State-Based Tags
393
-
394
- | Tag Type | Examples |
395
- |----------|----------|
396
- | Future relevance | `#[[will be relevant in 5 years]]`, `#[[connects to unborn projects]]` |
397
- | Mental state triggers | `#[[feeling stuck in patterns]]`, `#[[needing fresh perspective]]` |
398
- | Review scheduling | `[[For review]]: [[August 12th, 2026]]` |
368
+ This cheatsheet is syntax only. How to tag, when to use a page ref versus a hashtag, and the shape of a quote, definition, TODO, or footnote are conventions, and they belong to the graph's `[[roam/agent guidelines]]` page (returned by `roam_get_guidelines`) and the personalization layer appended below. Where a convention and this sheet appear to disagree, the convention wins on style; this sheet wins on what Roam will actually render.
399
369
 
400
370
  ---
401
-
402
- ## Formatting Conventions
403
-
404
- ### Quotes
405
- ```
406
- <quote text> —[[Author Name]] #quote #[[topic1]] #[[topic2]]
407
- ```
408
- Always include 2-3 relevant hashtags after quotes.
409
-
410
- ### TODOs and Follow-ups
411
- ```
412
- {{[[TODO]]}} <action needed>
413
- {{[[TODO]]}} #researchThis : <topic to investigate>
414
- ```
415
-
416
- ### Scheduled Reviews
417
-
418
- - Any block tagged with a date will show on that respective daily page.
419
-
420
- ```
421
- [[For review]]: [[Date in ordinal format]]
422
- ```
423
- Optional labels: "Deadline", "Approved", "Pending", "Deferred", "Postponed until"
424
-
425
- ### Aliasing for Case Sensitivity
426
- When a tag would awkwardly affect sentence capitalization:
427
- ```
428
- [Cognitive biases]([[cognitive biases]]) affect decision-making...
429
- ```
430
-
431
- ### Definitions (OVERRIDE)
432
- ```
433
- #def [[<term>]] : <definition>
434
- ```
435
-
436
- ---
437
-
438
- ## Constraints & Guardrails
439
-
440
- ### DON'T
441
- - **Overtag** Quality over quantity; each tag should earn its place
442
- - **Tag obvious/redundant** If parent block is tagged, children inherit context
443
- - **Use inconsistent capitalization** Tags are lowercase unless proper nouns
444
- - **Create orphan tags** — Check if existing page/tag serves the purpose
445
- - **Bold Attributes** - ❌ `**Attribute**::`, ✅ `Attribute::` (Roam auto-formats)
446
- - **Separators** - `---` Don't use them.
447
-
448
- ### DO
449
- - **Think retrieval-first** — How will you search for this later?
450
- - **Cross-pollinate domains** — Force unlikely intellectual meetings
451
- - **Update aging tags** — As interests evolve, so should tag vocabulary
452
- - **Track surprise discoveries** — When unexpected connections yield insights, engineer more of those patterns
453
-
454
- ---
455
-
456
- ## Custom Rules
457
-
458
- <!--
459
- CUSTOMIZE THIS SECTION with your specific conventions:
460
- - Naming patterns for certain page types
461
- - Required attributes for books/articles/people
462
- - Project-specific tagging schemes
463
- - Integration rules with other tools
464
- - etc.
465
- -->
466
-
467
- ### Example Custom Rules (modify as needed):
468
-
469
- **Books:**
470
- ```
471
- [[Book/<title> | <author>]]
472
- Type:: Book
473
- Author:: [[Author Name]]
474
- Status:: Reading | Completed | Abandoned
475
- Rating:: X/5
476
- ```
477
-
478
- **People:**
479
- ```
480
- [[Person Name]]
481
- Type:: Person
482
- Context:: How I know them
483
- ```
484
- - When linking bibliographic references —>
485
- Example: `McAdams, D.P. (2001) [The Psychology of Life Stories](https://journals.sagepub.com/doi/10.1037/1089-2680.5.2.100) — foundational paper`
486
- - [McAdams, D.P.]([[Dan McAdams]]) - author's name in the graph
487
- - If source URL, link to source: [The Psychology of Life Stories](https://journals.sagepub.com/doi/10.1037/1089-2680.5.2.100)
488
- - If notes page exists or will exist in Roam: append ` | [Notes]([[Article/The Psychology of Life Stories]]), if not, just leave it without link.
489
-
490
- **Projects:**
491
- ```
492
- [[Project/<project anme>]]
493
- Status:: Active | Paused | Completed
494
- Start:: [[Date]]
495
- ```
496
- ---
497
-
498
- ## Integration Notes
499
-
500
- <!-- CUSTOMIZE: Any rules about how Roam integrates with your other tools/systems -->
501
-
502
- - Daily pages serve as: <!-- inbox / journal / task list / etc. -->
503
- - Weekly reviews occur on: <!-- day of week -->
504
- - Content flows from: <!-- capture tools, read-later apps, etc. -->
505
- - Content flows to: <!-- publishing, archives, etc. -->
506
-
507
- ---
508
-
509
- *End of Personalization Layer*
371
+ <personalization_layer>
372
+ # Roam: Nova operating notes (server-side supplement)
373
+
374
+ Style conventions (voice, `#Nova` attribution, tagging, namespacing, formatting) live on the `[[roam/agent guidelines]]` page and arrive through `roam_get_guidelines`. This file carries only what that page deliberately leaves out: the mechanics behind the `#@Nova` handshake, so an interactive session produces the same artifacts as the nightly job and neither one answers a block twice.
375
+
376
+ ## The `#@Nova` handshake: mechanics
377
+
378
+ Ownership rules are on the guidelines page (`#@Nova` is Ian's, `#@Nova-DONE` is Nova's, never self-tag). What follows is the output shape the nightly job writes and checks for.
379
+
380
+ ### Reading the request
381
+ - The request is the tagged block's text with `#@Nova` removed, plus every child block that is not a `[NOVA]` reply, read as further instructions.
382
+ - Context is the page title, the ancestor chain, and the sibling blocks under the same parent. Fetch any URL present before answering.
383
+ - Empty tag line and no children means "review and respond to the surrounding context."
384
+
385
+ ### Writing the reply
386
+ - First child of the tagged block: `[NOVA] <one-line summary, at most 150 characters>`.
387
+ - The rest of the reply nests under that `[NOVA]` block as ordinary blocks. Headings and bullets become nesting.
388
+ - Then flip the tag in place: `#@Nova` becomes `#@Nova-DONE`. Leave the rest of the block text untouched. A `{{[[TODO]]}}` on the block stays as-is; that checkbox is Ian's.
389
+ - Then embed on today's daily page under a `## Nova Responses` heading: a block `Nova response on [[<origin page>]]` with the child `{{[[embed]]: ((<uid of the [NOVA] block>))}}`. The guidelines' root-level rule for daily pages applies to memories; Nova responses go under this heading.
390
+
391
+ ### Idempotency
392
+ - A block that already has a `[NOVA]` child is answered. Do not answer it again, even if the tag still reads `#@Nova`. The nightly cleanup pass removes stray duplicates; do not create them.
393
+ - `#@Nova-DONE` is never a work queue. Search `@Nova` for open items and skip anything containing `-DONE`.
394
+
395
+ ### Who runs when (America/Los_Angeles)
396
+ | When | Job | Scope |
397
+ |---|---|---|
398
+ | 03:00 nightly | AI Intel sweep, `Skills/AIIntel/Tools/ProcessTriage.ts` | `#@Nova` and `#Status/*` on the `AI Intel Report` and `PAI-Anthropic Report` pages |
399
+ | 04:30 nightly | `Skills/Productivity/scripts/ProcessNovaTodos.ts`, EventBridge job `nova_todo_processing` | `#@Nova` on every other page |
400
+ | Interactive | This session, when Ian points at a block or asks for a sweep | Same output shape as above |
401
+
402
+ - Both nightly jobs post their own Telegram summary. An interactive answer does not need one.
403
+ - Dry run by hand: `bun $PAI_DIR/Skills/Productivity/scripts/ProcessNovaTodos.ts --dry-run`. Add `--cleanup` to only remove duplicate `[NOVA]` blocks.
404
+ - On AI Intel pages, `#Status/ExecSum`, `#Status/Explore`, `#Status/Backburner`, and `#Status/Reject` run unattended. `#Status/Implement`, `#Status/PRD`, and `#@Nova` wait for an interactive session. Processed items end as `#Status/Processed`.
405
+
406
+ ## Retired conventions
407
+ - `#PAI/do`, `#PAI/done`, `#PAI/hold`, `#PAI/ignore`: replaced by `#@Nova` and `#@Nova-DONE`. Nothing processes them and no block carried them as of September 6th, 2026. Never create them. If one turns up, treat it as `#@Nova`.
408
+ - The `| [[Date]]` suffix once used to surface nested results on the daily page: replaced by the embed above.
409
+
410
+ ## Ian's capture tags (not Nova's queue)
411
+ - `{{[[TODO]]}} #task <action>` on the daily page is Ian's capture inbox. ProcessFlow routes it to OmniFocus weekly. Do not convert `#task` to `#@Nova` unprompted; Ian decides what to delegate.
412
+ - `#researchThis` means look into this later. It is not a task and not a delegation.
413
+ - `#suggested-next-action` is a follow-up Nova proposes under a reply. Ian adds `#@Nova` to delegate it or checks it `{{[[DONE]]}}` to close it himself.
414
+ </personalization_layer>
@@ -1,26 +1,49 @@
1
+ import { escapeBlockString, needsNewlineEscaping, ESCAPED_NEWLINES_MARKER } from '../../shared/block-escaping.js';
1
2
  /**
2
- * Convert RoamBlock hierarchy to markdown with proper indentation
3
+ * Convert RoamBlock hierarchy to markdown with proper indentation.
4
+ *
5
+ * `escape` mirrors `PageOperations.fetchPageByTitle`'s markdown branch: a
6
+ * block may contain a soft line break (Shift+Enter), and this renderer emits
7
+ * one `- ` line per block, so an unescaped newline spills onto a second
8
+ * physical line with no bullet, resetting the parser's indentation baseline
9
+ * and reparenting everything after it. Callers that render a full page
10
+ * compute whether escaping is needed and set this flag; callers that never
11
+ * feed their output back through `updatePageMarkdown` can leave it off.
3
12
  */
4
- export function blocksToMarkdown(blocks, level = 0) {
13
+ export function blocksToMarkdown(blocks, level = 0, escape = false) {
5
14
  return blocks
6
15
  .map(block => {
7
16
  const indent = ' '.repeat(level);
8
17
  let md;
18
+ const text = escape ? escapeBlockString(block.string) : block.string;
9
19
  // Check block heading level and format accordingly
10
20
  if (block.heading && block.heading > 0) {
11
21
  const hashtags = '#'.repeat(block.heading);
12
- md = `${indent}${hashtags} ${block.string}`;
22
+ md = `${indent}${hashtags} ${text}`;
13
23
  }
14
24
  else {
15
- md = `${indent}- ${block.string}`;
25
+ md = `${indent}- ${text}`;
16
26
  }
17
27
  if (block.children && block.children.length > 0) {
18
- md += '\n' + blocksToMarkdown(block.children, level + 1);
28
+ md += '\n' + blocksToMarkdown(block.children, level + 1, escape);
19
29
  }
20
30
  return md;
21
31
  })
22
32
  .join('\n');
23
33
  }
34
+ /**
35
+ * Collect every block string in a tree, for deciding whether a page needs
36
+ * newline escaping at all (see `blocksToMarkdown`).
37
+ */
38
+ function collectBlockStrings(blocks, out = []) {
39
+ for (const block of blocks) {
40
+ out.push(block.string);
41
+ if (block.children && block.children.length > 0) {
42
+ collectBlockStrings(block.children, out);
43
+ }
44
+ }
45
+ return out;
46
+ }
24
47
  /**
25
48
  * Flatten block hierarchy to single-level list
26
49
  */
@@ -42,7 +65,11 @@ export function formatPageOutput(title, blocks, options) {
42
65
  return JSON.stringify({ title, children: data }, null, 2);
43
66
  }
44
67
  const displayBlocks = options.flat ? flattenBlocks(blocks) : blocks;
45
- return `# ${title}\n\n${blocksToMarkdown(displayBlocks)}`;
68
+ const escape = needsNewlineEscaping(collectBlockStrings(displayBlocks));
69
+ const body = blocksToMarkdown(displayBlocks, 0, escape);
70
+ return escape
71
+ ? `# ${title}\n${ESCAPED_NEWLINES_MARKER}\n\n${body}`
72
+ : `# ${title}\n\n${body}`;
46
73
  }
47
74
  /**
48
75
  * Format block content for output
@@ -53,7 +80,9 @@ export function formatBlockOutput(block, options) {
53
80
  return JSON.stringify(data, null, 2);
54
81
  }
55
82
  const displayBlocks = options.flat ? flattenBlocks([block]) : [block];
56
- return blocksToMarkdown(displayBlocks);
83
+ const escape = needsNewlineEscaping(collectBlockStrings(displayBlocks));
84
+ const body = blocksToMarkdown(displayBlocks, 0, escape);
85
+ return escape ? `${ESCAPED_NEWLINES_MARKER}\n\n${body}` : body;
57
86
  }
58
87
  /**
59
88
  * Format search results for output
@@ -111,13 +111,18 @@ function convertToRoamMarkdown(text) {
111
111
  * That was a real, unrecoverable defect: `- wrap it in ``` to make code`
112
112
  * followed by three blocks parsed to a single block "wrap it in", and
113
113
  * roam_update_page_markdown deleted the other three. Roam has no undo.
114
+ *
115
+ * This subsumes the earlier balanced-fence guard: a line carrying both an
116
+ * opening and a closing fence necessarily has content after the opener, so it
117
+ * never matches this shape.
114
118
  */
115
119
  function isBulletFenceOpener(trimmedLine) {
116
120
  return /^\s*[-*+]\s+```[A-Za-z0-9_+-]*\s*$/.test(trimmedLine);
117
121
  }
118
122
  /**
119
123
  * A fence line with content after its opening ``` is content, not a region
120
- * opener. Guards the bare (non-bullet) case the splice rule cannot see.
124
+ * opener. Guards the bare (non-bullet) case that the splice rule cannot see,
125
+ * e.g. a line rendered as ```js\ncode\n``` with no bullet prefix.
121
126
  */
122
127
  function fenceHasTrailingContent(trimmedLine) {
123
128
  const open = trimmedLine.indexOf('```');
@@ -135,12 +140,14 @@ function parseMarkdown(markdown) {
135
140
  const codeStartIndex = trimmedLine.indexOf('```');
136
141
  if (codeStartIndex > 0 && isBulletFenceOpener(trimmedLine)) {
137
142
  // Under this rule the text before the fence is ALWAYS just the bullet
138
- // marker, so there is no real content to preserve as its own node.
139
- // Pushing it anyway left a bare "-" line that the parser could not
140
- // recognise as a bullet once trimmed, so it emitted a spurious "-" block
141
- // ahead of the code block it introduces. Dropping it loses nothing: the
142
- // fence line below carries the same leading whitespace, so
143
- // indentation-based nesting is unaffected.
143
+ // marker (isBulletFenceOpener only matches "bullet + nothing but the
144
+ // fence"), so there is no real content to preserve as its own node.
145
+ // Pushing it anyway used to leave a bare "-" line that the parser can't
146
+ // recognise as a bullet once trimmed (no trailing content survives
147
+ // trimEnd), so it fell through to the plain-line branch and emitted a
148
+ // spurious "-" block ahead of the code block it introduces. Dropping it
149
+ // loses nothing: the fence line below carries the same leading
150
+ // whitespace, so indentation-based nesting is unaffected.
144
151
  const indentationWhitespace = line.match(/^\s*/)?.[0] ?? '';
145
152
  processedLines.push(indentationWhitespace + trimmedLine.substring(codeStartIndex));
146
153
  }
@@ -331,7 +331,7 @@ export class RoamServer {
331
331
  res.setHeader('Access-Control-Allow-Origin', '*');
332
332
  }
333
333
  res.setHeader('Access-Control-Allow-Methods', 'GET, POST, DELETE, OPTIONS');
334
- res.setHeader('Access-Control-Allow-Headers', 'Content-Type, Authorization, Mcp-Session-Id');
334
+ res.setHeader('Access-Control-Allow-Headers', 'Content-Type, Authorization, Mcp-Session-Id, MCP-Protocol-Version, Last-Event-ID');
335
335
  res.setHeader('Access-Control-Expose-Headers', 'Mcp-Session-Id');
336
336
  res.setHeader('Access-Control-Allow-Credentials', 'true');
337
337
  // Handle preflight OPTIONS requests
@@ -0,0 +1,74 @@
1
+ /**
2
+ * Encoding a Roam block string so it occupies exactly one line of markdown.
3
+ *
4
+ * A Roam block may contain a newline — a soft line break (Shift+Enter). Our
5
+ * markdown renderers emit one `- ` line per block, so an unescaped newline
6
+ * spills onto a second physical line with no bullet and no indentation. The
7
+ * parser then reads it as a separate block AT ROOT LEVEL, which resets the
8
+ * indentation baseline and reparents everything after it. A single soft line
9
+ * break flattens the rest of the page.
10
+ *
11
+ * Keeping each block on one line is therefore not cosmetic: it is what makes
12
+ * the round trip preserve hierarchy at all.
13
+ *
14
+ * A soft break is rendered as a single sentinel character, U+23CE `⏎`, rather
15
+ * than a backslash escape. Two earlier revisions used `\n` (with backslash
16
+ * doubling to disambiguate a literal backslash-n); both corrupted ordinary
17
+ * content, because `\n` is a common PREFIX in authored text — LaTeX commands
18
+ * (`\nabla`, `\neq`) and Windows paths (`C:\new…`) all begin with it, so
19
+ * decoding `\n` anywhere corrupted them. `⏎` essentially never occurs in
20
+ * authored text, so decoding it is safe even for blocks the agent wrote
21
+ * itself. No backslash rules exist at all any more.
22
+ *
23
+ * See `docs/multiline-block-roundtrip-spec.md`.
24
+ */
25
+ /**
26
+ * The single-line stand-in for a soft line break: U+23CE RETURN SYMBOL.
27
+ *
28
+ * Revision 2 used `\n` escapes with backslash doubling. That failed twice:
29
+ * `\n` is a common PREFIX in ordinary content (`\nabla`, `\neq`, `C:\new…`),
30
+ * so decoding it anywhere an agent may have authored text corrupted LaTeX and
31
+ * Windows paths — even the marker could not make it safe per-block. `⏎`
32
+ * essentially never occurs in authored text, so decoding it inside a marked
33
+ * payload is safe INCLUDING for blocks the agent wrote. No backslash rules
34
+ * exist at all any more.
35
+ *
36
+ * The accepted corner: a block genuinely containing a literal `⏎` round-trips
37
+ * it into a newline. Content-level, degrades to a soft break, documented, and
38
+ * pinned by a test so it stays a choice rather than an accident.
39
+ */
40
+ export const SOFT_BREAK_SENTINEL = '⏎';
41
+ /** Render a block string as a single line. Identity for newline-free text. */
42
+ export function escapeBlockString(text) {
43
+ return text.includes('\n') ? text.replace(/\n/g, SOFT_BREAK_SENTINEL) : text;
44
+ }
45
+ /** Restore soft line breaks. Call ONLY on marker-carrying payloads. */
46
+ export function unescapeBlockString(text) {
47
+ return text.includes(SOFT_BREAK_SENTINEL)
48
+ ? text.replace(/⏎/g, '\n')
49
+ : text;
50
+ }
51
+ /**
52
+ * Marks a rendered payload whose block strings were escaped.
53
+ *
54
+ * Revision 1 decoded `\n` unconditionally, which corrupted every LaTeX command
55
+ * and Windows path that begins `\n` — `$$\nabla f$$` became `$$<newline>abla
56
+ * f$$`. An escape sequence built from characters that occur in ordinary content
57
+ * can only be decoded where the ENCODER is known to have run. This marker is
58
+ * how the decoder knows.
59
+ *
60
+ * An HTML comment on purpose: inert everywhere, and harmless if it ever lands
61
+ * in a block by accident.
62
+ */
63
+ export const ESCAPED_NEWLINES_MARKER = '<!-- roam:escaped-newlines -->';
64
+ /**
65
+ * Does this page need escaping at all?
66
+ *
67
+ * Escaping is conditional so a page with no multi-line block renders exactly
68
+ * as it did before any of this work — no marker, no backslash doubling. That
69
+ * is the overwhelming majority of pages, and it keeps the blast radius of the
70
+ * encoding near zero.
71
+ */
72
+ export function needsNewlineEscaping(blockStrings) {
73
+ return blockStrings.some((s) => s.includes('\n'));
74
+ }
@@ -4,6 +4,7 @@ import { getPageUid as getPageUidHelper } from '../helpers/page-resolution.js';
4
4
  import { resolveRefs } from '../helpers/refs.js';
5
5
  import { fetchChildrenByDepth } from '../helpers/fetch-children.js';
6
6
  import { collectHiddenUids, pruneHiddenBlocks, isHiddenBlockString } from '../helpers/hidden.js';
7
+ import { escapeBlockString } from '../../shared/block-escaping.js';
7
8
  export class FullPageViewOperations {
8
9
  constructor(graph, pageOps) {
9
10
  this.graph = graph;
@@ -107,7 +108,11 @@ export class FullPageViewOperations {
107
108
  });
108
109
  }
109
110
  const linkedReferenceGroups = Array.from(groupMap.values());
110
- // 7. Render as markdown
111
+ // 7. Render as markdown. This output is display-only — never valid
112
+ // `roam_update_page_markdown` input, and nothing decodes it — so
113
+ // `renderBlocks` applies the `⏎` sentinel to every block string
114
+ // unconditionally (it's the identity for newline-free text) rather than
115
+ // gating on a per-page predicate.
111
116
  return this.renderMarkdown(title, pageBlocks, linkedReferenceGroups, truncated ? allUniqueRefs.length : undefined);
112
117
  }
113
118
  // ─── Private: fetch all blocks that reference this page ──────────────────────
@@ -268,11 +273,17 @@ export class FullPageViewOperations {
268
273
  // This mirrors Roam's ancestor context display
269
274
  for (let i = 0; i < ref.breadcrumbs.length; i++) {
270
275
  const prefix = '> '.repeat(i + 1);
271
- lines.push(`${prefix}${ref.breadcrumbs[i].string}`);
276
+ // Same reasoning as `renderBlocks` below: a breadcrumb string can
277
+ // itself carry a soft line break, and an unescaped one spills onto
278
+ // a bare physical line with no `> ` prefix at all.
279
+ lines.push(`${prefix}${escapeBlockString(ref.breadcrumbs[i].string)}`);
272
280
  }
273
- // The referring block itself, indented to sit visually under its breadcrumbs
281
+ // The referring block itself, indented to sit visually under its
282
+ // breadcrumbs. Escaped for the same reason `renderBlocks` escapes
283
+ // every other block string in this file: an unescaped newline spills
284
+ // the rest of the block onto a bare physical line with no bullet.
274
285
  const refIndent = ' '.repeat(ref.breadcrumbs.length);
275
- lines.push(`${refIndent}- ${ref.block.string}`);
286
+ lines.push(`${refIndent}- ${escapeBlockString(ref.block.string)}`);
276
287
  // Children of the referring block
277
288
  if (ref.block.children.length > 0) {
278
289
  lines.push(this.renderBlocks(ref.block.children, ref.breadcrumbs.length + 1));
@@ -286,13 +297,17 @@ export class FullPageViewOperations {
286
297
  renderBlocks(blocks, baseIndent) {
287
298
  const renderBlock = (block, depth) => {
288
299
  const indent = ' '.repeat(depth);
300
+ // Unconditional: `escapeBlockString` is the identity for newline-free
301
+ // text, so pages with no soft line breaks render byte-identical to how
302
+ // they did before this feature existed — no predicate needed.
303
+ const text = escapeBlockString(block.string);
289
304
  let line;
290
305
  if (block.heading && block.heading > 0) {
291
306
  const hashes = '#'.repeat(block.heading);
292
- line = `${indent}${hashes} ${block.string}`;
307
+ line = `${indent}${hashes} ${text}`;
293
308
  }
294
309
  else {
295
- line = `${indent}- ${block.string}`;
310
+ line = `${indent}- ${text}`;
296
311
  }
297
312
  const childLines = block.children.map(c => renderBlock(c, depth + 1)).join('\n');
298
313
  return childLines ? `${line}\n${childLines}` : line;
@@ -7,10 +7,11 @@ import { executeBatch, executeBatchSafe } from '../helpers/batch-utils.js';
7
7
  import { pruneHiddenBlocks } from '../helpers/hidden.js';
8
8
  import { convertToRoamMarkdown, generateBlockUid } from '../../markdown-utils.js';
9
9
  import { executeStagedBatch } from '../../shared/staged-batch.js';
10
+ import { escapeBlockString, unescapeBlockString, needsNewlineEscaping, ESCAPED_NEWLINES_MARKER, SOFT_BREAK_SENTINEL } from '../../shared/block-escaping.js';
10
11
  import { pageUidCache } from '../../cache/page-uid-cache.js';
11
12
  import { buildTableActions } from './table.js';
12
13
  import { BatchOperations } from './batch.js';
13
- import { parseExistingBlocks, pruneHiddenExistingBlocks, countHiddenExistingBlocks, markdownToBlocks, diffBlockTrees, generateBatchActions, getDiffStats, isDiffEmpty, summarizeActions, } from '../../diff/index.js';
14
+ import { parseExistingBlocks, pruneHiddenExistingBlocks, countHiddenExistingBlocks, flattenExistingBlocks, markdownToBlocks, diffBlockTrees, generateBatchActions, getDiffStats, isDiffEmpty, summarizeActions, } from '../../diff/index.js';
14
15
  // Helper to get ordinal suffix for dates
15
16
  function getOrdinalSuffix(day) {
16
17
  if (day > 3 && day < 21)
@@ -419,7 +420,17 @@ export class PageOperations {
419
420
  // renders a page (markdown, raw, structure) inherits it.
420
421
  return { title, blocks: pruneHiddenBlocks(rootBlocks) };
421
422
  }
422
- async fetchPageByTitle(title, format = 'raw') {
423
+ async fetchPageByTitle(title, format = 'raw',
424
+ /**
425
+ * Encode newlines so each block is one line (`shared/block-escaping.ts`).
426
+ * Required by anything whose output may be fed back to
427
+ * `roam_update_page_markdown`; wrong for anything shown as prose.
428
+ *
429
+ * Defaults to OFF so a caller that has not considered this renders today's
430
+ * output rather than silently acquiring doubled backslashes. `guidelines.ts`
431
+ * relies on that default.
432
+ */
433
+ options = {}) {
423
434
  if (!title) {
424
435
  throw new McpError(ErrorCode.InvalidRequest, 'title is required');
425
436
  }
@@ -592,21 +603,36 @@ export class PageOperations {
592
603
  await Promise.all(visibleBlocks.map(async (b) => {
593
604
  b.string = await resolveRefs(this.graph, b.string);
594
605
  }));
606
+ // Collect every visible block string to decide whether this page needs the
607
+ // encoding at all. A page with no soft line break renders exactly as it
608
+ // did before this feature existed.
609
+ const allStrings = [];
610
+ const collectStrings = (blocks) => {
611
+ for (const b of blocks) {
612
+ allStrings.push(b.string);
613
+ collectStrings(b.children);
614
+ }
615
+ };
616
+ collectStrings(visibleRoots);
617
+ const escaping = options.escapeNewlines === true && needsNewlineEscaping(allStrings);
595
618
  // Convert to markdown with proper nesting
596
619
  const toMarkdown = (blocks, level = 0) => {
597
620
  return blocks
598
621
  .map(block => {
599
622
  const indent = ' '.repeat(level);
600
623
  let md;
624
+ const text = escaping
625
+ ? escapeBlockString(block.string)
626
+ : block.string;
601
627
  // Check block heading level and format accordingly
602
628
  if (block.heading && block.heading > 0) {
603
629
  // Format as heading with appropriate number of hashtags
604
630
  const hashtags = '#'.repeat(block.heading);
605
- md = `${indent}${hashtags} ${block.string}`;
631
+ md = `${indent}${hashtags} ${text}`;
606
632
  }
607
633
  else {
608
634
  // No heading, use bullet point (current behavior)
609
- md = `${indent}- ${block.string}`;
635
+ md = `${indent}- ${text}`;
610
636
  }
611
637
  if (block.children.length > 0) {
612
638
  md += '\n' + toMarkdown(block.children, level + 1);
@@ -615,7 +641,10 @@ export class PageOperations {
615
641
  })
616
642
  .join('\n');
617
643
  };
618
- return `# ${title}\n\n${toMarkdown(visibleRoots)}`;
644
+ const body = toMarkdown(visibleRoots);
645
+ return escaping
646
+ ? `# ${title}\n${ESCAPED_NEWLINES_MARKER}\n\n${body}`
647
+ : `# ${title}\n\n${body}`;
619
648
  }
620
649
  /**
621
650
  * Update an existing page with new markdown content using smart diff.
@@ -664,7 +693,108 @@ export class PageOperations {
664
693
  const hiddenCount = countHiddenExistingBlocks(allExistingBlocks);
665
694
  const existingBlocks = pruneHiddenExistingBlocks(allExistingBlocks);
666
695
  // 4. Convert new markdown to block structure
667
- const newBlocks = markdownToBlocks(markdown, pageUid);
696
+ //
697
+ // Decode ONLY when our own renderer said it encoded. The marker may be
698
+ // the first non-empty line (header stripped by the caller) or the first
699
+ // non-empty line after a single leading `#` header (payload submitted
700
+ // verbatim — the path `roam save --update` takes). Revision 2 checked
701
+ // only the first line; a verbatim submit therefore never decoded, wrote
702
+ // the marker as a block, and deleted the blocks it rewrote. The tests
703
+ // that should have caught it stripped the header themselves.
704
+ const lines = markdown.split('\n');
705
+ let markerAt = -1;
706
+ let nonEmptySeen = 0;
707
+ for (let i = 0; i < lines.length; i++) {
708
+ const t = lines[i].trim();
709
+ if (t.length === 0)
710
+ continue;
711
+ nonEmptySeen++;
712
+ if (t === ESCAPED_NEWLINES_MARKER) {
713
+ markerAt = i;
714
+ break;
715
+ }
716
+ // A single leading header line may precede the marker; anything else
717
+ // (or a second line that is not the marker) means an unmarked payload.
718
+ if (nonEmptySeen === 1 && t.startsWith('#'))
719
+ continue;
720
+ break;
721
+ }
722
+ const isEscaped = markerAt !== -1;
723
+ // `fetchPageByTitle`'s markdown branch always prepends `# ${title}\n` --
724
+ // it is page-level metadata describing what page this is, never content.
725
+ // Tolerating the marker after that header (above) is not enough on its
726
+ // own: a verbatim round trip still hands that literal `# Title` line to
727
+ // the parser, which turns it into a real heading block with no match in
728
+ // the existing tree, so the diff creates it and reparents every sibling
729
+ // after it -- exactly the "no-op round trip" this marker exists to
730
+ // guarantee. Strip it whenever the first non-empty line matches this
731
+ // page's own title exactly -- but ONLY when the payload demonstrably came
732
+ // from our own renderer (`isEscaped`, or the body still carries the
733
+ // `⏎` sentinel -- the signature of a marker-dropped degradation, since
734
+ // that scenario is "an agent rebuilt our output and lost the marker" and
735
+ // the sentinel is what that rebuilding could not have removed). A FRESH,
736
+ // hand-authored page can legitimately open with an H1 that echoes its own
737
+ // title (imported docs; an author who titles their own first line) --
738
+ // essentially never with `⏎` in it. The two wrong calls are not
739
+ // symmetric: preserving the header when it should have been stripped
740
+ // creates a harmless, visible stray block; stripping it when it should
741
+ // not have been touched deletes -- or reparents into corruption -- real,
742
+ // unrecoverable content, with no undo. The gate always takes the harmless
743
+ // direction whenever the payload carries no provenance signal.
744
+ const trimmedTitle = String(title).trim();
745
+ const firstNonEmptyAt = lines.findIndex((l) => l.trim().length > 0);
746
+ const hasRendererProvenance = isEscaped || markdown.includes(SOFT_BREAK_SENTINEL);
747
+ const titleHeaderAt = hasRendererProvenance &&
748
+ firstNonEmptyAt !== -1 &&
749
+ lines[firstNonEmptyAt].trim() === `# ${trimmedTitle}`
750
+ ? firstNonEmptyAt
751
+ : -1;
752
+ let effectiveMarkdown = markdown;
753
+ const linesToStrip = [isEscaped ? markerAt : -1, titleHeaderAt]
754
+ .filter((i) => i !== -1)
755
+ .sort((a, b) => b - a);
756
+ if (linesToStrip.length > 0) {
757
+ // Strip the marker line itself, or it becomes a stray block on the
758
+ // page. Descending order so removing one index never shifts the other.
759
+ for (const idx of linesToStrip)
760
+ lines.splice(idx, 1);
761
+ effectiveMarkdown = lines.join('\n');
762
+ }
763
+ // Decode AFTER parsing, per block -- never on the whole document before
764
+ // parsing. Decoding the document first would turn every `\n` into a real
765
+ // newline and then split on newlines, which is exactly the flattening
766
+ // bug this whole effort exists to fix.
767
+ const newBlocks = markdownToBlocks(effectiveMarkdown, pageUid);
768
+ // Submitting empty/whitespace markdown is the documented way to clear a
769
+ // page, and stays untouched below. This guards the DIFFERENT case: markdown
770
+ // that is NOT empty but still parsed to zero blocks. That only happens when
771
+ // the parser swallowed the payload -- the known cause is a first block
772
+ // whose string is a bare fence opener (e.g. a line reading "- ```js" with
773
+ // nothing to close it), which is exactly the shape our own renderer emits
774
+ // for a Roam block whose string starts with a fence, and the shape a
775
+ // pasted code snippet produces. `markdownToBlocks` then returns an empty
776
+ // array, and diffing 0 new blocks against N existing ones deletes all N --
777
+ // silently, against an API with no undo. The asymmetry is deliberate: an
778
+ // empty parse of EMPTY input is the documented clear-the-page instruction;
779
+ // an empty parse of NON-EMPTY input is the parser losing the payload, never
780
+ // an instruction to delete anything.
781
+ if (effectiveMarkdown.trim().length > 0 && newBlocks.length === 0) {
782
+ const existingCount = flattenExistingBlocks(existingBlocks).length;
783
+ if (existingCount > 0) {
784
+ throw new McpError(ErrorCode.InvalidParams, `markdown parsed to zero blocks even though it is not empty. This usually ` +
785
+ `means an unterminated \`\`\` fence swallowed the whole payload (a first ` +
786
+ `block that opens a code fence and never closes it consumes every line ` +
787
+ `after it). Refusing to write: this would have deleted all ${existingCount} ` +
788
+ `existing block${existingCount === 1 ? '' : 's'} on "${title}". Check the ` +
789
+ `markdown for an unbalanced fence, or call again with dry_run: true to ` +
790
+ `inspect the planned actions before writing.`);
791
+ }
792
+ }
793
+ if (isEscaped) {
794
+ for (const block of newBlocks) {
795
+ block.text = unescapeBlockString(block.text);
796
+ }
797
+ }
668
798
  // 5. Compute diff
669
799
  const diff = diffBlockTrees(existingBlocks, newBlocks, pageUid);
670
800
  // 6. Generate ordered batch actions
@@ -41,7 +41,9 @@ export class ToolHandlers {
41
41
  return this.pageOps.createPage(title, content);
42
42
  }
43
43
  async fetchPageByTitle(title, format) {
44
- return this.pageOps.fetchPageByTitle(title, format);
44
+ // The MCP read tool's markdown may be handed straight back to
45
+ // roam_update_page_markdown, so it must be round-trippable.
46
+ return this.pageOps.fetchPageByTitle(title, format, { escapeNewlines: true });
45
47
  }
46
48
  async fetchPageFullView(title, children_depth, max_references) {
47
49
  return this.fullPageViewOps.fetchPageFullView(title, children_depth, max_references);
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "roam-research-mcp",
3
- "version": "3.2.0",
3
+ "version": "4.0.1",
4
4
  "description": "MCP server and CLI for Roam Research",
5
5
  "private": false,
6
6
  "repository": {
@@ -31,6 +31,7 @@
31
31
  "scripts": {
32
32
  "build": "echo \"Using custom instructions: .roam/${CUSTOM_INSTRUCTIONS_PREFIX}custom-instructions.md\" && tsc && cat Roam_Markdown_Cheatsheet.md .roam/${CUSTOM_INSTRUCTIONS_PREFIX}custom-instructions.md > build/Roam_Markdown_Cheatsheet.md && chmod 755 build/index.js build/cli/roam.js",
33
33
  "clean": "rm -rf build",
34
+ "prepublishOnly": "node scripts/check-bundled-cheatsheet.mjs",
34
35
  "watch": "tsc --watch",
35
36
  "inspector": "npx @modelcontextprotocol/inspector build/index.js",
36
37
  "start": "node build/index.js",
@@ -45,7 +46,7 @@
45
46
  "test:watch": "vitest"
46
47
  },
47
48
  "dependencies": {
48
- "@modelcontextprotocol/sdk": "^1.13.2",
49
+ "@modelcontextprotocol/sdk": "1.25.1",
49
50
  "@roam-research/roam-api-sdk": "^0.10.0",
50
51
  "commander": "^14.0.2",
51
52
  "dotenv": "^16.4.7"