@geml/logseq-sync 2.0.0

package/LICENSE

@@ -0,0 +1,28 @@
+MIT License
+
+Copyright (c) 2026 GEML contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
+---
+
+NOTE: This MIT license covers the *code* in this repository (geml-parser/,
+integrations/geml-viewer/, integrations/geml-check-action/, docs/examples/
+tooling). The *specification* documents are licensed separately under
+CC-BY-4.0 — see LICENSE-spec.md.

package/README.md

@@ -0,0 +1,212 @@
+# Sync Vault with GEML
+
+Your Logseq DB graph as a **continuously synced, Git-friendly plain-text
+vault** — pages and journals back in readable files and folders, the way OG
+vaults felt, kept in step with the database.
+
+![How it works](docs/how-it-works.svg)
+
+## What you get
+
+- 🌿 **Real git workflows** — clean commits, readable line-by-line diffs, full
+  version history for a DB graph
+- 📦 **A plain-text escape hatch that stays yours** — every page a readable
+  file, not a database dump
+- 🔁 **Continuous, not one-shot** — edit in Logseq, and seconds later the file
+  and its git commit exist
+
+Logseq 2.0 ships both ends of a trade-off: `logseq export` gives Markdown
+(readable, lossy) and `logseq export-edn` gives EDN (lossless, not something a
+person edits). The vault's format, [GEML](https://github.com/geml-spec/geml),
+is the point between: **as readable as the Markdown export, as lossless as the
+EDN one** — and addressable, so external tools and agents can edit one block
+of a graph instead of round-tripping all of it.
+
+The tree is laid out the way an OG vault is — the thing a file-version user
+recognizes as "my graph, as files again":
+
+```
+{:pages-and-blocks [...]}          ontology.geml            :properties/:classes, verbatim EDN
+                          ⇄        graph.geml               page ORDER (an addressable data block,
+                                                            so filenames need no numeric prefixes)
+                                   journals/2025_02_20.geml journal pages, OG date names
+                                   pages/<name>.geml        one per page:
+                                     block title  → `=== text` body
+                                     block uuid   → `{#uuid}`  ← geml get/set address
+                                     outline tree → flat blocks with `level=N`
+                                     everything else rides along in `code {lang=edn}`
+```
+
+(`@logseq/cli` 0.4.3's `export-edn` does not include journal pages, so live
+exports show `pages/` only today; the journal mapping is fixture-tested.)
+
+## How it works — two halves, one honest boundary
+
+A Logseq 2.0 plugin runs in a sandboxed iframe: no arbitrary-path filesystem,
+no git, no shell (verified against the 2.0.1 app bundle). So the in-app plugin
+(`plugin/`) does the only two things only it can do:
+
+- **hear** the graph change (`logseq.DB.onChanged`, debounced) and write a
+  dirty-marker file through the plugin storage API;
+- **show** the last sync result in the toolbar (`⇄`) and command palette.
+
+Everything with side effects lives in the **watcher** (`watcher/bin/geml-sync.mjs`),
+built on the official `@logseq/cli` export. It reacts to the marker file
+immediately (interval polling stays on as a fallback), writes only the files
+that actually changed — so `git diff` is never noise — commits with a pathspec
+scoped strictly to the vault, and reports back for the toolbar to display.
+The two halves meet in the plugin's own storage directory
+(`<dotdir>/storages/logseq-plugin-sync-vault-with-geml/`), the one disk location both can
+reach. A file as the bridge beats a local HTTP API: no port, no server, no
+CORS.
+
+Real output, real DB graph (exported with the official CLI, validated by
+`logseq validate`):
+
+```text
+$ node watcher/bin/geml-sync.mjs geml-spike ~/vault-demo --git-commit --signal <storage>/geml-sync-dirty.json
+Starting GEML Sync: Graph "geml-spike" ➔ ~/vault-demo
+Git auto-commit: enabled (scoped to target paths)
+[19:29:14] Synced: 8 written, 0 unchanged.
+  Git: [master (root-commit) 9cc348c] logseq-geml: sync graph "geml-spike"
+ 9 files changed, 142 insertions(+)
+ create mode 100644 graph.geml
+ create mode 100644 pages/contents.geml
+ ...
+
+$ node watcher/bin/geml-sync.mjs geml-spike ~/vault-demo --git-commit --signal ...   # run again
+[19:29:55] Graph is up-to-date (0 written, 8 unchanged).
+```
+
+## Setup
+
+**1. Install the plugin** from the marketplace — or build and load it
+unpacked (`dist/` is not checked in):
+
+```sh
+cd plugin && npm install && npm run build
+```
+
+then Settings → Advanced → Developer mode → "Load unpacked plugin" → `plugin/`.
+
+**2. Get the watcher** — one npm install, gives you the `geml-sync` command:
+
+```sh
+npm install -g @geml/logseq-sync
+```
+
+(or run it ad hoc with `npx @geml/logseq-sync …`; the source lives in this
+repository under `watcher/` and `core/`)
+
+**3. Install `@logseq/cli`** (one time, anywhere). On Node 24 its
+`better-sqlite3` has no prebuilt binding until 12.11.1, so pin an override
+(without it, install tries to compile and node-gyp does not recognize
+VS 2026 yet):
+
+```sh
+mkdir logseq-cli && cd logseq-cli && npm init -y
+npm pkg set overrides.better-sqlite3=12.11.1
+npm i @logseq/cli
+```
+
+**4. Run the watcher**, with `LOGSEQ_CLI_DIR` pointing at that directory and
+`--signal` pointing at this plugin's storage directory:
+
+```sh
+geml-sync <your-graph> <your-vault-dir> --watch --git-commit \
+  --signal <logseq-dotdir>/storages/logseq-plugin-sync-vault-with-geml/geml-sync-dirty.json
+```
+
+Edit a block in Logseq → the plugin signals → the watcher syncs → the toolbar
+`⇄` button shows `Sync Vault with GEML: last sync at … — 1 written, 7 unchanged.`
+
+**Settings**: *Debounce (seconds)* — quiet period after the last change before
+the watcher is signalled (default 5; syncs feed git commits, so this is
+deliberately calmer than UI-style debounce).
+
+## Honesty corner
+
+- Sync is **export-direction** today (graph → files, continuously). The
+  write-back path (edit a `.geml` file → import back by UUID) is proven in the
+  engine (`syncDiskToEdn`) and lands next; deletions are reported, never
+  auto-propagated (`--signal` never deletes your hand-written files either — a
+  manifest tracks what the sync owns).
+- Journal pages appear as soon as `@logseq/cli` exports them (0.4.3 does not).
+- The watcher half is tested end-to-end in CI (a planted fake CLI exports
+  fixture EDN, so the signal → re-sync → status round trip runs with no Logseq
+  installed). The in-app half is verified against the 2.0.1 runtime — the
+  plugin API surface, `hook:db:changed`, the storage-file bridge — and its
+  SDK is `@logseq/libs` 0.3.x (the `next` tag). If anything misbehaves in
+  your setup, an issue with your Logseq version is gold.
+
+## Proven on a live DB graph, judged by Logseq's own validator
+
+`npm test` proves, on fixtures lifted from Logseq's own `deps/db` export tests:
+
+1. **EDN → GEML → EDN is a structural identity** (EDN map/set semantics).
+2. Every generated document parses as GEML with **zero error diagnostics**.
+3. A block Logseq considers addressable (exported uuid) is **addressable in
+   GEML by the same id**.
+4. **Editing one block's text changes exactly that block** in the EDN — no
+   collateral change anywhere in the graph.
+
+And `bin/live-roundtrip.mjs` has confirmed all four against a real DB graph
+(2026-08-20, schema 65.22): export → 6 clean documents → identity; then with
+`--edit`, a `geml set` on one block imported back with `logseq import-edn`,
+**`logseq validate`: Valid!**, and the re-export showed the edit landed **in
+place by uuid, exactly once — whole-graph re-import merges, it does not
+duplicate**.
+
+The design and the reasoning live in the
+[GEML monorepo](https://github.com/geml-spec/geml)
+(`docs/design/specs/2026-08-20-logseq-integration-scoping.md`); the community
+threads are
+[logseq/logseq#13086](https://github.com/logseq/logseq/discussions/13086) and
+[the forum post](https://discuss.logseq.com/t/35193).
+
+## Development
+
+```
+core/      converter (mapping.mjs), sync engine, bridge.mjs (the signal/status file contract)
+watcher/   the geml-sync CLI and its end-to-end tests — published to npm as @geml/logseq-sync
+plugin/    the in-app half (this package.json is the Logseq plugin manifest)
+```
+
+Source of truth is
+[`integrations/logseq/`](https://github.com/geml-spec/geml/tree/main/integrations/logseq)
+in the GEML monorepo; this repository mirrors it for the marketplace and
+carries the releases. Please open issues here, and PRs against the monorepo.
+
+The converter is two pure functions in `core/src/mapping.mjs` —
+`ednToGemlFiles(ednText)` and `gemlFilesToEdn(files, lib)` — with the reference
+parser injected. The tests import the parser's build:
+
+```sh
+cd geml-parser && npm install && npm run build && cd ../integrations/logseq
+npm install
+npm test
+```
+
+Live-stage demos (need `@logseq/cli` via `LOGSEQ_CLI_DIR`, see Setup step 3):
+
+```sh
+node watcher/bin/create-graph.mjs my-graph      # create a DB graph WITHOUT the desktop app
+node watcher/bin/live-roundtrip.mjs my-graph            # read-only: export → GEML → back → compare
+node watcher/bin/live-roundtrip.mjs my-graph --edit     # + geml set → import-edn → logseq validate
+```
+
+Versioning: the MAJOR version tracks the Logseq major it targets — this is
+2.x because it speaks Logseq 2.x (DB graphs) and nothing older. Minor/patch
+are this package's own.
+
+## Next
+
+- **Reference translation**: block refs in titles are literally `[[<uuid>]]`,
+  one character away from GEML's checked `[[#uuid]]` — translating them lets
+  `geml check` catch broken block refs, the actual headline of the proposal.
+- Property readability: scalar `:build/properties` as GEML attributes instead
+  of the `.block-meta` EDN ride-along (NAME rules permitting).
+- **Write-back**: wiring `syncDiskToEdn` to the CLI so the vault is
+  two-way — edit the file, the graph follows.
+
+MIT © GEML contributors

package/core/src/bridge.mjs

@@ -0,0 +1,8 @@
+// The contract between the two halves of Sync Vault with GEML. The in-app plugin
+// writes SIGNAL_FILE through logseq.FileStorage; the watcher reacts to it and
+// writes STATUS_FILE back beside it. Both land in the plugin's storage
+// directory (<dotdir>/storages/<plugin-id>/) — the one disk location both
+// sides can reach. These names ARE the protocol: change them only together.
+
+export const SIGNAL_FILE = "geml-sync-dirty.json";
+export const STATUS_FILE = "geml-sync-status.json";

package/core/src/mapping.mjs

@@ -0,0 +1,243 @@
+// EDN ⇄ GEML for Logseq DB graphs.
+//
+// Input is what `logseq export-edn` (@logseq/cli) produces — sqlite.build EDN:
+//
+//   {:pages-and-blocks [{:page {...} :blocks [{:block/title ".." :build/children [..]} ..]} ..]
+//    :properties {..}    ; ontology: property definitions
+//    :classes    {..}}   ; ontology: class/tag definitions
+//
+// The mapping keeps two promises, in this order:
+//
+//   1. LOSSLESS. The round-trip test is EDN → GEML → EDN structural equality
+//      (EDN map/set semantics: entry order does not count). Anything this
+//      version does not give a GEML shape of its own rides along VERBATIM as
+//      EDN inside `code {lang=edn}` blocks — carried, not dropped.
+//   2. ADDRESSABLE where it pays. A block's title becomes the body of a
+//      `=== text` block; a block that has a uuid keeps it as `{#uuid}`, so
+//      `geml get/set` address exactly the blocks Logseq itself considers
+//      addressable (uuids are only exported for referenced blocks).
+//
+// Structure choice: the outline tree is a FLAT sequence of blocks in
+// depth-first order, each carrying `level=N` — a complete encoding of the tree
+// (it is how outlines print), without nesting GEML fences to the outline's
+// depth.
+//
+// Everything runs on edn-data's TYPED representation (keywords as {key}, sets
+// as {set}, maps as {map: [[k,v]..]}, vectors as arrays), so nothing is coerced
+// through JSON and nothing un-EDN-able is invented.
+
+import { parseEDNString, toEDNString } from "edn-data";
+
+// --- typed-EDN helpers -------------------------------------------------------
+
+const kw = (name) => ({ key: name });
+const isKw = (v, name) => v !== null && typeof v === "object" && v.key === name;
+const mapEntries = (m) => (m !== null && typeof m === "object" && Array.isArray(m.map) ? m.map : []);
+const mapGet = (m, name) => {
+  for (const [k, v] of mapEntries(m)) if (isKw(k, name)) return v;
+  return undefined;
+};
+const mapWithout = (m, names) => ({ map: mapEntries(m).filter(([k]) => !names.some((n) => isKw(k, n))) });
+const mapSize = (m) => mapEntries(m).length;
+const edn = (v) => toEDNString(v);
+
+// edn-data renders `#uuid "..."` as a tagged value; accept both spellings.
+const uuidOf = (v) => {
+  if (typeof v === "string") return v;
+  if (v && typeof v === "object") {
+    if (typeof v.uuid === "string") return v.uuid;
+    if (v.tag === "uuid" && typeof v.val === "string") return v.val;
+  }
+  return undefined;
+};
+
+// --- GEML text helpers -------------------------------------------------------
+
+// A fence must be longer than any `=` run opening a line of the body (§3).
+function fenceFor(body) {
+  let longest = 2;
+  for (const m of body.matchAll(/^=+/gm)) longest = Math.max(longest, m[0].length + 1);
+  return "=".repeat(Math.max(3, longest));
+}
+
+function gemlBlock(type, attrs, body) {
+  const f = fenceFor(body);
+  const a = attrs ? ` {${attrs}}` : "";
+  return `${f} ${type}${a}\n${body}\n${f}\n`;
+}
+
+// --- export: EDN → GEML files ------------------------------------------------
+
+// Returns Map<relativePath, gemlText>. Page order is preserved by a numeric
+// filename prefix: :pages-and-blocks is a vector, and order is content.
+export function ednToGemlFiles(ednText) {
+  const top = parseEDNString(ednText);
+  const files = new Map();
+
+  const pages = mapGet(top, "pages-and-blocks") ?? [];
+  const properties = mapGet(top, "properties");
+  const classes = mapGet(top, "classes");
+  const rest = mapWithout(top, ["pages-and-blocks", "properties", "classes"]);
+
+  // Ontology and any top-level keys this version does not model: verbatim.
+  let onto = '=== meta\ntitle = "Logseq graph ontology"\n===\n\n';
+  if (properties !== undefined) onto += gemlBlock("code", "#properties lang=edn", edn(properties));
+  if (classes !== undefined) onto += gemlBlock("code", "#classes lang=edn", edn(classes));
+  if (mapSize(rest) > 0) onto += gemlBlock("code", "#graph-extra lang=edn", edn(rest));
+  files.set("ontology.geml", onto);
+
+  const order = [];
+  pages.forEach((entry) => {
+    const page = mapGet(entry, "page") ?? { map: [] };
+    const blocksVal = mapGet(entry, "blocks");
+    const blocks = blocksVal ?? [];
+    const entryRest = mapWithout(entry, ["page", "blocks"]);
+    // A present-but-empty :blocks is not the same EDN as an absent one, and
+    // real exports write `:blocks []` on block-less pages. An empty vector has
+    // no text blocks to speak for it, so it rides along with the rest.
+    if (Array.isArray(blocksVal) && blocksVal.length === 0) entryRest.map.push([kw("blocks"), []]);
+
+    const title = mapGet(page, "block/title");
+    const journal = mapGet(page, "build/journal");
+    // The tree is laid out the way an OG vault is: journals under `journals/`
+    // with the OG date filename (20250220 → 2025_02_20.geml), everything else
+    // under `pages/` named by the page itself. No numeric prefixes — page
+    // ORDER is content, but it belongs in the graph.geml index, not in
+    // filenames a person has to look at.
+    let path;
+    if (typeof journal === "number") {
+      const j = String(journal);
+      path = `journals/${j.slice(0, 4)}_${j.slice(4, 6)}_${j.slice(6, 8)}.geml`;
+    } else {
+      const nameSeed = typeof title === "string" ? title : "page";
+      const slug = nameSeed.toLowerCase().replace(/[^a-z0-9一-鿿]+/gu, "-").replace(/^-+|-+$/g, "") || "page";
+      path = `pages/${slug}.geml`;
+    }
+    // Two titles may slug identically; the index carries order and identity,
+    // so filenames only need to be unique.
+    for (let n = 2; files.has(path); n++) path = path.replace(/\.geml$/, "") .replace(/-\d+$/, "") + `-${n}.geml`;
+    order.push(path);
+
+    // The page's identity, verbatim — reconstruction reads THIS; the heading
+    // below is presentation, not data.
+    let out = gemlBlock("code", ".page-meta lang=edn", edn(page));
+    if (mapSize(entryRest) > 0) out += gemlBlock("code", ".page-extra lang=edn", edn(entryRest));
+    if (typeof title === "string") out += `\n# ${title}\n\n`;
+
+    const walk = (bs, level) => {
+      for (const b of bs) {
+        const btitle = mapGet(b, "block/title");
+        const children = mapGet(b, "build/children") ?? [];
+        const meta = mapWithout(b, ["block/title", "build/children"]);
+        // The uuid stays inside the meta EDN too — losslessness never depends
+        // on the id attribute; `{#uuid}` is the ADDRESS.
+        const u = uuidOf(mapGet(b, "block/uuid"));
+        const id = u ? `#${u} ` : "";
+        out += gemlBlock("text", `${id}level=${level}`, typeof btitle === "string" ? btitle : edn(btitle ?? null));
+        if (mapSize(meta) > 0) out += gemlBlock("code", ".block-meta lang=edn", edn(meta));
+        walk(children, level + 1);
+      }
+    };
+    walk(blocks, 1);
+    files.set(path, out);
+  });
+
+  // Page order is content (:pages-and-blocks is a vector), but it lives in the
+  // index rather than in filename prefixes: the tree stays human-shaped, and
+  // one addressable block carries what the machine needs.
+  files.set("graph.geml",
+    '=== meta\ntitle = "Logseq graph index"\n===\n\n' +
+    gemlBlock("data", "#page-order", JSON.stringify(order, null, 1)));
+
+  return files;
+}
+
+// --- import: GEML files → EDN ------------------------------------------------
+
+// The parser library is injected ({parse, addressedUnits, sliceUnit} from
+// @geml/geml), so this module stays dependency-light and the caller decides
+// which parser build to trust.
+//
+// Why two reads per document: `parse` gives structure (types, classes, attrs),
+// but a `text` block is FLOW content — its node carries parsed inlines, not
+// raw bytes. The bytes come from `sliceUnit` over the block's span, exactly the
+// route `geml get` takes. Blocks arrive in document order from both, so the
+// two sequences zip.
+export function gemlFilesToEdn(files, lib) {
+  const { parse, addressedUnits, sliceUnit } = lib;
+  const blocksOf = (text) => {
+    const nodes = parse(text).children.filter((c) => c.kind === "block");
+    const units = [...addressedUnits(text)].map((a) => a.unit).filter((u) => u.kind === "block");
+    return nodes.map((node, i) => ({
+      node,
+      body: () => {
+        const s = sliceUnit(text, units[i].span, "body");
+        return s.endsWith("\n") ? s.slice(0, -1) : s;
+      },
+    }));
+  };
+
+  const onto = blocksOf(files.get("ontology.geml") ?? "");
+  const grab = (blocks, id) => {
+    const b = blocks.find((x) => x.node.id === id);
+    return b ? parseEDNString(b.body()) : undefined;
+  };
+  const properties = grab(onto, "properties");
+  const classes = grab(onto, "classes");
+  const graphExtra = grab(onto, "graph-extra");
+
+  // Page order comes from the graph.geml index; a tree without one (hand-built,
+  // or index deleted) falls back to path order, which at least is deterministic.
+  const indexBlocks = files.has("graph.geml") ? blocksOf(files.get("graph.geml")) : [];
+  const orderBlock = indexBlocks.find((b) => b.node.id === "page-order");
+  const pagePaths = (orderBlock && Array.isArray(orderBlock.node.value)
+    ? orderBlock.node.value
+    : [...files.keys()].filter((p) => p.startsWith("pages/") || p.startsWith("journals/")).sort()
+  ).filter((p) => files.has(p));
+  const pages = pagePaths.map((p) => {
+    let page = { map: [] };
+    let entryRest = { map: [] };
+
+    // Flat level-tagged sequence → tree. Each frame owns the children vector
+    // its node's `:build/children` will become; the vector is written into the
+    // node only if anything landed in it.
+    const roots = [];
+    const stack = [{ level: 0, node: null, children: roots }];
+    const close = (frame) => {
+      if (frame.node && frame.children.length > 0) frame.node.map.push([kw("build/children"), frame.children]);
+    };
+
+    let last = null;
+    for (const b of blocksOf(files.get(p))) {
+      const { type, classes, attrs } = b.node;
+      if (type === "code" && classes.includes("page-meta")) { page = parseEDNString(b.body()); continue; }
+      if (type === "code" && classes.includes("page-extra")) { entryRest = parseEDNString(b.body()); continue; }
+      if (type === "code" && classes.includes("block-meta")) {
+        // Meta re-attaches to the block it followed. Splicing the entries into
+        // the node keeps one map, as the export wrote it.
+        if (last) last.map.push(...mapEntries(parseEDNString(b.body())));
+        continue;
+      }
+      if (type !== "text") continue;
+
+      const level = typeof attrs["level"] === "number" ? attrs["level"] : 1;
+      const node = { map: [[kw("block/title"), b.body()]] };
+      while (stack[stack.length - 1].level >= level) close(stack.pop());
+      stack[stack.length - 1].children.push(node);
+      stack.push({ level, node, children: [] });
+      last = node;
+    }
+    while (stack.length > 1) close(stack.pop());
+
+    const entry = { map: [[kw("page"), page]] };
+    if (roots.length > 0) entry.map.push([kw("blocks"), roots]);
+    entry.map.push(...mapEntries(entryRest));
+    return entry;
+  });
+
+  const out = { map: [[kw("pages-and-blocks"), pages]] };
+  if (properties !== undefined) out.map.push([kw("properties"), properties]);
+  if (classes !== undefined) out.map.push([kw("classes"), classes]);
+  if (graphExtra !== undefined) out.map.push(...mapEntries(graphExtra));
+  return toEDNString(out);
+}