@rubytech/create-maxy-code 0.1.158 → 0.1.161

package/payload/platform/plugins/substack-import/skills/substack-import/SKILL.md

@@ -0,0 +1,182 @@
+---
+name: substack-import
+description: Ingest an extracted Substack "Export your data" archive into the graph as one `:KnowledgeDocument {kind:'substack-post'}` per essay plus one `:KnowledgeDocument {kind:'substack-subscriber-roster'}` carrying `:MENTIONS` edges to each subscriber `:Person`. Triggers when the operator drops a Substack archive directory or `<publication>.zip` path into chat. Owner + publication name confirm against the importing `:AdminUser` before any read. Idempotent on `substackPostId` and `(accountId, email)`.
+---
+
+# Substack Import
+
+**Invoked from `specialists:archive-ingest-operator`** — the admin agent dispatches the archive-ingest-operator when the operator hands over a Substack archive, and the specialist loads this skill. The admin agent never calls this skill directly.
+
+One Substack archive becomes:
+
+- **N `:KnowledgeDocument` nodes** (`source='substack'`, `kind='substack-post'`) — one per essay, MERGEd on the synthetic stable key `attachmentId = "substack-post-${substackPostId}"`. Substack post IDs survive edits, so re-importing after an edit replaces the prior KD's sections under the same `sourceDocumentId`. `document-ingest`'s in-turn typed-section classification chunks the body into `:Section` children.
+- **One `:KnowledgeDocument`** (`source='substack'`, `kind='substack-subscriber-roster'`) per import run, MERGEd on `(accountId, kind, publicationName, importId)`, carrying the subscriber count and one `:MENTIONS` edge per subscriber.
+- **N `:Person` nodes** — one per subscriber row, MERGEd on `(accountId, email)`. Existing `:Person` nodes are property-update no-ops on re-import.
+- **N `:DigitalDocument` nodes** (`kind='substack-image'`) — one per image referenced from any post body, MERGEd on `(accountId, sha256)`, copied to `{accountDir}/archive/substack/<importId>/`.
+
+## Anchor model
+
+A Substack archive has one operator-side identity (the publication's author / owner) and N subscribers. Provenance anchors on the importing `:AdminUser` via `[:IMPORTED]` to every node this skill writes. Subscribers are not "owned by" the importer — they are `:Person` nodes scoped by `accountId`, with the subscription expressed as a `:MENTIONS` edge from the roster KD to each `:Person`. The `publicationName` short string lives as a property on every post KD and the roster KD; there is no `:SubstackPublication` anchor node.
+
+Every node this skill writes (or that document-ingest writes on this skill's dispatch) carries `importId` (UUID per skill run) so a future dream-cycle pass can detect orphans or selectively re-ingest.
+
+## Archive shape
+
+See [archive-shape.md](references/archive-shape.md) for the directory layout, the `posts.csv` columns this skill consumes, the subscriber CSV columns, and the engagement-events filename detection list. Substack export filenames vary by export version — the skill probes a known list and continues with engagement properties absent if none match.
+
+## Pipeline (deterministic, four phases)
+
+### Phase 0 — Provenance confirmation
+
+Before any file body is read:
+
+1. **Identify the importing AdminUser.** Default to the live operator (`$USER_ID`). Surface to the operator: `"Substack archive: importing as :AdminUser <name> ({userId}). Confirm (yes/no)."`. If the operator is importing on behalf of someone else, accept either an existing AdminUser userId or a new `:Person` natural-key (`givenName` + `familyName` + at least one of `email` / `telephone`). Persist as `$importerNodeId`.
+
+2. **Confirm the publication label.** Ask: `"What's the publication's display name? (short string, e.g. 'Stories from the Field')"`. Persist as `$publicationName` — used as a property on every post KD and the roster KD. **No `:SubstackPublication` node is MERGEd** — the label is a property, not an anchor.
+
+3. **Persist run identity.** Generate `$importId = randomUUID()` and `$sessionId` (the writer-session for this skill run). Both flow into every write.
+
+Refuse to read the archive until both `$importerNodeId` and `$publicationName` are echo-confirmed.
+
+### Phase 1 — Walk + engagement detection
+
+1. **Walk the archive directory.** Classify entries per [archive-shape.md](references/archive-shape.md):
+   - `posts.csv` — post metadata index.
+   - `posts/<slug>.html` — per-post body files.
+   - `subscribers.csv` — subscriber roster.
+   - Engagement events file — try in order: `email_activity.csv`, `subscriber_activity.csv`, `emails.csv`. Persist the matched filename as `$engagementFile` or `null`.
+   - `images/` (or any directory referenced from post-body `<img src="…">`) — image enclosures.
+
+2. **Count candidates.** `postCount = posts.csv row count`, `subscriberCount = subscribers.csv row count`, `imageCount = distinct image src refs across all posts/*.html`.
+
+3. **Emit walk-complete log line.** `[substack-import] start importId=<id> publication=<name> exportPath=<path> posts=<n> subscribers=<n> images=<n> engagementFile=<name|none>`. This line is the operator's grep-able run-shape signal.
+
+### Phase 2 — Bulk-import gate
+
+If `postCount > 200` OR `subscriberCount > 2000`:
+
+1. Emit `[substack-import:gate] threshold=posts>200|subscribers>2000 reason=bulk awaiting=filter`.
+2. Surface to the operator: `"This archive has <n> posts and <n> subscribers. Filter before write? Available axes: publishedAt date range (e.g. 'since 2024-01-01'), post tag (e.g. 'only tag X'), subscriber tier ('free' | 'paid' | 'comp'). Reply with one or more, or 'all' to bypass the gate."`.
+3. Apply the operator's filter to the candidate set before any write begins.
+4. Re-emit a fresh `start` line with the reduced counts.
+
+Below threshold, skip this phase entirely.
+
+### Phase 3 — Writes (three sub-passes, in this order)
+
+**3a. Per-post `:KnowledgeDocument` via librarian/document-ingest.**
+
+For each post (after gate filter), dispatch `librarian` via the Task tool. The dispatch brief carries:
+
+- **Anchor:** the confirmed `$importerNodeId` (`:AdminUser` if the importer was the operator, else the named `:Person`).
+- **`attachmentId`:** `"substack-post-${substackPostId}"` — synthetic stable key. Substack post IDs survive edits; re-imports of an edited post replace prior sections under the same `sourceDocumentId`.
+- **`storagePath`:** `{accountDir}/archive/substack/<importId>/posts/<slug>.html` (copy the source file there before dispatch so the storage path is stable).
+- **`mimeType`:** `text/html`.
+- **`sourceUrl`:** `https://<publication-domain>/p/<slug>` if the export carries it, else absent.
+- **`sourceType`:** `'substack'` (carried into `:KnowledgeDocument.source`).
+- **`scope`:** `'admin'` (the post is the operator's authored content).
+- **Additional KD properties** (set by document-ingest from the brief's `extraProperties` block): `kind: 'substack-post'`, `substackPostId`, `slug`, `publishedAt`, `isPaid`, `publicationName`, `bodyTruncated` (true if the body is a paywall stub — detected by length < 200 chars on `isPaid=true` posts), `importId`, `createdByAgent: 'substack-import'`, `createdBySession: $sessionId`.
+
+Per-post outcomes (created / updated / skipped) are returned by document-ingest and accumulated into the per-pass counter — no per-post log line is emitted.
+
+See [posts.md](references/posts.md) for HTML-to-markdown body conversion rules and inline-mention handling.
+
+**3b. Roster KD + per-subscriber `:Person` + `:MENTIONS` edges.**
+
+`archive-ingest-operator` runs these writes directly using its `memory-write` tool — no dispatch needed.
+
+1. **Read engagement aggregates** (once, before the per-subscriber loop). If `$engagementFile` is non-null, parse it per [engagement.md](references/engagement.md) and build an in-memory map `emailToAggregates: { email -> { totalOpens, totalClicks, lastOpenedAt, lastClickedAt, engagementWindowDays } }`. If `$engagementFile` is null, emit `[substack-import:warn] no-engagement-file` once and use an empty map.
+
+2. **MERGE the roster KD** via `memory-write`:
+   ```cypher
+   MERGE (r:KnowledgeDocument {accountId: $accountId, kind: 'substack-subscriber-roster', publicationName: $publicationName, importId: $importId})
+   ON CREATE SET r.source = 'substack', r.subscriberCount = $subscriberCount,
+                 r.createdByAgent = 'substack-import', r.createdBySession = $sessionId,
+                 r.createdAt = datetime()
+   ```
+   Persist the returned elementId as `$rosterNodeId`.
+
+3. **Write the `:IMPORTED` edge** from `$importerNodeId` to `$rosterNodeId` carrying `importId` and `createdAt`.
+
+4. **Per-subscriber loop.** For each row in `subscribers.csv` (after gate filter), parse per [subscribers.md](references/subscribers.md), look up `emailToAggregates[row.email]`, then one `memory-write` MERGE:
+   ```cypher
+   MERGE (p:Person {accountId: $accountId, email: $email})
+   ON CREATE SET p.givenName = $givenName, p.familyName = $familyName,
+                 p.source = 'substack', p.createdByAgent = 'substack-import',
+                 p.createdBySession = $sessionId, p.createdAt = datetime(),
+                 p.importId = $importId
+   WITH p
+   MATCH (r:KnowledgeDocument) WHERE elementId(r) = $rosterNodeId
+   MERGE (r)-[m:MENTIONS {mentionContext: 'substack-subscription'}]->(p)
+   ON CREATE SET m.tier = $tier, m.subscribedAt = $subscribedAt,
+                 m.createdByAgent = 'substack-import', m.createdAt = datetime()
+   SET m.unsubscribedAt = $unsubscribedAt,
+       m.totalOpens = $totalOpens, m.totalClicks = $totalClicks,
+       m.lastOpenedAt = $lastOpenedAt, m.lastClickedAt = $lastClickedAt,
+       m.engagementWindowDays = $engagementWindowDays
+   ```
+   Engagement properties OVERWRITE on re-import (the export carries totals-to-date, not deltas). Absent engagement (no row in the events file) sets the properties to `null`, which Neo4j removes — so missing aggregates are absent, not zero.
+
+Per-row outcomes accumulate into the per-pass counter — no per-row log line is emitted.
+
+**3c. Image enclosures.**
+
+For each distinct image referenced from any imported post body:
+
+1. Copy file to `{accountDir}/archive/substack/<importId>/<sha256><ext>` (perms 0o600).
+2. `memory-write` MERGE:
+   ```cypher
+   MERGE (d:DigitalDocument {accountId: $accountId, sha256: $sha256})
+   ON CREATE SET d.kind = 'substack-image', d.source = 'substack',
+                 d.storagePath = $storagePath, d.mimeType = $mimeType,
+                 d.createdByAgent = 'substack-import', d.createdBySession = $sessionId,
+                 d.createdAt = datetime(), d.importId = $importId
+   ```
+3. For each post KD that references this image, attach the canonical enclosure edge:
+   ```cypher
+   MATCH (k:KnowledgeDocument), (d:DigitalDocument)
+   WHERE k.attachmentId = $postAttachmentId AND elementId(d) = $imageElementId
+   MERGE (k)-[e:HAS_ENCLOSURE]->(d)
+   ON CREATE SET e.createdByAgent = 'substack-import', e.createdAt = datetime()
+   ```
+   `:HAS_ENCLOSURE` is the canonical attachment edge for `:KnowledgeDocument → :DigitalDocument` declared in `schema-base.md`. If at skill-load the canonical edge is absent from the schema, fall back to `:MENTIONS {mentionContext: 'substack-image-enclosure'}` — the documented closed-edge fallback.
+
+See [attachments.md](references/attachments.md) for the image-reference discovery walk and storage-path conventions.
+
+### Phase 4 — Completion log
+
+Emit one grep-able summary line:
+
+```
+[substack-import] importId=<id> posts=<created>/<updated>/<skipped> subscribers=<created>/<updated>/<skipped> images=<copied>/<skipped> engagementSubscribers=<n> ms=<elapsed>
+```
+
+`engagementSubscribers` counts subscribers whose `:MENTIONS` edge received engagement properties (non-null). A `start` line that named an engagement file followed by `engagementSubscribers=0` at completion is a parse defect — investigate the events-file shape.
+
+## Idempotency
+
+- Re-importing identical archive bytes is a no-op: post KDs match on the synthetic `attachmentId = "substack-post-${substackPostId}"`; `document-ingest` skips identical content. Subscribers match on `(accountId, email)`; the `:Person` MERGE is a property-update no-op. Images match on `(accountId, sha256)`.
+- Re-importing an archive with edited posts: the synthetic `attachmentId` is stable, so the same KD is updated in place (`document-ingest` regenerates sections under the same `sourceDocumentId`).
+- Re-importing an archive with newer engagement data: engagement properties OVERWRITE via `ON MATCH SET`. The export carries totals-to-date, not deltas; the latest export's totals are authoritative.
+- Re-importing with an additive subscriber list: existing subscribers' `:MENTIONS` edges keep their `subscribedAt`; new subscribers get a fresh edge. Unsubscribes set `unsubscribedAt` (and do not delete the edge).
+- The roster KD is one-per-import-run (keyed on `importId`), so each import leaves an auditable trail of which run produced which `:MENTIONS` set.
+
+## Closed-edge defence
+
+The skill never invents an edge type. Every edge written:
+
+- `:IMPORTED` (AdminUser → roster KD) — provenance, schema-declared.
+- `:MENTIONS {mentionContext:'substack-subscription', ...engagement}` (roster KD → Person) — documented catch-all for any relation not in the closed enumeration.
+- `:HAS_ENCLOSURE` (post KD → DigitalDocument) — canonical attachment edge if schema-base declares it; `:MENTIONS {mentionContext:'substack-image-enclosure'}` fallback.
+
+Audit query: `MATCH ()-[r]->() WHERE r.createdByAgent='substack-import' RETURN DISTINCT type(r)` must return only the above. Any other edge type is a defect.
+
+## What this skill does not do
+
+- **New labels.** No `:SubstackPublication`, `:Subscriber`, `:Subscription`, `:Post`. Everything maps to existing `:KnowledgeDocument` + `:Person` + `:DigitalDocument`.
+- **New edge types.** No `:SUBSCRIBED_TO`, `:OPENED`, `:CLICKED`. Subscription is `:MENTIONS`; engagement is properties on that edge.
+- **Per-event engagement detail.** Per-subscriber-per-post events are aggregated to per-edge totals; the individual open/click events are not preserved.
+- **Comments.** Substack export does not include reader comments.
+- **Inbound newsletter ingestion.** A Substack post landing in the operator's Gmail goes through `conversation-archive --source email` (Task 426), not this plugin.
+- **Live API or RSS poll.** Operator-dropped export only.
+- **Audio / video enclosures.** Substack podcast / video posts use additional file shapes the v1 plugin does not handle; only images referenced from post-body `<img>` tags are enclosed.

package/payload/platform/plugins/substack-import/skills/substack-import/references/archive-shape.md

@@ -0,0 +1,68 @@
+# Substack archive shape
+
+The "Export your data" download from Substack arrives as a zip. Once extracted, the top-level directory contains the following entries. Filenames have varied across export versions — the skill probes the listed candidates in order and binds to the first match.
+
+## Directory layout
+
+```
+<archive-root>/
+├── posts.csv                     # post metadata index (required)
+├── posts/
+│   ├── <post-id>.<slug>.html     # one HTML body per post (filename = "<post_id>.<slug>.html")
+│   └── …
+├── subscribers.csv               # subscriber roster (required)
+├── email_activity.csv            # engagement events (one of these names, optional)
+│   OR subscriber_activity.csv
+│   OR emails.csv
+├── images/                       # image enclosures referenced from post bodies
+│   └── <files>
+└── (other files — manifest.json, etc.) ignored by this skill
+```
+
+Required: `posts.csv` and `subscribers.csv` must exist. If either is missing, abort the walk with `[substack-import:error] file=<archive-root> reason="missing required file: <name>"`.
+
+Optional: engagement events file. If none of the three candidate names exist, emit `[substack-import:warn] no-engagement-file` once and continue with engagement properties absent on every `:MENTIONS` edge.
+
+## `posts.csv` columns this skill consumes
+
+| Column | Used as | Notes |
+|---|---|---|
+| `post_id` | `substackPostId` (stable across edits) | Required. MERGE key via synthetic `attachmentId = "substack-post-${post_id}"`. |
+| `post_date` | `publishedAt` (ISO 8601) | Required. |
+| `is_published` | filter | Only `true` rows are imported; drafts are dropped at walk. |
+| `type` | filter | Only `newsletter` / `podcast` / `thread` rows imported; system rows dropped. |
+| `audience` | `isPaid` | `everyone` → `isPaid=false`; `only_paid` / `founding` → `isPaid=true`. |
+| `slug` | `slug` | Used to compose the post body filename `posts/<post_id>.<slug>.html`. |
+| `title` | `title` | KD title. |
+| `subtitle` | `subtitle` | Optional KD property. |
+| `email_sent_at` | optional context | Recorded if present. |
+
+Columns the skill does not read (`url`, `webRecommendations`, etc.) are ignored — adding them later does not require a parser change.
+
+## `subscribers.csv` columns this skill consumes
+
+| Column | Used as | Notes |
+|---|---|---|
+| `email` | `:Person.email` (MERGE key) | Required. Lowercased. |
+| `name` | `:Person.givenName` + `:Person.familyName` | Split on first whitespace; surname = remainder. Single-token names use the token as `givenName`. |
+| `type` (or `membership_state` / `subscription_state`) | `:MENTIONS.tier` | `free` / `paid` / `comp` / `gift`. Unknown values pass through verbatim. |
+| `created_at` (or `subscribed_at`) | `:MENTIONS.subscribedAt` | ISO 8601. |
+| `expiry` / `unsubscribed_at` | `:MENTIONS.unsubscribedAt` | Optional. Present only for unsubscribed rows. |
+
+Columns the skill does not read (e.g. `stripe_customer_id`) are ignored.
+
+## Engagement-events filename detection
+
+The skill probes for the engagement events file in this order. The first match binds `$engagementFile`:
+
+1. `email_activity.csv`
+2. `subscriber_activity.csv`
+3. `emails.csv`
+
+If a new export version uses a different name, add it to this list — the skill does not auto-discover.
+
+## Hard limits
+
+- `posts.csv` row count > 200: bulk-gate fires.
+- `subscribers.csv` row count > 2000: bulk-gate fires.
+- Individual post body > 50 MB: skipped with `[substack-import:error] file=<…> reason="body exceeds 50MB"`. (Substack does not generally produce bodies this large; the limit is a defence against malformed archives.)

package/payload/platform/plugins/substack-import/skills/substack-import/references/attachments.md

@@ -0,0 +1,72 @@
+# Image enclosures
+
+Image files referenced from post bodies become `:DigitalDocument` nodes attached to the parent post KD via the canonical `:HAS_ENCLOSURE` edge (or `:MENTIONS {mentionContext:'substack-image-enclosure'}` as the documented closed-edge fallback if `schema-base.md` does not declare `:HAS_ENCLOSURE` for `:KnowledgeDocument → :DigitalDocument` at skill-load).
+
+## Discovery walk
+
+After Phase 3a (post writes) but before Phase 3c (image writes), build a per-post image-reference map:
+
+1. For each successfully-imported post HTML body (the operator's source file, before HTML→markdown), parse `<img src="…">` tags.
+2. Resolve the `src` relative to the archive root. Substack export sources are typically `images/<filename>` or `../images/<filename>` from a post-body directory; resolve both to the absolute path under `<archive-root>/images/`.
+3. Record `{ postAttachmentId → [imageAbsolutePath, …] }`.
+
+Image files referenced from `<img>` are the only enclosures considered. Audio (`<audio>`) and video (`<video>`) embeds are out of scope per SKILL.md.
+
+## Per-image write
+
+For each distinct image path (one MERGE per file, regardless of how many posts reference it):
+
+1. **Compute content hash.** SHA-256 of the file bytes. Persist as `$sha256`.
+2. **Copy file.** From the archive's `images/<filename>` to `{accountDir}/archive/substack/<importId>/<sha256><ext>` (perms 0o600). The destination filename is content-addressed so the same image referenced from multiple imports overlays cleanly.
+3. **Detect mime type.** From the file extension first (`.jpg`/`.jpeg` → `image/jpeg`, `.png` → `image/png`, `.gif` → `image/gif`, `.webp` → `image/webp`). If the extension is missing or unrecognised, sniff the first 16 bytes against known image magic numbers. Fallback: `application/octet-stream`.
+4. **MERGE the `:DigitalDocument`.**
+   ```cypher
+   MERGE (d:DigitalDocument {accountId: $accountId, sha256: $sha256})
+   ON CREATE SET d.kind             = 'substack-image',
+                 d.source           = 'substack',
+                 d.storagePath      = $storagePath,
+                 d.mimeType         = $mimeType,
+                 d.originalFilename = $originalFilename,
+                 d.createdByAgent   = 'substack-import',
+                 d.createdBySession = $sessionId,
+                 d.createdAt        = datetime(),
+                 d.importId         = $importId
+   ```
+   Re-imports of the same image are property-update no-ops.
+
+## Per-post attach
+
+For each `(postAttachmentId, imageElementId)` pair from the discovery map:
+
+```cypher
+MATCH (k:KnowledgeDocument {accountId: $accountId, attachmentId: $postAttachmentId}),
+      (d:DigitalDocument)   WHERE elementId(d) = $imageElementId
+MERGE (k)-[e:HAS_ENCLOSURE]->(d)
+ON CREATE SET e.createdByAgent = 'substack-import',
+              e.createdAt      = datetime()
+```
+
+If at skill-load the canonical `:HAS_ENCLOSURE` edge is absent from `schema-base.md` for `:KnowledgeDocument → :DigitalDocument`, swap to:
+
+```cypher
+MERGE (k)-[e:MENTIONS {mentionContext: 'substack-image-enclosure'}]->(d)
+ON CREATE SET e.createdByAgent = 'substack-import',
+              e.createdAt      = datetime()
+```
+
+The choice is fixed at skill-load (one read of `schema-base.md`), not per-image — so every image enclosure in this run uses the same edge type. The closed-edge audit query distinguishes the two.
+
+## Missing image files
+
+If an `<img src="…">` reference points at a file that does not exist in the archive's `images/` directory:
+
+- Emit `[substack-import:error] file=<postFilename> reason="image reference missing: <src>"`.
+- The post KD is unaffected (it already wrote in Phase 3a); the enclosure is simply absent.
+- Continue with remaining images.
+
+## Counter
+
+The per-pass image counter (in the completion log) accumulates:
+
+- `copied`: count of files where the destination at `{accountDir}/archive/substack/<importId>/<sha256><ext>` did not exist before (first-time content). Re-imports of the same image bytes do not advance this counter.
+- `skipped`: count of files where the destination already existed (re-import of identical content).

package/payload/platform/plugins/substack-import/skills/substack-import/references/engagement.md

@@ -0,0 +1,61 @@
+# Engagement events aggregation
+
+Substack engagement export carries per-subscriber-per-post events (opens, clicks). This skill aggregates them to per-subscriber totals folded onto the `:MENTIONS` edge between the roster KD and each `:Person` — per-event detail is intentionally dropped.
+
+## Detection
+
+Per [archive-shape.md](archive-shape.md), the skill probes for the events file in this order:
+
+1. `email_activity.csv`
+2. `subscriber_activity.csv`
+3. `emails.csv`
+
+The first match binds `$engagementFile`. None match: emit `[substack-import:warn] no-engagement-file` once and proceed with an empty aggregates map.
+
+## Column tolerance
+
+Substack has used at least three column-naming schemes across export versions. Probe for each value in the order listed; first match wins:
+
+| Aggregate field | Candidate column names |
+|---|---|
+| subscriber email | `email`, `subscriber_email`, `user_email` |
+| event type | `event`, `event_type`, `action` |
+| event timestamp | `event_at`, `timestamp`, `time` |
+| post identifier (optional, for cross-check) | `post_id`, `email_id` |
+
+If the email or event-type column cannot be resolved, emit `[substack-import:error] file=<engagementFile> reason="cannot resolve email or event-type column"` and proceed with empty aggregates — do not abort the whole import.
+
+## Aggregation
+
+Walk the file once, accumulating per email:
+
+```
+emailToAggregates: Map<email-lowercased, {
+  totalOpens: int,           // count of rows where event matches /^open$/i
+  totalClicks: int,          // count of rows where event matches /^click$/i
+  lastOpenedAt: ISO|null,    // max timestamp where event matched open
+  lastClickedAt: ISO|null,   // max timestamp where event matched click
+  engagementWindowDays: int  // (now - earliest event timestamp for this email) in days, ceil
+}>
+```
+
+Event types other than open / click are tallied separately and reported in the completion log as `otherEvents=<n>` if non-zero, but are not folded onto the edge (no `:MENTIONS.totalForwards` etc. — would inflate the property set without a concrete query motivating it).
+
+## Cardinality bound
+
+A worst-case substack archive: 2000 subscribers × 500 posts × 2 event types = 2,000,000 events. Reading this twice (once to count, once to aggregate) wastes I/O; the aggregation pass runs streaming with a single map allocation. Map size is bounded by distinct emails (≤ subscriber count), not by event count — so the in-memory cost is O(subscribers), not O(events).
+
+If the events file exceeds 500 MB, emit `[substack-import:warn] engagement-file-large size=<bytes>` so the operator knows the parse pass will take a few minutes; proceed anyway.
+
+## Re-import semantics
+
+Each Substack export carries totals-to-date, not deltas. On re-import:
+
+- The aggregation walk recomputes the per-email totals from the new file.
+- The per-subscriber MERGE's unconditional `SET` block overwrites engagement properties on the edge.
+- Subscriber 1 with 3 opens in import A and 5 opens in import B: edge ends with `totalOpens=5` (the latest export's total), not 8.
+- A subscriber who appears in import A's events file but not import B's: their edge's engagement properties are set to `null` in import B (Neo4j removes them), so the edge stops showing engagement. This is correct — if Substack's export omits a subscriber, they have no recorded engagement and the edge should reflect that.
+
+## Why not per-event KDs
+
+Per-event detail (which specific day of which specific open) is high-volume and low-query-value at v1. The aggregates answer the "warm vs cold" question that the operator actually asks. If a downstream feature needs per-event detail (e.g., "show me a timeline of when subscriber X engaged with my posts"), it lands as a separate task — likely as `:KnowledgeDocument {kind:'substack-email-event'}` children of the per-post KD with `:MENTIONS` to the `:Person`. That schema choice is deferred until a concrete query motivates it.

package/payload/platform/plugins/substack-import/skills/substack-import/references/posts.md

@@ -0,0 +1,80 @@
+# Post body handling
+
+Per-post writes route through `librarian` / `document-ingest`. This reference documents how the body is prepared and what the dispatch brief carries.
+
+## File pairing
+
+For each row in `posts.csv`, the body file is at `posts/${post_id}.${slug}.html`. If the file is missing:
+
+- If `is_published=true` and the post is `paid_only` and the operator's export does not include paid content: write the post KD anyway with `bodyTruncated=true` and an empty body (so the metadata + subscription-derivation work; the body lands when the operator re-exports with paid-content entitlement).
+- Otherwise: emit `[substack-import:error] file=posts/<filename> reason="body file missing"` and skip the post.
+
+## HTML → markdown conversion
+
+Substack post bodies use a constrained HTML subset. Convert before dispatch so `document-ingest`'s in-turn classification sees clean markdown.
+
+Block-level elements:
+
+| HTML | Markdown |
+|---|---|
+| `<h1>` … `<h6>` | `#` … `######` |
+| `<p>` | paragraph |
+| `<blockquote>` | `> ` prefix per line |
+| `<ul>` / `<ol>` | `-` / `1.` per item |
+| `<pre><code>` | fenced ``` block |
+| `<hr>` | `---` |
+
+Inline elements:
+
+| HTML | Markdown |
+|---|---|
+| `<a href="…">text</a>` | `[text](href)` |
+| `<strong>` / `<b>` | `**text**` |
+| `<em>` / `<i>` | `*text*` |
+| `<code>` | `` `text` `` |
+| `<img src="…" alt="…">` | `![alt](src)` — `src` is preserved verbatim for the attachment walk to resolve |
+
+Substack-specific embeds (`<div class="subscription-widget">`, paywall stubs, audio player blocks): replace with a single placeholder line `[Substack embed: <type>]` and continue. The embed-detail-loss is documented; full embed preservation would require re-rendering the player on read which is out of scope.
+
+## Paywall stub detection
+
+A paywall stub is a body where:
+
+- `audience` (from `posts.csv`) is `only_paid` or `founding`, AND
+- the converted-markdown body length is < 200 characters, AND
+- the body contains a recognisable paywall marker (e.g., "This post is for paid subscribers", `<div class="paywall">`, etc.).
+
+When detected, set `bodyTruncated=true` on the dispatch brief's `extraProperties`. The KD still writes; downstream "this post body is incomplete" handling can branch on the property.
+
+## Inline `@person` mentions
+
+Substack post bodies occasionally reference `@handle`-style mentions. `document-ingest`'s in-turn typed-section classification handles these — the dispatch brief does not pre-resolve them. Per `archive-ingest-operator.md`'s account-isolation rule, mentions resolve against `:Person {accountId}` only; no auto-create from mention text.
+
+## Dispatch brief shape (the Task-tool input the operator builds)
+
+```
+Anchor: <importerNodeId> (:AdminUser or :Person)
+attachmentId: "substack-post-<post_id>"
+storagePath: {accountDir}/archive/substack/<importId>/posts/<post_id>.<slug>.html
+mimeType: text/html
+sourceUrl: https://<publication-domain>/p/<slug>   (omit if unknown)
+sourceType: 'substack'
+scope: 'admin'
+extraProperties:
+  kind: 'substack-post'
+  substackPostId: <post_id>
+  slug: <slug>
+  publishedAt: <ISO 8601>
+  isPaid: <bool>
+  publicationName: $publicationName
+  bodyTruncated: <bool>            (only when paywall-stub)
+  importId: $importId
+  createdByAgent: 'substack-import'
+  createdBySession: $sessionId
+```
+
+The librarian dispatched against this brief writes the `:KnowledgeDocument` and classified `:Section` children, then returns `{ elementId, outcome: 'created' | 'updated' | 'skipped' }` which feeds the per-pass counter.
+
+## Storage path
+
+Before dispatch, copy `posts/${post_id}.${slug}.html` from the operator's export directory to `{accountDir}/archive/substack/<importId>/posts/${post_id}.${slug}.html` (perms 0o600). This is the stable on-disk location the KD's `storagePath` property references — the operator may delete the export directory after import; the archived copy survives.

package/payload/platform/plugins/substack-import/skills/substack-import/references/subscribers.md

@@ -0,0 +1,74 @@
+# Subscriber roster handling
+
+The subscribers pass writes one `:KnowledgeDocument {kind:'substack-subscriber-roster'}` and one `:Person` per row, with a `:MENTIONS` edge between them carrying subscription tier and engagement aggregates.
+
+## Per-row parse
+
+For each row in `subscribers.csv` (after bulk-gate filter):
+
+1. **Email — required.** Lowercase. If missing or malformed, emit `[substack-import:error] file=subscribers.csv reason="missing email at row <n>"` and skip.
+2. **Name — optional.**
+   - Split on first whitespace.
+   - First token → `givenName`.
+   - Remainder (if any) → `familyName`.
+   - Single-token names: `givenName=token, familyName=null`.
+   - Empty name: both null. The `:Person` writes with email only.
+3. **Tier — required.** Read from `type` / `membership_state` / `subscription_state` (first present). Pass through verbatim — `free`, `paid`, `comp`, `gift`, anything else. Validation is downstream's responsibility.
+4. **Subscribed-at — required.** Read from `created_at` / `subscribed_at`. ISO 8601.
+5. **Unsubscribed-at — optional.** Present only on unsubscribed rows. ISO 8601.
+
+## Per-row write
+
+The MERGE shape (one `memory-write` Cypher call per subscriber row):
+
+```cypher
+MERGE (p:Person {accountId: $accountId, email: $email})
+ON CREATE SET p.givenName            = $givenName,
+              p.familyName           = $familyName,
+              p.source               = 'substack',
+              p.createdByAgent       = 'substack-import',
+              p.createdBySession     = $sessionId,
+              p.createdAt            = datetime(),
+              p.importId             = $importId
+WITH p
+MATCH (r:KnowledgeDocument) WHERE elementId(r) = $rosterNodeId
+MERGE (r)-[m:MENTIONS {mentionContext: 'substack-subscription'}]->(p)
+ON CREATE SET m.tier                 = $tier,
+              m.subscribedAt         = $subscribedAt,
+              m.createdByAgent       = 'substack-import',
+              m.createdAt            = datetime()
+SET m.unsubscribedAt                 = $unsubscribedAt,
+    m.totalOpens                     = $totalOpens,
+    m.totalClicks                    = $totalClicks,
+    m.lastOpenedAt                   = $lastOpenedAt,
+    m.lastClickedAt                  = $lastClickedAt,
+    m.engagementWindowDays           = $engagementWindowDays
+```
+
+Notes:
+
+- The `ON CREATE` block on `:Person` sets `givenName` / `familyName` only on first write. Existing `:Person` nodes (e.g., a contact already imported from LinkedIn) keep their operator-set names; the subscription edge still attaches.
+- The `ON CREATE` block on `:MENTIONS` sets `tier` and `subscribedAt` only once (the original subscription). Subsequent imports overwriting tier (e.g., free → paid upgrade) need `SET m.tier = $tier` explicitly — see "tier updates on re-import" below.
+- The unconditional `SET` block at the bottom overwrites engagement properties and `unsubscribedAt`. The export carries totals-to-date, not deltas — overwrite is correct.
+- Engagement properties from `$totalOpens` etc. set to Cypher `null` (because the events file lacked an entry for this email) remove the properties from the edge; Neo4j does not distinguish "property=null" from "property absent". This is the intended semantic — absent engagement is absent, not zero.
+
+## Tier updates on re-import
+
+Substack tier can change (free → paid upgrade, paid → comp, etc.). The current MERGE keeps the original `tier` from the first import. To overwrite on re-import, the writer adds `SET m.tier = $tier` to the unconditional `SET` block. This sacrifices "first-subscribed-as" auditability — if the operator needs that, the original tier can be queried from the per-import roster KD's `createdAt` lineage.
+
+For v1, the unconditional `SET m.tier = $tier` is the right call (engagement-warm signal depends on current tier, not historical). The `subscribedAt` stays `ON CREATE` because that is the immutable first-subscription timestamp.
+
+## Anchor-edge from the importing operator
+
+After the per-row loop, write one `:IMPORTED` edge from the importing operator to the roster KD:
+
+```cypher
+MATCH (a) WHERE elementId(a) = $importerNodeId
+MATCH (r:KnowledgeDocument) WHERE elementId(r) = $rosterNodeId
+MERGE (a)-[i:IMPORTED]->(r)
+ON CREATE SET i.importId = $importId,
+              i.source = 'substack',
+              i.createdAt = datetime()
+```
+
+This edge is the provenance trail: "who pushed the button for this subscription roster."

package/payload/platform/plugins/whatsapp/PLUGIN.md

@@ -79,7 +79,7 @@ Four tools for reading WhatsApp conversation context. The first two read from th
 |------|---------|
 | `whatsapp-conversations` | JIDs with recent message activity, conversation type (group/DM), last message timestamp, message count |
 | `whatsapp-messages` | In-memory ring-buffer messages for a specific JID — sender, body, timestamp, reply context. Supports optional `limit` for most recent N messages |
-| `whatsapp-conversation-graph-state` | Persisted `:Message:WhatsAppMessage` rows attached to the `:Conversation` (sessionKey-keyed) plus the resolved `conversationId`, the cypher string, and a divergence header reporting graph-row count vs in-memory count. Use when the question is "what got persisted" — never compose the cypher manually |
+| `whatsapp-conversation-graph-state` | Persisted `:Message:WhatsAppMessage` rows attached to the `:Conversation` (sessionKey-keyed) plus the resolved `sessionId`, the cypher string, and a divergence header reporting graph-row count vs in-memory count. Use when the question is "what got persisted" — never compose the cypher manually |
 | `whatsapp-group-info` | Group metadata from WhatsApp servers — subject, description, participants with admin flags, creation date. Requires active connection |
 
 ## Group access — three-way mention recognition

package/payload/platform/plugins/whatsapp/mcp/dist/index.js

@@ -379,7 +379,7 @@ server.tool("whatsapp-messages", "Retrieve stored WhatsApp messages for a specif
 // graph-row vs in-memory-store count drift as an operator-visible signal.
 server.tool("whatsapp-conversation-graph-state", [
     "Read persisted graph state for a WhatsApp conversation deterministically.",
-    "Returns the conversation's `conversationId` and every `:Message:WhatsAppMessage` row attached via `(:PART_OF)->`, plus the in-memory store's row count for divergence visibility.",
+    "Returns the conversation's `sessionId` and every `:Message:WhatsAppMessage` row attached via `(:PART_OF)->`, plus the in-memory store's row count for divergence visibility.",
     "Use this whenever the question is 'what messages got persisted to <jid> / <sessionKey>?'. Do not compose cypher by hand — the cypher in the platform uses the writer-known filter (sessionKey on the Conversation node, not jid match on the Message node).",
     "Input: pass `jid` (e.g. group `120363...@g.us` or DM `44...@s.whatsapp.net`) — the platform derives the public-agent sessionKey. For admin-agent sessionKeys, pass `sessionKey` directly.",
 ].join("\n"), {
@@ -417,7 +417,7 @@ server.tool("whatsapp-conversation-graph-state", [
         const rows = result.graphRows ?? [];
         const headerParts = [
             `sessionKey=${result.sessionKey}`,
-            `conversationId=${result.conversationId ?? "null"}`,
+            `sessionId=${result.sessionId ?? "null"}`,
             `graphRows=${rows.length}`,
         ];
         if (memoryCount !== null)