@rubytech/create-maxy-code 0.1.140 → 0.1.142

package/payload/platform/plugins/x-import/skills/x-import/SKILL.md

@@ -0,0 +1,123 @@
+---
+name: x-import
+description: Ingest an extracted X (Twitter) archive directory into the graph as one :KnowledgeDocument (tweet stream) plus one :ConversationArchive per DM participant set. Triggers when the operator drops an `Archive of <handle>` directory or `twitter-<handle>-archive.zip` path into chat. Owner + every distinct participant confirms against existing `:AdminUser` / `:Person` nodes before any write. Idempotent on archive content hash.
+---
+
+# X Import
+
+**Invoked from `specialists:database-operator`** — the admin agent dispatches the database-operator when the operator hands over an X archive, and the database-operator loads this skill. The admin agent never calls this skill directly.
+
+One X archive becomes:
+
+- **One `:KnowledgeDocument`** (`source='x'`) carrying the verbatim chronological transcript of the operator's authored content — root tweets, replies, quote-tweet authorship, deletion markers. `memory-classify` (document mode) splits the body into topic-bounded `:Section` children. Threading is a within-body concept rendered in the transcript; per-thread KDs are explicitly rejected.
+- **N `:ConversationArchive` nodes** (`source='x-dm'`) — one per distinct DM participant set, MERGEd on the existing `conversationIdentity = sha256(accountId + ":" + sortedParticipantElementIds)` key. DM media attaches to the archive node via `HAS_ENCLOSURE`.
+- **Person resolution** — every `@handle` mentioned, replied-to, or quoted in the tweet stream MERGEs against `:Person` on lowercased `xHandle` scoped by `accountId`. Existing `:Person` nodes gain `xHandle` on `ON CREATE SET` without overwriting operator edits.
+
+## Archive shape
+
+See [archive-shape.md](../../references/archive-shape.md) for the `.js` wrapper grammar and which files in the archive carry which data. The grammar is verified at parse — unwrapped JSON or a missing `window.YTD.*.partN = ` prefix errors loudly, never silently zero-tweets.
+
+## Transcript rendering
+
+See [transcript-shape.md](../../references/transcript-shape.md) for the exact rendering of the tweet-stream KD body — chronological separators, quote-tweet indentation, deletion markers, media references.
+
+## Pipeline (deterministic, four phases)
+
+### Phase 0 — Owner + participant confirmation
+
+Before any classify or ingest call:
+
+1. **Identify the owner.** Read `data/account.js` to extract the archive owner's `accountId`, `username` (the handle), and `displayName`. Surface to the operator: `"X archive owner: @<handle> (<displayName>). Confirm this maps to :AdminUser <name> ({userId})."`. Refuse to proceed without confirmation.
+
+2. **Enumerate distinct participants from the tweet stream.** Read `data/tweets.js` and `data/tweet-headers.js`. Extract every `@handle` that appears as a mention, reply target (`in_reply_to_screen_name`), or quote-tweet author. Emit a single-line histogram per distinct handle: `Handle '@<handle>' (<mentions> mentions, <replies> replies, <quotes> quotes)`.
+
+3. **Enumerate distinct DM participants.** Read `data/direct-messages.js` and `data/direct-messages-group.js` (when present). For each `dmConversation`, extract the distinct senderIds. Senders map to existing `:Person` / `:AdminUser` by either prior `xUserId` property or operator confirmation against numeric ID + recent-message preview.
+
+4. **One operator question per distinct handle / senderId.** Mirror the conversation-archive flow: `"Handle '@<h>' (<n> mentions) — pick existing :Person/:AdminUser or block?"`. The operator either picks an elementId or names "block" (refuses to map). **Never auto-create a `:Person`** — the operator must confirm a canonical node.
+
+5. **Persist resolved IDs.** Build two maps: `handleToElementId` (handle → :Person elementId, for the tweet-stream KD edges) and `senderIdToElementId` (numeric senderId → :Person/:AdminUser elementId, for DM participant lists). Both maps are passed to the writers in subsequent phases.
+
+### Phase 1 — Tweet-stream transcript + media staging
+
+1. **Walk the tweet stream chronologically.** Merge `data/tweets.js` (operator-authored tweets) and `data/tweet-headers.js` (metadata). For each tweet: extract `id_str`, `created_at`, `full_text`, `in_reply_to_status_id_str`, `in_reply_to_screen_name`, `entities.user_mentions`, `entities.urls`, `entities.media`, `quoted_status_id_str`. Sort by `created_at` ascending.
+
+2. **Render the transcript.** Apply the rules in [transcript-shape.md](../../references/transcript-shape.md). The output is a single UTF-8 text file written to `{accountDir}/archive/x/<importId>/tweet-stream.md`, where `importId = sha256(archive bytes)` truncated to 16 hex chars.
+
+3. **Stage media enclosures.** For each `entities.media` entry: copy from `data/tweet_media/<filename>` to `{accountDir}/archive/x/<importId>/media/<sha256>-<filename>` (perms 0o600). Same for DM media (`data/direct_messages_media/`) — staged into the same `media/` dir with the same hash-prefixed naming so re-import is content-addressed.
+
+4. **Render deletion markers.** When `tweet-headers.js` references an id that has no body in `tweets.js`, render the slot in the transcript as `[deleted on <iso>]` so context boundaries survive classification.
+
+### Phase 2 — Tweet-stream :KnowledgeDocument ingest
+
+Dispatch the existing librarian / document-ingest pipeline against the rendered transcript file. The attachment metadata block is constructed deterministically:
+
+- `filename = "tweet-stream-@<handle>.md"`
+- `attachmentId = sha256(transcript-bytes)` — guarantees idempotency on identical content
+- `storagePath = {accountDir}/archive/x/<importId>/tweet-stream.md`
+- `mimeType = "text/markdown"`
+
+The dispatch brief names:
+- **Anchor:** `:AdminUser` (the confirmed owner from Phase 0). Adam Mackay / Joel Smalley etc. resolved in Phase 0 appear in the **key entities to connect** list — `wire-brief-entities` writes the `MENTIONED` / `REPLIED_TO` / `QUOTED` edges from the new KD to each `:Person`.
+- **`sourceUrl`:** `https://x.com/<handle>`
+- **`sourceType`:** `'x'` (carried into `:KnowledgeDocument.source`)
+- **`scope`:** `'admin'` (the tweet stream is the operator's authored content)
+
+After document-ingest returns, write the inverse `REPLIED_TO` edges from each inbound-reply author (`:Person`) to the tweet-stream KD — one Cypher MERGE per distinct inbound-replier resolved in Phase 0. Edges carry `accountId`, `sourceDocumentId = attachmentId`, and `createdByAgent = 'x-import'` so re-ingest cleans them up.
+
+### Phase 3 — DM `:ConversationArchive` ingests (one per dmConversation)
+
+For each `dmConversation` in `data/direct-messages.js` (and `data/direct-messages-group.js` when present):
+
+1. **Split.** Write a per-conversation slice file at `{accountDir}/archive/x/<importId>/dm-<conversationId>.json` — a wrapper-stripped JSON array of length 1 whose single element is the dmConversation envelope. This is the file the x-dm normaliser reads.
+
+2. **Resolve participants.** Take the `senderIdToElementId` map from Phase 0; the participant set for this conversation is the distinct senderIds in the slice. The owner is the operator's confirmed `:AdminUser` elementId.
+
+3. **Invoke the bash.** Run in the background (heartbeat protocol per `conversation-archive` SKILL.md):
+
+   ```bash
+   bash platform/plugins/memory/bin/conversation-archive-ingest.sh \
+     {accountDir}/archive/x/<importId>/dm-<conversationId>.json \
+     --source x-dm \
+     --participant-person-ids <ownerEid>,<otherEid>,... \
+     --scope admin \
+     --session-id x-import-<importId>-dm-<conversationId>
+   ```
+
+4. **Surface the three operator messages** (parse → classify → write) from each invocation per the existing conversation-archive contract.
+
+### Phase 4 — Media enclosures
+
+After both the KD and all `:ConversationArchive` writes complete:
+
+- For each tweet-media file, write `(:KnowledgeDocument {tweetStreamKd})-[:HAS_ENCLOSURE]->(:ImageObject)` (image media) or write `(:KnowledgeDocument)-[:HAS_ENCLOSURE { contentUrl }]->(...)` skipping body ingest for binary-only video.
+- Document-shaped media (OCR-able images, PDFs from quote-tweeted articles when archived) recurse through `document-ingest` to land a child `:KnowledgeDocument`, attached via `HAS_ENCLOSURE` from the parent tweet-stream KD.
+- DM media: same `HAS_ENCLOSURE` shape but attaches from the relevant `:ConversationArchive`.
+
+## Diagnostic line
+
+After the full ingest completes, emit a single grep-able line:
+
+```
+[x-import] importId=<id16> tweets=<n> kdCreated=<0|1> kdUpdated=<0|1> people=<p> enclosures=<e> dmConversations=<d> ms=<ms>
+```
+
+Surfaces to the operator as one chat line. `kdCreated + kdUpdated == 1` per tweet-stream ingest (one or the other, never both, never zero on a valid archive).
+
+## Idempotency
+
+- Re-running the same archive bytes is a no-op for the tweet-stream KD — `attachmentId = sha256(transcript-bytes)` matches the prior write; `document-ingest` skips on identical `attachmentId`.
+- A newer archive containing additional tweets produces a different transcript and therefore a different `attachmentId` — the prior KD is replaced (sections regenerated under the same `sourceDocumentId`).
+- DM `:ConversationArchive` re-ingest follows the conversation-archive cursor model: same `conversationIdentity` → delta-append. No `--rebuild` is ever issued autonomously.
+- Person `xHandle` writes are `ON CREATE SET` — re-import never overwrites operator edits.
+
+## Wrapper-grammar defence
+
+Every `.js` shim read by this skill is gated through the wrapper regex documented in [archive-shape.md](../../references/archive-shape.md). Unwrapped-JSON input (missing the `window.YTD.<feed>.part0 = ` prefix on `tweets.js`, `direct-messages.js`, or `tweet-headers.js`) errors at parse with `phase=parse reason="missing window.YTD wrapper on <filename>"`. Silent zero-tweet imports are the failure mode this defence exists to prevent.
+
+## What this skill does not do
+
+- **Per-thread KDs.** One archive = one tweet-stream KD. Thread boundaries are section boundaries the classifier owns, not graph objects.
+- **Bespoke labels.** No `:Post`, `:Tweet`, `:DirectMessage`, `:DMThread`, `:Attachment`. Everything maps to the existing `:KnowledgeDocument` + `:ConversationArchive` + `:ImageObject` shapes.
+- **Tweet-level typed edges.** `:REPLIES_TO` / `:QUOTES` as graph edges off individual tweets are rejected; document-level `REPLIED_TO` / `QUOTED` from the KD to each `:Person` covers the same traversals at the right granularity. The Task 305 typed-edge pass enumerates the new KD and writes any further `COMMITTED_TO` / `MENTIONS` / `REFERENCES` edges from in-body references.
+- **Retweets-without-comment.** Not persisted reliably in X archives; dropped at parse.
+- **Live X API.** Archive-only.

package/payload/platform/templates/specialists/agents/database-operator.md

@@ -3,7 +3,7 @@ name: database-operator
 description: "Graph-write executor reachable from admin via the Task tool. Admin names what should land in the graph; database-operator reads the schema, decides operations, executes them against the memory graph, and reports `{label, elementId}` back. (Reachable historically also from the now-dormant Stop hook in `platform/plugins/admin/hooks/turn-completed-graph-write.sh`; that path is preserved as infrastructure but no longer registered, see Task 214.)"
 summary: "Executes graph writes on admin's behalf when delegated via the Task tool."
 model: claude-sonnet-4-6
-tools: Read, mcp__plugin_memory_memory__memory-write, mcp__plugin_memory_memory__memory-update, mcp__plugin_memory_memory__memory-search, mcp__plugin_memory_memory__memory-typed-edge-pass, mcp__plugin_memory_memory__session-retrospective-skip-rate, mcp__plugin_memory_memory__profile-update, mcp__plugin_memory_memory__profile-read, mcp__plugin_memory_memory__obsidian-vault-import, mcp__plugin_contacts_contacts__contact-create, mcp__plugin_contacts_contacts__contact-update, mcp__plugin_work_work__work-create, mcp__plugin_work_work__work-update, mcp__plugin_work_work__project-create, mcp__plugin_work_work__project-update
+tools: Read, mcp__plugin_memory_memory__memory-write, mcp__plugin_memory_memory__memory-update, mcp__plugin_memory_memory__memory-search, mcp__plugin_memory_memory__memory-archive-write, mcp__plugin_memory_memory__memory-typed-edge-pass, mcp__plugin_memory_memory__session-retrospective-skip-rate, mcp__plugin_memory_memory__profile-update, mcp__plugin_memory_memory__profile-read, mcp__plugin_memory_memory__obsidian-vault-import, mcp__plugin_scheduling_scheduling__schedule-archive-ics, mcp__plugin_contacts_contacts__contact-create, mcp__plugin_contacts_contacts__contact-update, mcp__plugin_work_work__work-create, mcp__plugin_work_work__work-update, mcp__plugin_work_work__project-create, mcp__plugin_work_work__project-update
 ---
 
 You are an expert Neo4J graph operator. Admin delegates a graph-write to you through the Task tool; the prompt admin sends names what should land in the graph in plain English.
@@ -22,7 +22,7 @@ Decide the precise operations against the schema, execute them with the writers
 
 ## Typed-edge auto-extraction pass (session-end)
 
-When admin's brief names the typed-edge pass (Task 305) and supplies a `sinceIso` timestamp, call `memory-typed-edge-pass` with that timestamp exactly once. The tool reads prose-bearing nodes (Message/Page/Meeting/Email/Idea/Note/Post/Report) the account wrote since `sinceIso`, asks Haiku for typed-edge proposals against a closed allowlist, validates each one, and MERGEs accepted edges with `createdBy` provenance. Return the full counter object verbatim — admin needs `nodesProcessed` and `accepted` for the `session-retrospective-mark-complete` call. Do not reinterpret the counts or summarise them.
+When admin's brief names the typed-edge pass (Task 305) and supplies a `sinceIso` timestamp, call `memory-typed-edge-pass` with that timestamp exactly once. The tool reads prose-bearing nodes (Message/Page/Meeting/KnowledgeDocument/Idea/Note/Post/Report — `:KnowledgeDocument` covers email threads via `source:'email'` since Task 321) the account wrote since `sinceIso`, asks Haiku for typed-edge proposals against a closed allowlist, validates each one, and MERGEs accepted edges with `createdBy` provenance. Return the full counter object verbatim — admin needs `nodesProcessed` and `accepted` for the `session-retrospective-mark-complete` call. Do not reinterpret the counts or summarise them.
 
 ## External-archive ingestion
 
@@ -30,8 +30,12 @@ The database-operator also owns ingestion of bulk external archives that map int
 
 - **Obsidian vault import** — when the operator points at an extracted Obsidian vault directory ("import my Obsidian notes", "ingest this vault at `<path>`"), load [platform/plugins/obsidian-import/skills/obsidian-import/SKILL.md](../../../plugins/obsidian-import/skills/obsidian-import/SKILL.md) and run the two-phase flow (owner confirmation → `mcp__plugin_memory_memory__obsidian-vault-import` `mode='dry-run'` → operator disambiguation → `mode='commit'` with resolutions). Parsing, content-hash idempotency, and attachment staging happen entirely inside the tool — you never read the markdown.
 
+- **ICS calendar archive ingest** — when the operator points at an `.ics` file ("ingest this calendar export at `<path>`", "import my Google Calendar export", "load this .ics into the graph"), call `mcp__plugin_scheduling_scheduling__schedule-archive-ics` with `mode='dry-run'` first, present the attendee-resolution preview to the operator (how many already exist under this account, how many would be cross-account-skipped, how many new Person stubs would be created, plus the sample list), then call again with `mode='commit'` after the operator confirms. Each VEVENT becomes one `:Meeting` MERGEd on `(accountId, uid)` (idempotent re-ingest); attendees resolve to `:Person` by lowercased email; the importing `:AdminUser` anchors every meeting via `[:IMPORTED]`. Cross-account email collisions skip the attendance edge — do not propose to override this. Description bodies pass to `:Meeting.description` and the typed-edge pass picks up commitments + mentions automatically; you do not call `memory-write` for them.
+
 Other archive types (LinkedIn export, Notion export, Logseq, Roam, gbrain-repo, Gmail) live under separate plugins owned by `specialists:librarian`; they are not on this surface. If the operator's request points at one of those, hand back to admin so it can dispatch the right specialist.
 
+- **X (Twitter) archive import** — when the operator points at an extracted X archive directory (`Archive of <handle>/`, or a `twitter-<handle>-archive.zip` path), load [platform/plugins/x-import/skills/x-import/SKILL.md](../../../plugins/x-import/skills/x-import/SKILL.md) and run its four-phase flow (owner + participant confirmation → tweet-stream transcript rendering → `document-ingest` dispatch for the tweet-stream `:KnowledgeDocument` → one `conversation-archive-ingest.sh --source x-dm` invocation per distinct DM participant set). Every `@handle` and DM senderId resolves to an existing `:Person` / `:AdminUser` before any write; participants are never auto-created. Writes route through the existing `memory-ingest` and `conversation-archive` surfaces — no new graph writer, no `:Post` / `:DirectMessage` labels.
+
 ## Skip-rate visibility (session-start)
 
 When admin's brief names the skip-rate check, call `session-retrospective-skip-rate` once and return its result verbatim. The tool reports how many of the last N sessions for this account closed without firing the retrospective. Admin uses the rate to inform the operator that closing via Sidebar Archive or tab-close defers the typed-edge pass to the next `/end`. Pass the `windowSize` admin provides; default 10 is fine when absent.