@rubytech/create-realagent 1.0.828 → 1.0.830

package/payload/platform/plugins/whatsapp-import/skills/whatsapp-import/references/conversation-archive-shape.md

@@ -1,143 +0,0 @@
-# Conversation Archive — graph shape, identity, edges, delta protocol
-
-The reference document for the `:ConversationArchive` shape introduced by Task 891. This is the source of truth for what the graph looks like after a `whatsapp-ingest.sh` run — the SKILL.md prescribes the workflow; this file specifies the schema.
-
-## Labels
-
-| Label | Role | MERGE key | Schema constraint |
-|---|---|---|---|
-| `:ConversationArchive` | Parent node — one per chat | `conversationIdentity` | `FOR (a:ConversationArchive) REQUIRE a.conversationIdentity IS UNIQUE` |
-| `:Section:Conversation` | Topic-bounded chunk of messages | (no MERGE — CREATE only) | inherits `:Section` indices |
-
-`:Section:Conversation` is the chat-mode counterpart of document-mode `:Section:Chapter` etc. Same `:Section` base label, same indexing, different secondary label.
-
-## Identity formula
-
-```
-conversationIdentity = sha256(accountId + ":" + sortedParticipantElementIds.join(","))
-```
-
-- Stable across re-exports — same accountId + same operator-confirmed participant set always produces the same identity, regardless of the source file's bytes.
-- DM and group are identical — the difference is array length.
-- Participant order is sorted before joining, so the operator can supply elementIds in any order.
-
-## Required indices
-
-```
-INDEX :ConversationArchive(accountId)
-INDEX :ConversationArchive(createdBySession)
-```
-
-Plus the constraint above (which doubles as a uniqueness index on `conversationIdentity`).
-
-## Properties on `:ConversationArchive`
-
-| Property | Type | Source | When set |
-|---|---|---|---|
-| `conversationIdentity` | string (sha256-hex) | derived | ON CREATE |
-| `accountId` | string (UUID) | argv | ON CREATE |
-| `scope` | string (`admin` / `public`) | argv | ON CREATE |
-| `summary` | string | classifier (synthetic) | ON CREATE |
-| `keywords` | string[] | classifier (aggregated across sessions) | ON CREATE |
-| `embedding` | float[] | embed(summary) | ON CREATE |
-| `archiveSourceFile` | string (basename) | this export | ON CREATE |
-| `createdAt` | ISO 8601 | this run | ON CREATE |
-| `createdByAgent` | string | constant `"whatsapp-import"` | ON CREATE |
-| `createdBySession` | string (UUID) | env / argv | ON CREATE |
-| `source` | string | constant `"whatsapp"` | ON CREATE |
-| `updatedAt` | ISO 8601 | this run | ON CREATE / ON MATCH |
-| `lastIngestedMessageHash` | string (sha256-hex) | derived from last delta line | ON CREATE / ON MATCH |
-| `lastIngestedMessageAt` | ISO 8601 | last delta line's `dateSent` | ON CREATE / ON MATCH |
-| `lastIngestedBySession` | string (UUID) | this run | ON CREATE / ON MATCH |
-| `lastIngestedArchiveSha256` | string (sha256-hex) | this export's file bytes | ON CREATE / ON MATCH |
-
-## Properties on `:Section:Conversation` (each chunk)
-
-| Property | Type | Source |
-|---|---|---|
-| `accountId` | string | inherited |
-| `title` | string | classifier |
-| `body` | string | classifier (verbatim turn-attributed text — `[ts] Sender: body\n…`) |
-| `bodyPreview` | string (≤150 chars) | first 150 chars of body |
-| `position` | int | chunk index within this run |
-| `scope` | string | inherited |
-| `embedding` | float[] | embed(body) |
-| `summary` | string | classifier (1–3 sentences) |
-| `keywords` | string[] | classifier |
-| `firstMessageAt` | ISO 8601 | first `[ts]` in chunk |
-| `lastMessageAt` | ISO 8601 | last `[ts]` in chunk |
-| `participantNames` | string[] | distinct senderNames in chunk |
-| `messageCount` | int | message count in chunk |
-| `archiveSha256` | string (sha256-hex) | this export's file bytes (cleanup discriminator) |
-| `archiveSourceFile` | string (basename) | this export |
-| `createdAt` | ISO 8601 | this run |
-| `createdByAgent` / `createdBySession` / `source` | provenance | this run |
-
-## Edges
-
-| Edge | From | To | Cardinality | When written |
-|---|---|---|---|---|
-| `:HAS_SECTION` | `:ConversationArchive` | `:Section:Conversation` | one per chunk | every run |
-| `:NEXT` | `:Section:Conversation` | `:Section:Conversation` | chunks − 1 (chronological chain across all sessions across all runs) | extending tail per run |
-| `:PARTICIPANT_IN` | `:Person` / `:AdminUser` | `:ConversationArchive` | one per confirmed participant | MERGEd every run (idempotent) |
-
-Every edge carries `createdAt`, `createdByAgent`, `createdBySession`, `source`, plus `archiveSha256` for HAS_SECTION + NEXT (so cleanup-by-archiveSha256 catches the right edges).
-
-## Delta-append protocol
-
-```
-                        first ingest (or empty graph)
-                                │
-                                ▼
-   parsed_lines  ──── all of them ────► sessionize ──► classify ──► memory-ingest
-                                                                       │
-                                                                       ▼
-                                                       :ConversationArchive (NEW)
-                                                       └── :HAS_SECTION ──► chunks (NEXT chain length K-1)
-
-
-                        re-import (delta)
-                                │
-                                ▼
-   parsed_lines  ──┐
-                   │
-                   ▼
-              find cursor where deriveMessageContentHash(line) == archive.lastIngestedMessageHash
-                   │
-        ┌──────────┼──────────┬───────────────────┐
-        │          │          │                   │
-       found     missing    empty (cursor at last line)
-        │          │          │
-        ▼          ▼          ▼
-   slice from   FAIL    noop (exit 0,
-   cursor+1     non-zero  no writes)
-        │
-        ▼
-   delta_lines ──► sessionize ──► classify ──► memory-ingest (parentLabel='ConversationArchive')
-                                                  │
-                                                  ▼
-                                  :ConversationArchive (MERGE on conversationIdentity)
-                                  └── :HAS_SECTION ──► NEW chunks
-                                  ── :NEXT extends from prior tail
-                                  prior chunks unchanged
-                                  cursor advances
-```
-
-## Why the cursor is content-only (not file-byte-based)
-
-`lastIngestedMessageHash = sha256(dateSent + "|" + NFKC-trim-lower(senderName) + "|" + body)`. The hash deliberately excludes the source file's bytes — a fresh re-export of the same chat has different file bytes (different SHA-256 of the file) but the SAME message tuples. Without content-only hashing, every delta-import would `delta-cursor-missing` because the file SHA-256 always changes.
-
-## Why DM and group are identical
-
-The brief's "DM-only" Task 887 §A0 contract was a workaround for the per-message writer's auto-Person leak. Under the chunked archive shape, the writer never auto-creates anyone — every participant is operator-confirmed up front. The 2-vs-3-vs-N participant case is just the array length on the right of the identity formula. No special-casing.
-
-## Provenance discipline (for cleanup correctness)
-
-Every node and edge written by this pipeline is stamped with:
-- `source = 'whatsapp'`
-- `createdByAgent = 'whatsapp-import'`
-- `createdBySession = <this run's session UUID>`
-- `archiveSha256 = <this export's file SHA-256>` (chunks + HAS_SECTION + NEXT only — the parent records the LAST archiveSha256 separately)
-- `archiveSourceFile = <this export's basename>` (parent + chunks)
-
-The cleanup-by-`archiveSha256` step in memory-ingest's chat path drops only chunks whose `archiveSha256` matches THIS run's. Re-running the same export bytes is a no-op idempotently; re-running with a fresh delta export (different bytes, different SHA-256) leaves prior chunks untouched and appends new ones at the tail.

package/payload/platform/plugins/whatsapp-import/skills/whatsapp-import/references/export-parse.md

@@ -1,109 +0,0 @@
-# Reference: `_chat.txt` parsing — implementation reference
-
-> **This is no longer operator instruction.** The agent does NOT walk this grammar in its own LLM turn. Parsing runs deterministically in [`platform/plugins/whatsapp-import/lib/src/parse-export.ts`](../../../lib/src/parse-export.ts), invoked in-process by [`bin/ingest.mjs`](../../../bin/ingest.mjs) (which the operator calls via [`bin/whatsapp-ingest.sh`](../../../bin/whatsapp-ingest.sh) — the single deterministic Bash entry). The legacy MCP wrapper is blocked at the harness gate. The vitest grid in [`lib/src/__tests__/parse-export.test.ts`](../../../lib/src/__tests__/parse-export.test.ts) is the executable contract; this prose is the human-readable companion. Extend the grammar by adding a failing test first.
-
-WhatsApp's "Export Chat" produces a UTF-8 text file with a deterministic line grammar. This reference describes what the parser library does when it converts that file into the `{senderName, dateSent, body, sequenceIndex}[]` structure the SKILL.md consumes.
-
-## File-open invariants
-
-1. **UTF-8 only.** Open the file with explicit UTF-8 decoding. On encoding error, abort the import with a named LOUD-FAIL — never silently substitute or corrupt bodies. WhatsApp's modern apps emit UTF-8 reliably; an encoding error usually means the operator manually edited the file with a tool that broke it. Surface the named error so they can re-export.
-2. **No size cap from the parser.** The parser handles arbitrarily large files; the [SKILL.md](../SKILL.md)'s 100-message selective-ingest gate is the operator-facing compression layer.
-3. **Compute `archiveSourceFile = sha256(file bytes)` first.** The hash drives `conversationId` and lets re-imports of the same archive land idempotently.
-
-## Line grammar
-
-Every message line begins with a square-bracketed timestamp prefix followed by `<Sender>: <body>`:
-
-```
-[DD/MM/YYYY, HH:MM:SS] <Sender>: <body>      ← modern WhatsApp default (4-digit year)
-[DD/MM/YY, HH:MM:SS] <Sender>: <body>        ← legacy exports (2-digit year)
-```
-
-- **Day/month ordering.** `DD/MM` is the WhatsApp default everywhere except US iOS, which emits `MM/DD`. The parser auto-detects from the first prefix-matching line when `dateFormat` is omitted: probe DD/MM first; if range-valid, lock DD/MM; otherwise lock MM/DD. The lock is per-file — a single export never mixes orderings (the locale is set by the device that generated the export). Manually concatenated multi-locale archives are an explicit out-of-scope: pass `dateFormat` to override.
-- **Year shape.** Both 2-digit (`\d{2}` → `2000+yy`) and 4-digit (`\d{4}` → as-is) years are accepted by the same regex (`\d{2,4}`). A single file may hold both shapes; year semantics are resolved per-line from the captured length, not per-file.
-- **Time.** `HH:MM:SS` 24-hour; older exports may emit `HH:MM` (no seconds — treat as `:00`).
-- **Sender.** Saved contact name, phone number with country code (`+44 7700 900123`), or `You` for legacy operator-sent messages.
-- **Body.** Message text, possibly multi-line.
-
-Trim trailing whitespace from each line before parsing.
-
-## Multi-line bodies
-
-A body that wraps to multiple lines continues onto subsequent lines that do **not** match the timestamp-prefix pattern. The parser must accumulate these continuation lines into the previous message's body, joining with `\n`. End-of-message is detected by the next timestamp prefix or end-of-file.
-
-```
-[14/03/26, 10:15:23] Joel: Quick question about the deck —
-do you have the v3 PDF anywhere?
-I checked Drive and only see v2.
-[14/03/26, 10:16:01] Sarah: Sec, will dig it out
-```
-
-Yields two messages:
-- `Joel: "Quick question about the deck —\ndo you have the v3 PDF anywhere?\nI checked Drive and only see v2."`
-- `Sarah: "Sec, will dig it out"`
-
-## System messages — skip with counter
-
-WhatsApp injects system-generated lines into the export for group events, contact changes, and security messages. These lines match the timestamp prefix but are **not** sent by a person and have no first-class graph value. Skip them at parse time, increment `systemSkipped`, and do not pass to `memory-archive-write`.
-
-Patterns to recognise (English; localisation expands the list):
-
-- `<Sender> created group "<name>"`
-- `<Sender> changed the subject from "<old>" to "<new>"`
-- `<Sender> changed this group's icon`
-- `<Sender> added <other>`
-- `<Sender> removed <other>`
-- `<Sender> left`
-- `<Sender>'s security code changed.`
-- `Messages and calls are end-to-end encrypted. ...` (the conversation header)
-- `You deleted this message.`
-- `This message was deleted.`
-
-Heuristic for the parser: if the body contains no spaces between `<Sender>` and a verb-phrase token, AND the body lacks a colon-after-sender separator, treat as a system message. Conservative — when uncertain, prefer to ingest as a message; the insight pass tolerates noisy bodies better than the parser would tolerate dropped real messages.
-
-## Media attachments — skip with counter
-
-Lines whose body indicates a media-only message (no text content) get skipped at parse time, increment `mediaSkipped`. Patterns:
-
-- `<Media omitted>` (when the operator chose "Without Media" on export)
-- `IMG-<digits>-<digits>.jpg (file attached)` / `.jpeg` / `.png` / `.heic` / `.gif`
-- `VID-<digits>-<digits>.mp4 (file attached)`
-- `PTT-<digits>-<digits>.opus (file attached)` (voice notes)
-- `AUD-<digits>-<digits>.opus (file attached)` (audio)
-- `STK-<digits>-<digits>.webp (file attached)` (stickers)
-- `<filename>.pdf (file attached)`
-- `<filename>.docx (file attached)`
-- `‎<...> attached: <filename>` (alternative format on some platforms)
-
-Mixed messages (text + media reference in one body) are kept as messages — only pure-media-only lines are skipped. The text body is retained.
-
-## Forwarded messages
-
-A forwarded message is prefixed with the invisible Unicode `‎` (U+200E LEFT-TO-RIGHT MARK) followed by metadata WhatsApp injects. Parse the body normally; the LRM character is preserved in `body` (the insight pass's classifier sees it as benign). Do not strip — the raw body's fidelity matters for downstream queries.
-
-## Edge cases
-
-- **Empty body** (timestamp prefix followed by sender colon but no text). Rare. Skip with `systemSkipped` increment — usually corresponds to a deleted message stub.
-- **Leading BOM** (U+FEFF at file start). Strip before parsing the first line.
-- **Mixed line endings** (`\r\n` vs `\n`). Normalise to `\n` before tokenisation.
-- **Sender containing a colon** (e.g., a contact named "Joel: Work"). The grammar splits on the FIRST `: ` (colon-space) after the timestamp prefix's closing `]`. Subsequent colons in the sender or body are preserved verbatim.
-
-## Parser output shape
-
-The parser returns `{conversationId, archiveSourceFile, parsedLines[], counters}` where:
-
-- `parsedLines[]: Array<{senderName: string, dateSent: string (ISO 8601 with operator-supplied timezone), body: string, sequenceIndex: number}>`
-- `counters: {parsed: n, systemSkipped: n, mediaSkipped: n, parseErrors: n}`
-
-The skill consumes this directly. The `messageId` is computed by the skill (not the parser) so the `lineHash` covers the original raw line, not the post-parse normalised body.
-
-## When to LOUD-FAIL
-
-The parser throws (and `whatsapp-export-parse` returns `isError: true`) on:
-
-- Encoding error at file open (UTF-8 decode fails — the parser uses `TextDecoder` with `fatal: true`, so any invalid byte sequence aborts loudly rather than silently substituting U+FFFD).
-- Empty file or zero parsed lines after walking the file (the file isn't a `_chat.txt`). The thrown error and the `[whatsapp-import] parse-grammar-miss first-line="<sample>"` stderr line both carry a sanitised first-line sample (control chars stripped, truncated to 80 chars) so the operator can recognise the offending header shape without re-running with a debugger.
-- A timestamp prefix matches but the body parse fails (no `: ` separator after the closing `]` AND no system-pattern match) — emits `parse-error file=<...> line=<n> reason=no-sender-body-separator content="<...>"`.
-- Missing required input (`accountId`, `timezone`).
-
-Never silently drop data the parser couldn't classify. The operator chooses to skip; the parser does not choose for them.