mercury-agent 0.16.0 → 0.16.2

package/docs/goals/whatsapp-bot-hardening/archive/media-size-and-silent-drop.md

@@ -0,0 +1,178 @@
+# Media Size Cap + No Silent Drops
+
+**Status**: Done
+**Slug**: media-size-and-silent-drop
+**Goal**: whatsapp-bot-hardening
+**Milestone**: 4
+**Created**: 2026-08-09
+**Last updated**: 2026-08-09
+
+---
+
+## Goal
+
+> Part of goal [whatsapp-bot-hardening](../goals/whatsapp-bot-hardening/goal.md).
+> HANDOFF Bug 3.
+
+A 50 MB docx sent to the greece space was rejected seven times by the 10 MB cap
+(`mercury.yaml:74`, enforced pre-download at `whatsapp-media.ts:165` and `:284`
+for the quoted-reply retry) — and the worse half was the **silence**: the
+"could not use your attachment" note only fires when the message has no text,
+so a captioned file left the agent uninformed and it invented a wrong
+explanation, sending the user chasing a non-existent bug. Raise the cap to
+64 MB (D-004 — `MERCURY_MEDIA_MAX_SIZE_MB` verified unset, so the YAML edit
+takes effect) and make every dropped attachment produce an explanatory note.
+
+## User Stories
+
+- As a group member, I want to send a ~50 MB document and have the bot read it.
+- As any user, I want the bot to *tell me why* it can't use an attachment
+  (too large / media disabled / download failed) instead of guessing.
+
+## MVP Scope
+
+**In scope:** `media.max_size_mb: 64` in the live `mercury.yaml`; an
+unconditional `promptNote`-style system note on any dropped attachment,
+distinguishing the three failure reasons, copied from the existing
+`media-gate.ts` pattern.
+
+**Out of scope:** download→extract-text→discard for oversize documents —
+queued as a follow-up idea (HANDOFF fix 3, optional item).
+
+---
+
+## Context for Claude
+
+- `~/whatsapp-bot/mercury.yaml:74` (`max_size_mb`) — live config, edit inside
+  WSL; confirm the effective value, not just the YAML (env overrides win).
+- `src/adapters/whatsapp-media.ts:165` and `:284` — the pre-download size gates
+  (initial + quoted-reply retry paths).
+- `src/core/runtime.ts` — where the existing text-only-when-no-caption
+  attachment note lives.
+- `src/core/media-gate.ts` — the `promptNote` pattern to copy.
+- `HANDOFF.md` Bug 3 — evidence incl. the journal WARN lines.
+
+---
+
+## Architecture & Data
+
+### Data Models
+
+None. Config change:
+
+```yaml
+media:
+  max_size_mb: 64
+```
+
+### API Contracts
+
+None.
+
+### File & Folder Structure
+
+| Path | New / Modified | Purpose |
+|------|---------------|---------|
+| `~/whatsapp-bot/mercury.yaml` | Modified | Cap 10 → 64 (live config, not in the repo) |
+| `src/adapters/whatsapp-media.ts` | Modified | Surface the drop reason to the ingress path (both gates) |
+| `src/core/runtime.ts` | Modified | Always inject the note when an attachment is dropped, caption or not |
+| test files alongside | New/Modified | See tests below |
+
+### Implementation Sequence
+
+1. Thread the drop reason from both size gates in `whatsapp-media.ts` so the
+   runtime knows *that* and *why* an attachment was skipped.
+   - Read first: `src/adapters/whatsapp-media.ts:150-300`
+   - Verify: unit — an over-limit attachment yields a structured drop reason on
+     both the initial and quoted-reply paths.
+2. Unconditional note in `runtime.ts` using the `media-gate.ts` `promptNote`
+   pattern — fires regardless of caption text; wording distinguishes
+   "too large (limit N MB)" / "media disabled" / "download failed".
+   - Read first: `src/core/runtime.ts` (existing attachment-note branch),
+     `src/core/media-gate.ts`
+   - Verify: unit — over-limit attachment **with a caption** still produces a
+     note; note text differs across the three reasons.
+3. Raise the live cap: edit `mercury.yaml` in WSL, confirm the *effective*
+   value (env overrides beat YAML), restart.
+   - Read first: memory note "Env overrides beat mercury.yaml"
+   - Verify: journal shows the new `maxBytes` on the next large-file attempt.
+
+---
+
+## Non-Negotiable Rules
+
+- Confirm `MERCURY_MEDIA_MAX_SIZE_MB` is still unset before trusting the YAML
+  edit (env silently wins). Do not Read `.env` — check the effective config.
+- The note must never leak file contents — filename, type, size, reason only.
+
+---
+
+## Edge Cases & Risks
+
+| Scenario | Handling |
+|----------|---------|
+| ~200 MB file | Still dropped; the bot now *says why* (manual test case) |
+| Disk pressure from 64 MB files | Accepted (D-004); revisit if it bites |
+| Quoted-reply retry of a large file | Same note — the `:284` gate is covered too |
+| Env override added later | Memory note + rule above; check effective value when debugging |
+
+---
+
+## Implementation Checklist
+
+### Phase 1 — Goal
+- [x] Goal, User Stories, and MVP Scope written and reviewed
+
+### Phase 2 — Architecture
+- [x] Architecture & Data section complete
+- [x] Non-Negotiable Rules defined
+- [x] Edge Cases covered
+- [x] Context for Claude pointers filled in
+
+### Implementation (commit `5734c04`)
+- [x] Drop reason threaded (step 1) — all four drop paths, not just the two size gates
+- [x] Unconditional note (step 2)
+- [x] Live cap raised + **effective** value confirmed (`mediaMaxSizeMb: 64`)
+- [x] 7 unit tests green; `bun run check` passes (1411 pass / 0 fail)
+- [ ] **Manual (needs a phone):** ~50 MB docx downloads and is readable
+- [ ] **Manual (needs a phone):** ~200 MB file → bot explains why it can't
+- [x] No secrets or `.env` files committed
+- [x] No unrelated files modified
+
+---
+
+## Open Questions
+
+- [ ] none
+
+---
+
+## Retrospective
+
+**Residual risks / follow-ups:**
+- Oversize-document text extraction → ideas/oversize-doc-extraction.md (create if wanted)
+
+**What changed from the plan:**
+- The spec assumed the note only needed unblocking. Two upstream holes had to
+  be closed first, both found by reading the path rather than trusting it:
+  1. **The WhatsApp bridge never set `hadIncomingAttachments`.** The existing
+     no-text branch that was supposed to speak up could therefore never fire
+     on WhatsApp — the one platform where the bug was reported.
+  2. **`normalize()` returned null when there was no text and no attachment**,
+     so an oversize file sent *without* a caption vanished before the runtime
+     saw it at all. A recorded drop is now reason enough to build a message.
+- Reasons are reported from four paths, not the two the spec named: the
+  pre-download size check, the post-download size check, an empty buffer, and
+  a download exception. The last two are how "download failed" becomes
+  distinguishable from "too large".
+
+**Key decisions made during implementation:**
+- The reason travels as structured data (`DroppedAttachment`) rather than a
+  pre-formatted string, so the note wording lives in one place and the note
+  can state both numbers for an over-limit file.
+- The note explicitly instructs the agent not to invent an alternative
+  explanation. The original failure was not silence alone — it was the
+  confident wrong story the agent told to fill it.
+
+**Architecture impact** (update `docs/ARCHITECTURE.md` if any of these apply):
+- [ ] none expected

package/docs/goals/whatsapp-bot-hardening/archive/member-memory.md

@@ -0,0 +1,199 @@
+# Member Memory
+
+**Status**: Done
+**Slug**: member-memory
+**Goal**: whatsapp-bot-hardening
+**Milestone**: 4
+**Created**: 2026-08-09
+**Last updated**: 2026-08-09
+
+---
+
+## Goal
+
+> Part of goal [whatsapp-bot-hardening](../goals/whatsapp-bot-hardening/goal.md).
+> HANDOFF Bug 2, "Member memory feature" — mostly prompt + skill work once
+> attribution (M2.1) lands.
+
+Give the bot durable per-member knowledge in each space: **hard identity** (a
+`<members>` block generated from `space_roles` — id → name from the DB, never
+guessable by the model) plus **soft characteristics** (per-space
+`knowledge/members/<id>.md` notes the agent accumulates itself with its
+existing `write` tool), injected into the prompt the same way
+`<active_episodes>` already is (`container-entry.ts:513`). Built on existing
+episodic plumbing — no new subsystem, no schema migration for the notes half
+(D-010).
+
+## User Stories
+
+- As a group member, I want the bot to know who the 26 people in the space are
+  by name and id, reliably, from the DB.
+- As a user, I want the bot to remember soft facts it learns ("guy prefers
+  short answers", "Sahar admins the space") across sessions, per space.
+
+## MVP Scope
+
+**In scope:** `<members>` block from `space_roles` injected per turn;
+`knowledge/members/<id>.md` convention + injection alongside episodes; skill
+guidance telling the agent when/how to write member notes; per-space isolation.
+
+**Out of scope:** cross-space member profiles (naturally per-space by design);
+automatic summarization/consolidation of notes (napkin-style — future);
+any DB table for notes (D-010 rejected it).
+
+---
+
+## Context for Claude
+
+- `src/agent/container-entry.ts:513` — `<active_episodes>` injection; the
+  pattern to mirror for both `<members>` and member notes.
+- `space_roles` table — id → `display_name`, the hard-identity source (verified
+  clean, 26 members in greece2026).
+- `.mercury/spaces/<id>/` workspace layout — where `knowledge/members/` lives;
+  the agent's `write` tool already reaches it.
+- `message-author-attribution` spec (M2.1) — supplies the ids that make notes
+  addressable; must be Done first.
+- `HANDOFF.md` Bug 2, member-memory paragraph.
+
+---
+
+## Architecture & Data
+
+### Data Models
+
+None (workspace files + prompt rendering only).
+
+```
+.mercury/spaces/<space>/knowledge/members/<sanitized-member-id>.md
+```
+
+File naming: sanitized canonical id (e.g. `972500000000.md`) so the agent can
+address a note file from the `<members>` block without guessing.
+
+### API Contracts
+
+Prompt contract (rendered per turn):
+
+```xml
+<members>
+  <member id="whatsapp:972500000000@s.whatsapp.net" name="Member Name" role="member"/>
+  …
+</members>
+<member_notes>
+  <note member="whatsapp:9725...">…file contents…</note>   <!-- only files that exist -->
+</member_notes>
+```
+
+Exact tag names to match surrounding prompt idiom in `container-entry.ts`.
+
+### File & Folder Structure
+
+| Path | New / Modified | Purpose |
+|------|---------------|---------|
+| `src/agent/container-entry.ts` | Modified | Render `<members>` from space_roles + inject existing member note files |
+| space workspace `knowledge/members/` | New (runtime) | Agent-written notes |
+| skill / AGENTS.md guidance | Modified | When to write a member note, what belongs there, what doesn't |
+| test files alongside | New/Modified | See tests below |
+
+### Implementation Sequence
+
+1. Render the `<members>` block from `space_roles` in the container entry
+   (group spaces only; cap or elide gracefully for very large member lists).
+   - Read first: `src/agent/container-entry.ts:490-540` and the query helpers
+     it already uses
+   - Verify: unit — prompt for a group space contains each member id + name
+     from the DB; DM spaces unaffected.
+2. Inject `knowledge/members/*.md` files that exist, following the episodes
+   injection pattern (size-bounded).
+   - Read first: the episodes injection code path
+   - Verify: unit — a note file present in the workspace appears in the
+     prompt; absent files add nothing; oversized notes are truncated bounded.
+3. Skill/persona guidance: teach the agent to append observations to
+   `knowledge/members/<id>.md` via its `write` tool — identity facts stay
+   hard (from `<members>`), notes are observations only.
+   - Read first: existing space `AGENTS.md` + any relevant skill
+   - Verify: text review; manual test below.
+
+---
+
+## Non-Negotiable Rules
+
+- Identity (id ↔ name) always comes from `space_roles` — the model must never
+  be positioned to invent or override it; notes must not restate identity.
+- Notes are per-space; no cross-space leakage.
+- Prompt injection is size-bounded (member list and notes both) so a big group
+  can't blow the context budget — same discipline as D-003.
+
+---
+
+## Edge Cases & Risks
+
+| Scenario | Handling |
+|----------|---------|
+| Member with no display_name yet | Render id-only entry; name appears once known |
+| Very large group | Bounded rendering (cap + count note) |
+| Agent writes junk/sensitive notes | Skill guidance constrains content; notes are plain files — reviewable and deletable by the operator |
+| Member leaves the space | `space_roles` drives the block; stale note file stays until deleted (accepted) |
+| Notes grow unbounded | Size-bounded injection now; consolidation deferred (out of scope) |
+
+---
+
+## Implementation Checklist
+
+### Phase 1 — Goal
+- [x] Goal, User Stories, and MVP Scope written and reviewed
+
+### Phase 2 — Architecture
+- [x] Architecture & Data section complete
+- [x] Non-Negotiable Rules defined
+- [x] Edge Cases covered
+- [x] Context for Claude pointers filled in
+
+### Implementation (commit `44a7542`)
+- [x] `<members>` rendering (step 1)
+- [x] Note injection (step 2)
+- [x] Skill guidance (step 3)
+- [x] 12 unit tests green; `bun run check` passes (1423 pass / 0 fail)
+- [x] **Verified live:** asked the bot to list the `<members>` block without
+      guessing — it returned the real ids, names and roles from `space_roles`
+      (and correctly reported the block's *absence* before the image rebuild,
+      which is how the stale-image problem below was caught)
+- [ ] **Manual (needs a phone):** after real group chat, "what do you know
+      about <member>?" → correct identity + accumulated notes
+- [x] No secrets or `.env` files committed
+- [x] No unrelated files modified
+
+---
+
+## Open Questions
+
+- [ ] Should the agent be told to ask before recording personal observations
+  about members (privacy etiquette in a family group)? — owner: user
+
+---
+
+## Retrospective
+
+**Residual risks / follow-ups:**
+- Note consolidation (napkin-style distillation) — ideas/member-notes-consolidation.md (create if wanted)
+
+**What changed from the plan:**
+- The `<members>` data could not be read in the container the way episodes
+  are — `space_roles` lives in the host database — so it rides on the
+  container payload instead. `payload = { ...input }` meant adding it to the
+  runner's input type was enough for it to flow through.
+- Members are ordered named-first so a truncated list keeps the rows that
+  carry usable information; an id with no name tells the model nothing.
+
+**Key decisions made during implementation:**
+- Note bodies are XML-escaped even though the agent writes them. A note is
+  assembled from things people said in a group, so an unescaped body is a path
+  from group chat to forged prompt markup — the same hole closed in M2.1.
+- The guidance draws the hard/soft line explicitly (notes must not restate
+  id-to-name mappings), so the two sources of identity can never disagree.
+- Privacy is handled by instruction plus the storage choice: notes are plain
+  files the operator can read or delete, and the agent is told not to record
+  what someone would be uncomfortable reading, and to delete on request.
+
+**Architecture impact** (update `docs/ARCHITECTURE.md` if any of these apply):
+- [ ] New inter-package interaction (container-entry ← space_roles/members rendering)

package/docs/goals/whatsapp-bot-hardening/archive/message-author-attribution.md

@@ -0,0 +1,211 @@
+# Message Author Attribution
+
+**Status**: Done
+**Slug**: message-author-attribution
+**Goal**: whatsapp-bot-hardening
+**Milestone**: 2
+**Created**: 2026-08-09
+**Last updated**: 2026-08-09
+
+---
+
+## Goal
+
+> Part of goal [whatsapp-bot-hardening](../goals/whatsapp-bot-hardening/goal.md).
+> HANDOFF Bug 2.
+
+The bot attributes past group messages to the wrong person because stored
+messages carry **no author** — `messages` has no author column, history renders
+as bare `<user>…</user>`, and the only identity in the prompt is the *current*
+caller. Attributing all history to that one name is the only reading the prompt
+supports; this is a prompt bug, not a model glitch. Additionally, quoted-reply
+senders resolve to raw LIDs after every restart because `pushNames` is
+in-memory only. The clean identity data already exists in
+`space_roles.display_name` (26 members, keyed by canonical phone JID) — nothing
+uses it. This story lands **before** ambient (M2.2) so ambient text is
+attributed from day one.
+
+## User Stories
+
+- As a group member, I want the bot to know who said what so that "who said X?"
+  gets the right answer.
+- As a group member, I want quoted replies to show real names even after the
+  service restarts, not `136043810578453@lid`.
+
+## MVP Scope
+
+**In scope:** `author_id`/`author_name` columns + migration, populated at
+`addMessage`; `<user name id>` rendering in history XML (incl. ambient
+entries); DB-backed `pushNames` (seed at startup, write-through); async
+`canonicalizeJid` for quoted participants.
+
+**Out of scope:** backfilling authors for existing rows — they stay NULL and
+must still render; member *characteristics* memory — that's `member-memory`
+(M4.3).
+
+---
+
+## Context for Claude
+
+- `src/core/runtime.ts:1573` — `addMessage(spaceId, "user", finalPrompt, …)`;
+  the write site that must start passing author identity.
+- `src/agent/container-entry.ts:598` (`<user>` rendering in `buildHistoryXml`)
+  and `:634` (`<caller id name role/>` — the only identity today).
+- `src/adapters/whatsapp.ts:82` (`buildReplyContext`) and `:169` (`pushNames`
+  in-memory map) — the LID/restart bug.
+- `space_roles` table — `display_name` fully populated, canonical phone JIDs;
+  the seed source.
+- `HANDOFF.md` Bug 2 — full root-cause analysis.
+
+---
+
+## Architecture & Data
+
+### Data Models
+
+```sql
+-- migration: add author columns; existing rows backfill NULL
+ALTER TABLE messages ADD COLUMN author_id TEXT;    -- canonical JID, e.g. whatsapp:972...@s.whatsapp.net
+ALTER TABLE messages ADD COLUMN author_name TEXT;  -- display name at write time
+```
+
+Migration must be idempotent against the live DB (follow the existing `db.ts`
+migration pattern). NULL-author rows render as today's bare `<user>`.
+
+### API Contracts
+
+None (internal schema + prompt rendering only).
+
+### File & Folder Structure
+
+| Path | New / Modified | Purpose |
+|------|---------------|---------|
+| `src/storage/db.ts` | Modified | Migration + `addMessage` signature carries author |
+| `src/core/runtime.ts` | Modified | Pass author identity at every `addMessage` call site (incl. ambient at `:604` — promote the `Name: text` prefix to real fields) |
+| `src/agent/container-entry.ts` | Modified | `buildHistoryXml` renders `<user name="…" id="…">` |
+| `src/adapters/whatsapp.ts` | Modified | `pushNames` seeded from `space_roles.display_name` at startup + write-through; `buildReplyContext` uses async `canonicalizeJid` |
+| test files alongside each | New/Modified | See tests below |
+
+### Implementation Sequence
+
+1. Schema migration + `addMessage` author params.
+   - Read first: `src/storage/db.ts` (existing migration pattern, `addMessage`)
+   - Verify: migration runs idempotently on a **copy** of the live DB; existing
+     rows have NULL authors; `bun run check` green.
+2. Thread author identity through `runtime.ts` write sites — the ingress
+   message already carries the sender; promote the ambient `Name:` prefix
+   (`runtime.ts:604`) to the new columns too.
+   - Read first: `src/core/runtime.ts:1560-1620` and `:591-612`
+   - Verify: unit test — a stored group message row has `author_id` +
+     `author_name` matching the sender.
+3. Render authors in history: `<user name="…" id="…">` in `buildHistoryXml`;
+   NULL-author rows render without the attributes.
+   - Read first: `src/agent/container-entry.ts:580-640`
+   - Verify: unit test — history XML for messages from two different users
+     renders two distinct names; NULL-author rows still render.
+4. Persist `pushNames`: seed from `space_roles.display_name` at adapter
+   startup, write through on every message carrying a push name.
+   - Read first: `src/adapters/whatsapp.ts:150-200`
+   - Verify: unit test — simulated restart (fresh map, seeded from DB) still
+     resolves a known member's name.
+5. Async LID resolution in `buildReplyContext` via `canonicalizeJid`, with
+   graceful degradation to the bare jid when unknown.
+   - Read first: `src/adapters/whatsapp.ts:60-120` + the canonicalize helpers
+   - Verify: unit test — a known LID resolves to alias-store + `display_name`;
+     an unknown LID degrades to the bare jid without throwing.
+
+Step 1 must precede all others (they write/read the columns).
+
+---
+
+## Non-Negotiable Rules
+
+- Migration is idempotent and non-destructive; run it against a **copy** of the
+  live DB before deploying (the baduk incident is the cautionary tale).
+- Identity must come from the DB (`space_roles`), never inferred by the model.
+- No behaviour change for DMs beyond the added attributes.
+
+---
+
+## Edge Cases & Risks
+
+| Scenario | Handling |
+|----------|---------|
+| Pre-migration rows (NULL author) | Render as today — no name attribute, no crash |
+| Sender not in `space_roles` (new member, no push name yet) | Store canonical id with NULL name; render id only; write-through fills the name on their next named message |
+| LID that can't be canonicalized | `buildReplyContext` degrades to bare jid (current behaviour), never blocks the message |
+| Author name changes over time | `author_name` is name-at-write-time by design; live lookups use `space_roles` |
+| Migration meets the 0.9.0 migration on first start | Ordered after it in `db.ts`; tested together on the DB copy |
+
+---
+
+## Implementation Checklist
+
+### Phase 1 — Goal
+- [x] Goal, User Stories, and MVP Scope written and reviewed
+
+### Phase 2 — Architecture
+- [x] Architecture & Data section complete
+- [x] Non-Negotiable Rules defined
+- [x] Edge Cases covered
+- [x] Context for Claude pointers filled in
+
+### Implementation (commit `a7279e8`)
+- [x] Migration + `addMessage` (step 1)
+- [x] Runtime write sites (step 2)
+- [x] History rendering (step 3)
+- [x] `pushNames` persistence (step 4)
+- [x] Async LID resolution (step 5)
+- [x] 17 unit tests green; `bun run check` passes (1350 pass / 0 fail)
+- [x] Migration verified against a **copy of the live DB** — existing rows load
+      with NULL authors, new rows attribute, reopen is idempotent
+- [x] Deployed; live write path confirmed (`author_id` set on a new user row)
+- [ ] **Manual (needs a phone):** two people talk in a group, a third asks
+      "who said X?" → correct attribution
+- [ ] **Manual (needs a phone):** restart, quote-reply an old message →
+      `<reply_to name=…>` shows a name, not a raw LID
+- [x] No secrets or `.env` files committed
+- [x] No unrelated files modified
+
+---
+
+## Open Questions
+
+- [x] `addMessage` signature — settled as an optional trailing `author?:
+  MessageAuthor` param. Only 5 production call sites; matches the existing
+  optional-trailing-param style; every test call stays valid unchanged.
+
+---
+
+## Retrospective
+
+**Residual risks / follow-ups:**
+- Assistant rows deliberately carry no author. If the bot ever gains multiple
+  personas, that becomes wrong — accepted.
+- `findDisplayName` takes the most recently updated name across spaces. One
+  person with different nicknames in different groups gets the newest, not the
+  per-space one — accepted; identity (the id) is still exact.
+
+**What changed from the plan:**
+- The plan said to seed `pushNames` from the DB at startup. A lazy lookup is
+  better and is what shipped: `upsertMember` already persists names on every
+  message, so the DB is always fresher than a boot-time snapshot, and there is
+  no scan to run or cache to invalidate.
+- The async LID fix needed **no signature changes**. `canonicalizeJid` learns
+  the pair into the alias store as a side effect, so awaiting it before the
+  existing synchronous reply-context builder is enough — the sync path then
+  finds the mapping. This avoided making `buildReplyContext` async and
+  rippling through the sync `parseMessage` interface.
+
+**Key decisions made during implementation:**
+- Closed two prompt-injection holes found while editing these lines: ambient
+  content was interpolated into `<message>` unescaped, and `<reply_to>`
+  attributes (push names are user-typed) were unescaped, so a double quote
+  could close the attribute and inject another. Both are in scope by the
+  spec's own "identity stays hard" rule — a sender must not be able to forge
+  identity markup.
+- `MESSAGE_COLUMNS` extracted as a shared constant; the five message SELECTs
+  had drifted into four different hand-maintained column lists.
+
+**Architecture impact** (update `docs/ARCHITECTURE.md` if any of these apply):
+- [ ] New data model or schema change (`messages.author_id`, `author_name`)