mercury-agent 0.16.0 → 0.16.2

package/docs/goals/whatsapp-bot-hardening/open-threads.md

@@ -0,0 +1,396 @@
+# Open threads — Mercury WhatsApp Bot
+
+**Goal**: [whatsapp-bot-hardening](goal.md)
+**Opened**: 2026-08-10
+
+Everything still outstanding, in one place. The eight hardening stories are
+shipped (see [roadmap](roadmap.md)); this is what did *not* fit inside them,
+plus what the prompt-layer investigation turned up.
+
+Each item says what it is, why it matters, and what "done" looks like — so any
+of them can be picked up cold.
+
+---
+
+## A. Prompt layers & voice
+
+### A-1. There are **five** prompt layers, not three
+
+This was the root of the confusion about `system_prompt` vs preferences. Full
+map, verified against the running code:
+
+| # | Layer | Where it lands | Scope | Setter |
+|---|-------|----------------|-------|--------|
+| 1 | `.mercury/global/AGENTS.md` | pi auto-loads from the resource dir (`storage/memory.ts:16`) | global | edit the file |
+| 2 | `project_config.character` | system prompt, as `## Bot Character` via `MERCURY_EXT_SYSTEM_PROMPT` (`runtime.ts:1601`) | global | dashboard, `mrctl character set` |
+| 3 | `space_config.system_prompt` | system prompt (`runtime.ts:1614`) | per-space | **none — see A-2** |
+| 4 | `.mercury/spaces/<id>/AGENTS.md` | pi auto-loads from the space workspace | per-space | edit the file |
+| 5 | `space_preferences` | **user** prompt, as `<preferences>` XML (`preferences-prompt.ts`, `container-entry.ts:783`) | per-space | dashboard, `mrctl prefs set` |
+
+Current occupancy: layer 1 is the stock template. Layer 3 is empty everywhere.
+Layer 4 is written for `main` and `admin`, **empty for `19` and `greece2026`** —
+which is why those two spaces run almost entirely on layers 2 and 5.
+
+### A-2. `space_config.system_prompt` has no setter
+
+Its only writer is `conversation.ts:177` (`seedSpaceConfigIfAbsent`, from
+`autoSpace.defaultSystemPrompt`). Nothing in chat, `mrctl`, or the dashboard
+writes it. So it is a real prompt layer that no user can reach.
+
+**Decide:** add a setter (`mrctl config set system_prompt`), or drop the layer.
+Leaning: drop it. Layer 4 (`AGENTS.md`) already covers per-space long-form
+instructions and is editable, so a setter would add a fifth way to say the same
+thing. Either way, stop treating it as a live option.
+
+### A-3. Layer precedence is **not** deterministic — do not design around it
+
+Measured on the throwaway `prompttest` space over `POST /chat`, using target
+language as the probe (each layer told to answer in a different language):
+
+- A direct `system_prompt` vs `preferences` conflict went 8/10 to
+  `system_prompt` — but **not 10/10**, and a swapped-language control produced
+  the opposite result. Repeating one fixed config gave different winners on
+  different runs.
+- The global character contains its own conflict ("answer in the language they
+  wrote in" **and** "default Hebrew"); an English probe resolves it
+  inconsistently.
+
+**Conclusion:** you cannot rely on a lower layer overriding a higher one. The
+only reliable fix is to **not state the same rule in two layers**. Recorded as
+D-014.
+
+### A-4. Vague quantities produce vague behaviour — the emoji data
+
+Live emoji rate in assistant messages, by space:
+
+| Space | Instruction | Emoji rate |
+|-------|-------------|-----------|
+| `19` (baduk) | pref: "בלי אימוג'ים בכלל" (none at all) | **0 / 40 = 0%** |
+| `greece2026` | pref: "minimal emojis" | 33% |
+| `main` | character only: "אימוג'י פה ושם כשזה מתאים" (here and there when it fits) | 73% |
+
+Preferences demonstrably work — `19` is at a hard zero. What fails is the
+*wording*: "minimal" and "here and there" are not countable, so the model picks
+its own number. A countable rule is obeyed; an adjective is not.
+
+### A-5. Emoji rule and layer cleanup — **applied 2026-08-10**
+
+Owner's decision: *at most one emoji per ~5 lines, only if it carries meaning,
+default 0, in every space* — no exception for `19` even though its absolute ban
+had it at a clean 0/40. Applied together with the layer cleanup below; DB backup
+at `~/mercury-backup-prompts-20260810/`.
+
+| Layer | Change |
+|---|---|
+| `character` | Rewritten **in English**. Five clauses: persona, language, length, emoji, honesty. Countable emoji rule; anti-fabrication **and** stale-facts lines promoted here (both existed only in `main`/`admin` AGENTS.md, i.e. in neither real group); **"default Hebrew" removed** — matching the message is the only global language rule. |
+| `19` prefs | `tone` reworded (emoji ban dropped, now global; "one of the guys" → group-neutral); `language` = **always Hebrew**, everyone in the group speaks it |
+| `greece2026` prefs | `language.style` **deleted** — it bundled a hard constraint, a tone hint, and a vague "minimal emojis" that would have *loosened* the stricter global rule. Now `language` (English by default, because members share no other common language, **with an explicit-request escape hatch**) and `tone` (family + partners + friends, not just friends). `response.length` **deleted** — length is a global rule now. |
+| `main/AGENTS.md` | Duplicated language, brevity, warmth, anti-fabrication and stale-facts removed; keeps only what is space-specific |
+| `admin/AGENTS.md` | Explicit "this space overrides the global character" banner; Hebrew default removed |
+
+Both per-space language rules now **contradict** the global "match the message"
+rule outright, so each states the override in words. `greece2026` is two
+overrides deep — space rule beats global, explicit user request beats space
+rule — and that holds (P3 below).
+
+Preferences and character are read per turn (`runtime.ts:1601`, `:1799`), so no
+restart was needed.
+
+**Overrides are now stated in words, not implied by position.** `greece2026`'s
+language pref literally says it overrides the global rule, and `admin`'s
+`AGENTS.md` says it overrides the character. Per A-3 an implicit override is a
+coin flip; an explicit one is just an instruction.
+
+### A-6. Verification — measured on fresh spaces, 2026-08-10
+
+One fresh space per probe, no shared history.
+
+| Probe | Result |
+|---|---|
+| Hebrew in, character only | Hebrew out **4/4** |
+| English in, clean space | English out **3/3** (100% Latin) |
+| Hebrew conversation → English message | switches to English |
+| `19` prefs, **English** in | Hebrew out **3/3** — unconditional override beats the incoming message |
+| `greece2026` prefs, Hebrew in | English out **4/4** |
+| `greece2026` prefs + "answer me in Hebrew please" | Hebrew out **3/3** — the escape hatch beats the space rule |
+| Long question (4-day itinerary, 8 people) | 25 and 10 lines, complete, not truncated |
+| Short question (time zone) | **1 line** both times — the length rule did not become "always long" |
+
+Dropping `19`'s absolute emoji ban did **not** cause a spike.
+
+**Emoji: one leak found and mostly closed.** The first pass held on *count* but
+leaked on *placement* — `19` opened 2 of 3 replies with `מה קורה 👋`. One emoji
+in a 9-line reply is inside the "~1 per 5 lines" budget, so the number was never
+the problem: "never to open or close a message" was too abstract to bind against
+a tone pref pulling toward banter. Naming the exact case ("a wave, a smile or a
+thumbs-up attached to a greeting or a sign-off is decoration, however casual the
+conversation is") took it to **1 emoji over 12 lines, 1 of 4 replies** on
+deliberately greeting-shaped prompts. Same lesson as "minimal" (33%) vs "none at
+all" (0%): concrete beats abstract.
+
+Residual, left as a taste call: a bare thank-you (`אחלה, תודה!`) still gets
+`בכיף! 🙂`. One line, one emoji — over budget and carrying nothing. Closing it
+needs one more sentence ("a message of one or two lines gets no emoji at all"),
+at some risk of reading cold in a friends' group.
+
+> **Method note, worth keeping.** The first run appeared to show a regression:
+> an English prompt came back in Hebrew, 0/2. It was the *test* that was broken —
+> the English probe ran in `prompttest` straight after four Hebrew turns in the
+> same space, so the model was matching the conversation, not the message. Re-run
+> in virgin spaces it passes 3/3. The detector was also too crude (it scored any
+> reply containing a single Hebrew character as Hebrew, so English prose with a
+> Hebrew greeting counted as a failure); it now scores by majority script.
+> **One-space, sequential prompt probes are not trustworthy — use a fresh space
+> per probe.**
+
+### A-6b. Left open after the cleanup
+
+- **The `בכיף! 🙂` case** above — one sentence closes it, at some cost to warmth.
+  Owner's call.
+- **Re-measure the live emoji rate** after a week of real traffic. Everything
+  above is API probes, not `main` and `19` in daily use. Count
+  `\p{Extended_Pictographic}` over assistant rows since 2026-08-10 and compare
+  against the pre-change baseline (`main` 73%, `greece2026` 33%, `19` 0%).
+- **`admin` is now the only space whose voice is set by `AGENTS.md` rather than
+  preferences.** That is deliberate — it is long-form and structural — but it
+  means the one space with a genuinely different voice is also the one with no
+  dashboard-visible setting. Worth remembering before wondering why `admin`
+  reads differently.
+
+### A-7. The agent is never told preferences exist — **fixed 2026-08-10**
+
+`container-entry.ts` never mentioned `<preferences>` or `mrctl prefs set`, so
+the self-service flow the design implies was never explained to the model. It
+also carried an actively wrong instruction at `container-entry.ts:326`:
+per-space tone lives in `system_prompt` "set from the dashboard" — no such
+setter exists (A-2), and tone belongs in preferences.
+
+Both halves are now fixed in the `## Character` section: the `system_prompt`
+sentence is replaced with the four real `mrctl prefs` verbs and when to reach
+for them, plus a closing line carrying D-014 — do not state the same rule in
+two layers, and make an intended override explicit in words rather than relying
+on specificity. Now a PR item to *describe*, not to do.
+
+**Live since 2026-08-10.** Prompt-side, so it needed the image rebuild, not just
+a service restart: base rebuilt, derived tag moved `dd9a11d5 → 0597b302`,
+verified by grepping the new image rather than the repo. The rebuild also
+disproved the roadmap's claim that the derived image ignores a base change —
+corrected there.
+
+---
+
+## B. API exposure — remaining hardening
+
+Closed the acute part on 2026-08-10 (both secrets set; see roadmap). Left over:
+
+- **B-1. Done 2026-08-10.** `POST /chat` now requires
+  `content-type: application/json` and returns 415 otherwise
+  (`routes/chat.ts`). The check runs **before** the API-key check, so it holds
+  in the configuration it actually protects — the one with no key set. Killing
+  the browser drive-by is the point: a `text/plain` body makes a cross-origin
+  POST a CORS "simple request" with no preflight, so any open web page could
+  start a real agent turn. Demanding JSON forces a preflight this server never
+  answers. Covered by `tests/chat-content-type.test.ts`; the only in-repo
+  caller (`mercury chat`, `cli/mercury.ts:2067`) already sent the header.
+- **B-2.** Add a bind-host config option. `main.ts:128` calls `Bun.serve` with
+  no `hostname`, so it listens on `*:8787`. ⚠️ **Do not fix by hardcoding
+  `127.0.0.1`** — agent containers reach the host via
+  `host.docker.internal:host-gateway`, so loopback-only binding breaks every
+  turn. It needs to be configurable, defaulting to the docker-bridge address.
+- **B-3.** Bare `GET /dashboard` is registered at `server.ts:135`, *before* the
+  `/dashboard/*` auth middleware at `server.ts:191`, and Hono's `/dashboard/*`
+  does not match the bare path. With auth on, it serves the SPA shell whose data
+  calls then 401 — an empty page with no error and no login prompt. It should
+  redirect to the login page. (Version disclosure is the lesser half.)
+- **B-4.** `.env` is cwd-bound: hardcoded `join(CWD, ".env")`
+  (`cli/mercury.ts:112`), no override. CLI commands needing API keys must run
+  from `~/whatsapp-bot`. Not worth patching upstream on its own; note it if
+  another `.env` issue comes up.
+
+---
+
+## C. The PR
+
+**Opened 2026-08-10.** Push access arrived; **18 commits** on
+`fix/whatsapp-context-identity-and-docs`, rebased onto `origin/main`. Always
+count with `git rev-list origin/main..HEAD` after a `git fetch` — a stale local
+`main` gave a misleading 16 for most of this goal.
+
+All ops docs live in exactly one commit (`docs(ops): add whatsapp-bot-hardening
+goal, roadmap and story archive`) and **no code commit touches `docs/`**, so if
+the maintainer would rather not carry planning docs upstream that single commit
+can be dropped without disturbing anything else.
+
+Body covers the six points in the roadmap's Completion section, **plus** these
+accumulated since: B-2, B-3, and the ambient prompt-injection fix below. B-1 and
+A-7 are **done** (2026-08-10), so they are changes the PR describes rather than
+work it still owes.
+
+**New behaviour users will feel** (add to the roadmap's list of six): tagging
+the bot now always triggers it, whatever `trigger.patterns` says — see §G. New
+space config key `trigger.mention_always` to turn that off.
+
+**Also worth its own section:** ambient message content used to be interpolated
+into the prompt raw (`container-entry.ts`, `${m.content}`); it is now run
+through `escapeXmlText` like every other block, and reply-context attributes go
+through `escapeAttr`. Any group member could previously write XML straight into
+the prompt. That is a security fix and is not currently listed anywhere in the
+PR plan.
+
+---
+
+## D. Operations
+
+- **D-1. WSL does not auto-start.** Carried over from the setup plan. Lingering
+  (`loginctl enable-linger`) survives closing a terminal, but not
+  `wsl --shutdown` or a Windows reboot — the bot is down until something touches
+  the distro. Fix: a Task Scheduler task at logon running
+  `wsl.exe -d Ubuntu -u guysaha -e /bin/true`.
+- **D-2. Tighten `allowedDomains` from `"*"`** (D-011). The injection guard is
+  verified still on, so this is hardening, not a live hole. Deferred out of the
+  PR by decision; revisit after it merges.
+- **D-3. `prompttest` deleted 2026-08-10** (64 messages, 4 roles, 32 token-usage
+  rows — all from prompt experiments). Live spaces are now exactly `19`, `admin`,
+  `greece2026`, `main`.
+- **D-4. Orphaned workspace directories — cleaned 2026-08-10.** The prompt tests
+  left 23 dirs in `.mercury/spaces/` (`prompttest`, `langtest-*`, `v2-*`,
+  `v3-*`); removed by hand, leaving exactly `19`, `admin`, `greece2026`, `main`.
+  **Not an upstream bug** — `db.deleteSpace()` only clears rows, and every real
+  delete path pairs it with `removeSpaceWorkspace` (`routes/spaces.ts:87`,
+  `routes/console.ts:555`, `cli/mercury.ts:2362,2380`). The test scripts called
+  the DB layer directly and skipped that half.
+  **Rule for any future throwaway-space harness: go through the API, or call
+  `removeSpaceWorkspace` alongside `deleteSpace`.**
+- **D-5. `agent.env_passthrough` tightened to `claimed` — 2026-08-10.** Startup
+  was logging that the blind `MERCURY_*` passthrough carried `MERCURY_AGENT_IMAGE`
+  into every space's container, scoped to nothing. `claimed` narrows it: model
+  provider credentials stay exempt (otherwise the agent could not reach a model)
+  and extension-declared vars are injected separately behind their permission
+  gate, so the only var actually withdrawn is `AGENT_IMAGE` — host-side only,
+  and nothing in-container reads it (grepped across `src/`, `container/`,
+  `examples/`, `resources/`, `docker/`). Startup now logs the `claimed` line.
+  Backup: `mercury.yaml.bak-envpassthrough-20260810`.
+  **The value of the setting is the audit, not the one var** — it converts every
+  future `.env` addition from silently container-wide into a deliberate
+  declaration.
+- **D-6. Machine audit against the setup guide — 2026-08-10, clean.** Ubuntu
+  26.04, bun 1.3.14, node v24.19.0, docker 29.1.3 (enabled + active, user in
+  the `docker` group), git, curl. `mercury` 0.9.0 on PATH; `mercury doctor`
+  reports 8 passed / 1 warning / 0 errors, the warning being "port 8787 in use",
+  which is Mercury itself. Lingering is on. Derived image carries pandoc 3.1.3,
+  python `docx`/`openpyxl`/`odf`, and `mrctl`. `pandoc` is **absent from the
+  host** and that is correct — the docs extension needs it in the container.
+  The unit at `~/.config/systemd/user/mercury.service` matches what
+  `generateSystemdService()` emits, field for field, with one **deliberate**
+  documented deviation: `Restart=always` instead of the generated
+  `Restart=on-failure`, re-applied by hand because `service install` reverts it
+  (see `archive/run-from-source-switchover.md` step 5). Anyone re-running
+  `mercury service install` must re-apply it again.
+
+---
+
+## E. Phone-dependent manual checks — never run
+
+Every one of these needs the owner's phone; none block the PR, but the
+milestone checkpoints are not truly closed until they pass. Full detail lives in
+the per-story specs in `docs/goals/whatsapp-bot-hardening/archive/`.
+
+| # | Check | From |
+|---|-------|------|
+| E-1 | 3 untagged messages in a test group → exactly 3 new `role='ambient'` rows, with authors | M2 |
+| E-2 | Two people talk, a third asks "who said X?" → correct attribution | M2 |
+| E-3 | Restart the service, then quote-reply an old message → `<reply_to>` shows a **name**, not a raw LID | M2 |
+| E-4 | In a **throwaway** space: `/spaces delete <other-id>` → **reply** "yes" → named space dies, current one survives (the baduk regression) | M3 |
+| E-5 | Bare `/spaces list`, untagged, in a group | M3 |
+| E-6 | Send a ~50 MB docx → bot reads it. Send ~200 MB → bot says *why* it can't | M4 |
+| E-7 | "Make me a doc with X" → `.docx` arrives and opens in Word, triggered by a **member**, not the admin | M4 |
+| E-8 | Unlink a throwaway group, then re-link it entirely from chat | M3 |
+
+⚠️ E-1 to E-3 want `log_level: debug` for the test window only — it writes
+chat-content previews into journald, i.e. real family/friends chat into system
+logs. **Set it back to `info` afterwards.**
+
+---
+
+## F. Scheduled tasks — the 2026-08-10 football failure
+
+Filed in full at
+[`docs/debug/major/2026-08-11-scheduled-task-timeout-silent.md`](../../debug/major/2026-08-11-scheduled-task-timeout-silent.md)
+— fixed and archived. Deferred by decision on 2026-08-10; recorded here so it is
+not lost. The runtime half it spun off is
+[`scheduled-research-task-runtime-drift`](../../debug/moderate/2026-08-19-scheduled-research-task-runtime-drift.md).
+
+The daily football update never reached the group. The scheduler fired on time;
+the container ran the full 20-minute timeout and was killed. Two defects:
+
+- **F-1 (generic, affects every scheduled task).** A task whose handler throws is
+  reported nowhere — the scheduler logs and swallows, and `runtime.ts` only
+  sends on success. Worse, `next_run_at` is advanced *before* running, so one
+  failure costs a whole cron period with no retry. Same "no silent drops"
+  principle M4.1 applied to media, not yet applied to tasks.
+
+  ⚠️ **Re-verified against `origin/main` 9e78a68** ("stop dashboard Run now from
+  firing a second agent run"), which rewrote ~156 lines of
+  `task-scheduler.ts` after this was first written. **Both halves survive**, at
+  new locations: the pre-advance is now `runTask`, `task-scheduler.ts:155-158`,
+  and its doc comment states it as deliberate ("Both happen regardless of
+  handler success"); the swallow is `task-scheduler.ts:168-174`. So F-1 is a
+  design decision to argue with, not an oversight to patch quietly — and any
+  fix now has to keep 9e78a68's in-flight/`applyBookkeeping` contract intact.
+- **F-2 (this task).** The prompt asks for 6–8 improvised web searches daily, so
+  its runtime is unbounded: 5.9 min on Aug 9, 20.0 min (timeout) on Aug 10, with
+  a **byte-identical** prompt. Wants a faster, more deterministic pipeline —
+  possibly a skill or script — that keeps the freshness and the named-source
+  precision. Investigate before designing.
+
+First step for either is capturing a pi trace of one run: the container is
+`--rm`, so today's 20 minutes left no evidence of where the time went.
+
+---
+
+## G. Tagging the bot by number — **fixed 2026-08-10**
+
+**The bug.** In `greece2026`, tagging the bot's number did nothing. The chain:
+the adapter rewrites a tag into `@<botUsername>` (`@ויציבוט` here); the space
+sets `trigger.patterns = "Wiz, @Wiz"`; and `trigger.ts` *replaces* the default
+pattern list when a space defines its own, discarding the `@<botUsername>` that
+`router.ts` appends to the defaults. So the rewritten name matched nothing and
+a correct tag was filed as ambient.
+
+**Caught in the live DB:** `messages#1181`, greece2026, 2026-08-10T07:47:44Z,
+from Sharon — `@ויציבוט : I am very fond of cheeses, what can I expect and
+explore on our holiday?` — stored `role='ambient'`, never answered.
+
+**`main` was broken the same way** and nobody had noticed: its patterns are all
+bare (`ויציבוט`, no `@` form), and the mention regex requires start-of-string or
+whitespace before the pattern, so a bare pattern never matches `@`-prefixed
+text. Only space `19` worked, and only because `@ויציבוט` had been added to its
+list by hand.
+
+**The fix** stops re-deriving "was I addressed" from text. The adapter already
+knows from the mention's JID (`isBotJid`), so it now sets `isBotMentioned` and
+that flag rides the IngressMessage to both trigger gates (`router.ts`,
+`handler.ts` — they each have one). The text rewrite stays, but only as
+cosmetics for stripping and readability; it is allowed to find nothing.
+
+- **Off switch:** space config `trigger.mention_always` (default `true`), set
+  per space. `mrctl config set trigger.mention_always false` works as soon as
+  the key is in `BUILTIN_CONFIG_KEYS` — but the **dashboard row is not free**.
+  `routes/dashboard.ts` builds the "Triggers & ambient" rows from a hardcoded
+  list, so the control had to be added there by hand (it reads through
+  `mentionAlwaysEnabled()`, the same helper the gates use, so the two cannot
+  drift).
+- **Also fixed in passing:** `mergeIngresses` dropped `droppedAttachments`, so
+  the M4.1 media-drop note vanished whenever two messages merged inside the
+  debounce window — the exact silence that fix existed to remove.
+- Cover: `tests/mention-always-triggers.test.ts` (13 tests), including the live
+  failure and the case where the rewrite finds nothing.
+
+### G-1. Still open: Discord has the same latent bug
+
+`discord-native.ts:430` already computes a structured `isMention` from
+`msg.mentions.users`, then throws it away and relies on the same text rewrite.
+Only the WhatsApp bridge sets `isBotMentioned`. A Discord space with custom
+trigger patterns would fail exactly as `greece2026` did. Not wired up because
+this branch is WhatsApp-scoped — the flag and both gates are already generic, so
+it is a two-line bridge change whenever Discord matters.

package/docs/goals/whatsapp-bot-hardening/roadmap.md

@@ -0,0 +1,245 @@
+# Roadmap: Mercury WhatsApp Bot — Context, Identity & Command Hardening
+
+**Goal**: [whatsapp-bot-hardening](goal.md)
+**Last updated**: 2026-08-10
+**Still open**: [open-threads.md](open-threads.md)
+
+**All four milestones are done.** What remains is the PR (parked on push access)
+and the phone-dependent manual checks — both tracked in
+[open-threads.md](open-threads.md), which is now the live to-do list. This file
+is the record of what shipped.
+
+Ordering follows the handoff's §7 (archived at
+`docs/goals/whatsapp-bot-hardening/archive/handoff-2026-08-09.md`): context/identity
+first (highest impact,
+shared code path), then the three defects that destroy data or lock you out,
+then the rest of the command surface, then capabilities, then member memory.
+The switchover is Milestone 1 because nothing can be tested live before it.
+
+## Milestones
+
+### Milestone 1: Run from source
+> The live bot serves from the git checkout; fixes become edit→restart testable.
+
+| ID | Story | Slug | Depends on | Status |
+|----|-------|------|------------|--------|
+| M1.1 | Deployment switchover + commit the `.dockerignore` build fix | run-from-source-switchover | — | **done** |
+
+**Checkpoint:** DB backed up (all three files: `state.db`, `-wal`, `-shm`);
+`bun run check` green; service restarted and answering a WhatsApp DM;
+`SELECT model, provider FROM token_usage ORDER BY id DESC LIMIT 1` confirms the
+expected model; the 0.9.0 db.ts migration ran cleanly against the existing DB;
+rollback path (`npm unlink` + old unit) verified to exist but not exercised.
+
+### Milestone 2: Context & identity (Bugs 1 + 2)
+> The bot sees untagged group messages and knows who said what — with the
+> context budget capped so ambient can't starve real turns.
+
+| ID | Story | Slug | Depends on | Status |
+|----|-------|------|------------|--------|
+| M2.1 | Message author attribution (schema, rendering, pushNames persistence, LID resolution) | message-author-attribution | M1.1 | **done** (`a7279e8`) |
+| M2.2 | Ambient group context (recordAmbient, bounded query, retention TTL) | ambient-group-context | M2.1 | **done** (`4bf5505`) |
+
+M2.1 lands first so that when ambient starts flowing it is attributed from day
+one — ambient without attribution produces a wall of anonymous text that makes
+misidentification *worse* (HANDOFF §7.1).
+
+**Checkpoint:** with `log_level: debug` for the test window only — send 3
+untagged messages in a test group → exactly 3 new `role='ambient'` rows with
+authors; tag the bot and ask what was just said; two people talk, a third asks
+"who said X?" → correct attribution; restart the service, quote-reply an old
+message → `<reply_to>` shows a name, not a raw LID. Then set logging back to
+`info` and confirm ambient rows for `greece2026` keep growing:
+`SELECT space_id, role, COUNT(*) FROM messages GROUP BY space_id, role`.
+
+### Milestone 3: Command surface safety
+> No command can silently destroy data, lock a group out, or fail to stop a
+> runaway agent; chat and CLI gates agree.
+
+| ID | Story | Slug | Depends on | Status |
+|----|-------|------|------------|--------|
+| M3.1 | Destructive command guards — mrctl delete argument, confirmation matching, unlink lockout, /stop routing (Bugs 5.1, 6a, 6c) | destructive-command-guards | M1.1 | **done** (`c44720f`) |
+| M3.2 | Command routing consistency — untagged slash commands in groups, permission-gate alignment, /resume + /help pause, redundancy cleanup (Bugs 5 rest, 6b, 6d, 6e) | command-routing-consistency | M3.1 | **done** (`4f05fbb`) |
+
+**Checkpoint:** in a **throwaway** space/group only — `/spaces delete <other-id>`
+then **reply** "yes" → the named space dies, current one survives (the baduk
+regression test); `mrctl spaces delete foo` refuses with the explanatory error;
+`/stop` during a running turn aborts it; unlink a throwaway group then re-link
+it entirely from chat; bare `/spaces list` works untagged in a group.
+
+### Milestone 4: Capabilities & member memory
+> Large files are readable, failures are explained, the bot can produce real
+> documents, and it accumulates per-member knowledge.
+
+| ID | Story | Slug | Depends on | Status |
+|----|-------|------|------------|--------|
+| M4.1 | Media size cap + no silent drops (Bug 3) | media-size-and-silent-drop | M1.1 | **done** (`5734c04`) |
+| M4.2 | `docs` extension — real .docx creation (Bug 4) | docs-extension | M1.1 | **done** (`e1208b8`) |
+| M4.3 | Member memory — hard identity + soft notes | member-memory | M2.1 | **done** (`44a7542`) |
+
+**Checkpoint:** send a ~50 MB docx → bot reads it; send a ~200 MB file → bot
+says *why* it can't; "make me a doc with X" → a `.docx` arrives in WhatsApp and
+opens in Word, triggered by a member (not just admin); after some group chat,
+ask the bot what it knows about a member → answer draws on
+`knowledge/members/<id>.md` with correct identity.
+
+## Dependency Graph
+
+```
+M1.1 run-from-source-switchover
+ ├─→ M2.1 message-author-attribution ─→ M2.2 ambient-group-context
+ │                                   └─→ M4.3 member-memory
+ ├─→ M3.1 destructive-command-guards ─→ M3.2 command-routing-consistency
+ ├─→ M4.1 media-size-and-silent-drop
+ └─→ M4.2 docs-extension
+```
+
+M3.x is independent of M2.x and can be built in parallel, but M3.1 should not
+wait past M2 — it contains the fixes for the defects that already destroyed a
+space once.
+
+## Deploying prompt changes — read this before believing a test
+
+**`src/agent/container-entry.ts` runs inside the agent container and is baked
+into the image** (`container/Dockerfile:107`). Restarting the service does not
+pick up edits to it. Everything host-side (ambient recording, author storage,
+command guards, media notes) goes live on `systemctl --user restart mercury`;
+everything prompt-side (author names in history, ambient rendering,
+`<members>`, member notes) needs the image rebuilt.
+
+The derived image's cache key is
+`sha256(baseImage + "\n" + baseImageId + "\n" + installCommands)`
+(`extensions/image-builder.ts:313-314`). It **does** include the base image ID,
+so `bash ./container/build.sh` alone is enough to change the derived tag and
+force a rebuild — measured 2026-08-10, tag moved `dd9a11d5 → 0597b302` with the
+extension list untouched.
+
+The caveat is `resolveBaseImageId` (`image-builder.ts:327`): it shells out to
+`docker image inspect` and returns `null` on failure. A null base ID drops out
+of the key and the stale-cache trap comes back, silently. So `docker rmi` is
+belt-and-braces rather than mandatory — cheap enough to keep in the procedure.
+
+Full procedure after touching `container-entry.ts`:
+
+```bash
+cd ~/mercury-src && bash ./container/build.sh     # rebuild mercury-agent:latest
+docker rmi mercury-agent-ext:<current-hash>       # optional; forces it regardless
+systemctl --user restart mercury                  # ~4 min rebuild, then live
+docker run --rm --entrypoint sh mercury-agent-ext:<new-hash> \
+  -c 'grep -c members /app/src/agent/container-entry.ts'   # verify, do not assume
+```
+
+Then confirm the service actually resolved to the new image: its
+`Built derived agent image <tag>` / `Using cached agent image <tag>` log line
+must show the **new** hash, not the one you deleted.
+
+This was found by asking the bot to list its `<members>` block and getting an
+honest "there is no members block" — the tests were green and the service had
+been restarted. Verify prompt changes against the running bot, not the repo.
+
+Note also that `sync-mercury.sh` only warns you to rebuild when `container/` or
+`docker/` changed. `src/agent/container-entry.ts` is baked in too and is **not**
+in that check, so a prompt-only change syncs and restarts with no warning at
+all.
+
+## Completion — the PR (after Milestone 4)
+
+Not a story; the goal's exit gate (archived handoff §5). One PR from
+`fix/whatsapp-context-identity-and-docs` → `main`, one section per change,
+explicitly covering:
+
+1. The four pinchtab commits as **upstreaming an already-live hotfix**, with
+   the `allowedDomains "*"` trade-off stated plainly (D-011).
+2. The `.dockerignore` fix (exists in no commit anywhere until M1.1).
+3. The `messages` schema migration and its NULL-backfill behaviour.
+4. Ambient volume/retention implications.
+5. The baduk incident as motivation for the regression tests.
+6. **Behaviour changes users will feel:** `/stop` now aborts instead of
+   prompting; slash commands work untagged in groups; `/spaces unlink` removed
+   from chat; `/model active` dropped.
+
+### Pre-PR housekeeping — **done** (2026-08-10)
+
+| Item | Outcome |
+|------|---------|
+| D:-clone's `fix/pinchtab-browser-startup` | **Superseded, not pushed** (D-013). Combined patch byte-identical to `bfe5a33..b12d7c0` in `mercury-src`; clone's base is 3 commits behind `origin/main`. Branch left in place. |
+| Stray `~/.mercury/state.db` | **Deleted.** Verified schema-only first: 0 messages, 0 spaces, 0 space_roles. Live DB is `~/whatsapp-bot/.mercury/state.db` (1020 messages, 4 spaces, 27 roles) and was untouched. |
+| Debug logging | **Already `info`.** `mercury.yaml:45`; no `MERCURY_LOG_LEVEL` in `.env` (only the three API keys and `MERCURY_AGENT_IMAGE`), so no env override. Debug was never enabled during implementation. |
+
+### The cwd trap — root-caused and closed (2026-08-10)
+
+The stray DB existed because `data_dir: .mercury` is **relative**, resolved
+against `process.cwd()` by `resolveProjectPath` (`config.ts:412`). The service
+was never at risk (`WorkingDirectory=/home/guysaha/whatsapp-bot`), but running
+`mercury` by hand from `$HOME` created a fresh empty `~/.mercury/state.db` and
+used *that* — which is why `mercury auth status` reported "not logged in".
+
+Making `data_dir` absolute is **not sufficient on its own**: `mercury.yaml` is
+itself located relative to cwd (`config-file.ts:315`), so from `$HOME` no config
+is found and the built-in `.mercury` default applies regardless. Both halves
+were required:
+
+1. `runtime.data_dir: /home/guysaha/whatsapp-bot/.mercury` in `mercury.yaml`
+2. `export MERCURY_CONFIG_FILE=/home/guysaha/whatsapp-bot/mercury.yaml` in
+   `~/.profile` — deliberately **not** `~/.bashrc`, which returns early for
+   non-interactive shells (`case $- in *i*) ;; *) return;;`) and so would never
+   apply to `wsl -e bash -lc`.
+
+Verified by loading the real `loadConfig()` from both directories, with a
+negative control: unsetting `MERCURY_CONFIG_FILE` makes `$HOME` fall back to
+`/home/guysaha/.mercury`, proving the mechanism rather than assuming it. After
+the restart, `mercury auth status` from `$HOME` reports "✓ logged in", no
+`~/.mercury` is recreated, and the live DB is intact.
+
+**Residual limit:** `.env` is still cwd-bound — hardcoded `join(CWD, ".env")`
+(`cli/mercury.ts:112`), no override exists. CLI commands that need API keys must
+still run from `~/whatsapp-bot`; `mercury auth status` elsewhere will say "No API
+keys found in .env". Not worth patching upstream for this alone.
+
+**`mrctl` is not a host CLI.** It is the in-container agent tool, reading
+`API_URL`/`CALLER_ID`/`SPACE_ID` from the environment and exiting
+"API_URL not set" on the host. Inspect live state by querying `state.db`.
+
+Backups: `mercury.yaml.bak-datadir-20260810`, `~/.profile.bak-20260810`.
+
+### API exposure — closed 2026-08-10
+
+Found while building a phone-free test harness. `Bun.serve` is called with no
+`hostname` (`main.ts:128`), so the server listens on `*:8787`, and **both auth
+secrets were unset**. Unauthenticated, that meant:
+
+- `GET /dashboard/page/*` — every dashboard data page
+- `GET /api/*` — caller identity is self-asserted via `x-mercury-caller` /
+  `x-mercury-space` headers, so any space's data was readable
+- `POST /chat` — runs a real agent turn: spends tokens, invokes tools
+
+Not LAN-reachable (WSL2 default NAT, no `.wslconfig`, no `netsh portproxy`), but
+reachable from any process on the Windows host and — because `/chat` parses a
+`text/plain` body — from **any web page open in a browser**, since that makes a
+cross-origin POST a CORS "simple request" with no preflight. The response can't
+be read, but the turn still executes.
+
+Fixed by setting `MERCURY_API_SECRET` and `MERCURY_CHAT_API_KEY` in `.env`
+(backup `.env.bak-secrets-20260810`), then restarting. Verified after: all three
+surfaces return 401, and an authenticated `/chat` turn that runs `mrctl prefs
+list` still succeeds — confirming `API_SECRET` reaches the container
+(`container-runner.ts:950`).
+
+`GET /dashboard` still returns 200 unauthenticated. That is the static SPA shell
+only (`dashboard/index.html`, no space data); its `app.get` is registered at
+`server.ts:135`, before the `/dashboard/*` auth middleware at `server.ts:191`,
+and Hono's `/dashboard/*` does not match the bare path. Version disclosure only.
+
+**Do not "fix" this by binding to `127.0.0.1`** — containers reach the API via
+`host.docker.internal:host-gateway`, so loopback-only binding breaks every turn.
+
+Remaining hardening moved to [open-threads.md](open-threads.md) §B.
+
+### Prompt layers — investigated 2026-08-10
+
+Triggered by the question "do space preferences actually change behaviour?".
+They do — `19`'s "no emojis at all" holds at 0/40 assistant messages. But the
+investigation found **five** prompt layers rather than the three assumed, one of
+which (`space_config.system_prompt`) has no setter at all, and it measured that
+**precedence between layers is not deterministic** (D-014). Full map, the emoji
+data, and the live conflicts are in [open-threads.md](open-threads.md) §A.