mercury-agent 0.5.2 → 0.5.7

package/docs/debug/major/2026-07-02-bwrap-privileged-linux-docker.md

@@ -1,160 +0,0 @@
-# Bug: bwrap fails to mount /proc on Linux Docker Engine (non-Desktop)
-
-**Status:** Fixed  
-**Severity:** P1 — blocks all agent responses on standard Linux Docker hosts  
-**Affected:** Linux servers running Docker Engine (not Docker Desktop, not gVisor)  
-**File:** `src/agent/container-runner.ts`, bwrap compat branch (~line 980)  
-**Last updated:** 2026-07-02
-
-## Symptom
-
-```
-Container error: Container failed (exit code 1): Error: pi CLI failed (1):
-bwrap: Can't mount proc on /newroot/proc: Operation not permitted
-```
-
-Happens on Linux servers (e.g. Hetzner, AWS EC2, DigitalOcean) running
-Docker Engine with the default kernel config. The existing bwrap Docker compat
-flags (`seccomp=unconfined`, `apparmor=unconfined`, `CAP_SYS_ADMIN`) are not
-sufficient for bwrap to mount `/proc` inside the container.
-
-## Root cause
-
-On Linux Docker Engine, the default seccomp + AppArmor + capability set does
-not allow `mount(2)` inside a user namespace, even with:
-- `--security-opt seccomp=unconfined`
-- `--security-opt apparmor=unconfined`
-- `--cap-add SYS_ADMIN`
-
-This is a known limitation: Docker's default behavior strips many mount-related
-permissions that `--privileged` restores (device cgroup rules, all capabilities,
-full /sys and /proc access).
-
-**Verified:** `--privileged` makes bwrap work:
-```bash
-# Fails:
-docker run --rm --entrypoint bwrap \
-  --security-opt seccomp=unconfined --security-opt apparmor=unconfined \
-  --cap-add SYS_ADMIN mercury-agent:latest \
-  --ro-bind /usr /usr --proc /proc --dev /dev --unshare-pid echo hello
-# → bwrap: Can't mount proc on /newroot/proc: Operation not permitted
-
-# Works:
-docker run --rm --entrypoint bwrap --privileged mercury-agent:latest \
-  --ro-bind /usr /usr --proc /proc --dev /dev --unshare-pid echo hello
-# → execvp echo: No such file or directory  (bwrap ran fine, echo just not in /usr)
-```
-
-## Fix
-
-### Option A: Add `--privileged` flag to bwrap compat (recommended short-term)
-
-In `container-runner.ts` (~line 985), replace the current compat flags with
-`--privileged` when `containerBwrapDockerCompat` is explicitly set to `true`:
-
-```typescript
-    } else if (this.config.containerBwrapDockerCompat || isDockerDesktop()) {
-      logger.info("Enabling bwrap Docker compat (seccomp/apparmor/SYS_ADMIN)", {
-        configFlag: this.config.containerBwrapDockerCompat,
-        dockerDesktop: isDockerDesktop(),
-      });
-      if (this.config.containerBwrapDockerCompat) {
-        // Explicit opt-in: Linux Docker Engine needs --privileged for bwrap
-        // to mount /proc inside a user namespace. The finer-grained flags
-        // (seccomp=unconfined + apparmor=unconfined + SYS_ADMIN) are not
-        // sufficient on standard Linux kernels.
-        args.push("--privileged");
-      } else {
-        // Docker Desktop auto-detection: use the lighter-weight flags
-        // (Docker Desktop's VM has a more permissive kernel config)
-        args.push(
-          "--security-opt",
-          "seccomp=unconfined",
-          "--security-opt",
-          "apparmor=unconfined",
-          "--cap-add",
-          "SYS_ADMIN",
-        );
-      }
-    }
-```
-
-This keeps the lighter flags for Docker Desktop (where they work) and uses
-`--privileged` only when the user explicitly opted in via
-`MERCURY_CONTAINER_BWRAP_DOCKER_COMPAT=true`.
-
-### Option B: Add a separate `MERCURY_CONTAINER_PRIVILEGED` config flag
-
-If you prefer to keep the compat flag behavior unchanged, add a new flag:
-
-In `config.ts`, add:
-```typescript
-containerPrivileged: boolean; // default: false
-```
-
-In `config-file.ts`, add the env mapping:
-```typescript
-containerPrivileged: "MERCURY_CONTAINER_PRIVILEGED",
-```
-
-In `container-runner.ts`, after the existing compat block:
-```typescript
-    if (this.config.containerPrivileged) {
-      args.push("--privileged");
-    }
-```
-
-User sets `MERCURY_CONTAINER_PRIVILEGED=true` in `.env`.
-
-### Option C: Auto-detect and use --privileged as fallback
-
-Run a probe container on startup to test if bwrap works with the lighter
-flags. If it fails, automatically escalate to `--privileged` and log a
-warning. This is the most user-friendly but adds startup latency.
-
-## Workaround (immediate)
-
-Until the fix is in place, users can work around this by manually running
-mercury with Docker's `--privileged` flag. However, since `mercury run`
-doesn't expose extra Docker args, the only current workaround is to
-modify `MERCURY_AGENT_IMAGE` to point to a locally-built image and skip
-bwrap entirely by setting `MERCURY_CONTAINER_RUNTIME=runsc` (requires
-gVisor installed), or to apply the code change.
-
-## Linux server setup checklist
-
-These sysctl settings are needed regardless of the fix above:
-
-```bash
-sysctl -w kernel.unprivileged_userns_clone=1
-sysctl -w kernel.apparmor_restrict_unprivileged_userns=0
-cat > /etc/sysctl.d/99-bubblewrap.conf << 'EOF'
-kernel.unprivileged_userns_clone=1
-kernel.apparmor_restrict_unprivileged_userns=0
-EOF
-```
-
-## How to verify
-
-1. Set `MERCURY_CONTAINER_BWRAP_DOCKER_COMPAT=true` in `.env`
-2. Start mercury: `pm2 restart all` or `mercury service install`
-3. Run: `mercury chat 'hello'`
-4. Check logs — should see successful container run, no bwrap errors
-
----
-
-## Post-Mortem
-
-### Investigation
-Reconciled this hanging bug doc against the current code during `/f-doc-sync` triage. Read the bwrap compat branch in `src/agent/container-runner.ts` (grep for `privileged`/`containerBwrapDockerCompat`/`isDockerDesktop`).
-
-### Root Cause
-On standard Linux Docker Engine, the lighter compat flags (`seccomp=unconfined` + `apparmor=unconfined` + `CAP_SYS_ADMIN`) don't permit `mount(2)` inside a user namespace, so bwrap failed with `Can't mount proc on /newroot/proc: Operation not permitted`. Only `--privileged` restores the permissions bwrap needs.
-
-### Fix
-**Option A was already implemented** — no new code required. In `container-runner.ts` (lines 1003–1008), the compat branch now pushes `--privileged` when `containerBwrapDockerCompat` is explicitly `true` (with a log line "Enabling bwrap Docker compat with --privileged"), while Docker Desktop auto-detection keeps the lighter `seccomp/apparmor/SYS_ADMIN` flags. This matches the recommended fix exactly.
-
-### Lessons
-- Reconcile hanging bug docs against the code before opening a fix session — this P1 was already resolved but never archived.
-- Container-sandbox permissions differ between Docker Desktop (permissive VM kernel) and bare Linux Docker Engine; gate the heavier `--privileged` escalation behind an explicit opt-in rather than applying it everywhere.
-- `docs/bugs/` and `docs/debug/` are gitignored working docs — archival here is a local move only.

package/docs/debug/major/2026-07-02-e2big-prompt-delivery-fix.md

@@ -1,382 +0,0 @@
-# Fix: E2BIG — Argument list too long on bwrap/pi spawn
-
-**Status:** Fixed  
-**Severity:** P0 — causes container crash on long conversations  
-**Affected:** All runtimes (bwrap + gVisor/direct), Docker Desktop especially  
-**File:** `src/agent/container-entry.ts`  
-**Last updated:** 2026-07-02
-
-## Symptom
-
-```
-Container failed (exit code 1): Error: E2BIG: argument list too long, posix_spawn 'bwrap'
-```
-
-Happens when a conversation has enough history that the assembled CLI args + env vars
-exceed Linux's `ARG_MAX` (~2MB). The user sees: *"Something went wrong processing your request."*
-
-## Root Cause
-
-`invokePiOnce()` (line ~741) passes both the **system prompt** and **user prompt**
-(which includes full conversation history XML) as CLI arguments:
-
-```typescript
-// CURRENT — both prompts are argv strings
-const piArgs = [
-  "--print", "--mode", "json", ...sessionArgs,
-  "--provider", provider, "--model", model,
-  ...toolModeArgs, "--no-extensions",
-  "-e", "/app/src/extensions/permission-guard.ts",
-  "-e", "/app/resources/pi-extensions/subagent/index.ts",
-  systemPromptFlag,
-  systemPrompt,        // <-- can be large (system + ext prompt)
-  buildPrompt(payload), // <-- HUGE (conversation history XML)
-];
-```
-
-These args are then passed to `spawn("bwrap", [...bwrapArgs, ...piArgs])` or
-`spawn("pi", piArgs)`. Both go through `posix_spawn`/`execve`, which enforces
-`ARG_MAX` on the combined size of all argv + envp strings.
-
-The non-bwrap (gVisor) path has the **same latent bug** — it just has slightly
-more headroom because bwrap's ~20 setup args aren't in the way.
-
-## Fix: Hybrid stdin + file-path delivery
-
-pi already supports both mechanisms natively — **no pi changes needed**:
-
-1. **`resolvePromptInput()`** in `resource-loader.js:15-29`: if the value of
-   `--system-prompt` / `--append-system-prompt` is a path to an existing file,
-   pi reads the file content. Otherwise it uses the string as-is.
-
-2. **`readPipedStdin()`** in `main.js:40-55`: when `!process.stdin.isTTY`
-   (always true inside Docker with `stdio: ["pipe", ...]`), pi reads stdin
-   and prepends it to the initial message. In `--print` mode (which Mercury
-   uses), this becomes the user prompt.
-
-### Strategy
-
-| Prompt | Delivery | Why |
-|--------|----------|-----|
-| System prompt | Write to file in IO_DIR, pass **file path** as arg | `resolvePromptInput` reads it; path string is tiny |
-| User prompt | Pipe via **stdin** | `readPipedStdin` reads it; zero argv footprint |
-| `MERCURY_EXT_SYSTEM_PROMPT` | Append to system prompt file (already concatenated in code) | Covered by the same file-path mechanism |
-
-This removes both large strings from argv entirely.
-
----
-
-## Implementation Guide
-
-All changes are in **`src/agent/container-entry.ts`** unless noted.
-
-### Step 1: Add `unlinkSync` to fs import
-
-```diff
- import {
-   accessSync,
-   constants,
-   existsSync,
-   readdirSync,
-   readFileSync,
-   renameSync,
-+  unlinkSync,
-   writeFileSync,
- } from "node:fs";
-```
-
-### Step 2: Update `buildBwrapArgs` to accept optional `ioDir`
-
-The system prompt file lives in IO_DIR, which must be visible inside the
-bwrap sandbox. Add it as a **read-only** bind mount.
-
-```diff
--function buildBwrapArgs(workspace: string, command: string[]): string[] {
-+function buildBwrapArgs(workspace: string, command: string[], ioDir?: string): string[] {
-   const args: string[] = [
-     "--ro-bind", "/usr", "/usr",
-     "--symlink", "usr/lib", "/lib",
-     "--symlink", "usr/bin", "/bin",
-     "--symlink", "usr/sbin", "/sbin",
-   ];
-   if (existsSync("/usr/lib64")) {
-     args.push("--symlink", "usr/lib64", "/lib64");
-   }
-   args.push("--ro-bind", "/app", "/app", "--ro-bind", "/etc", "/etc");
-   if (existsSync("/docs")) {
-     args.push("--ro-bind", "/docs", "/docs");
-   }
-+  if (ioDir) {
-+    args.push("--ro-bind", ioDir, ioDir);
-+  }
-   args.push(
-     "--bind", "/spaces", "/spaces",
-```
-
-### Step 3: Rewrite prompt construction in `invokePiOnce`
-
-Replace the current piArgs construction (lines ~720-758) with:
-
-```typescript
-function invokePiOnce(
-  payload: Payload,
-  provider: string,
-  model: string,
-  capabilities: ModelCapabilities,
-): Promise<PiJsonlParseResult> {
-  return new Promise((resolve, reject) => {
-    const overridePiPrompt =
-      process.env.OVERRIDE_PI_SYSTEM_PROMPT === "true" ||
-      process.env.OVERRIDE_PI_SYSTEM_PROMPT === "1";
-
-    // Combine base system prompt with extension-injected fragments
-    let systemPrompt = buildSystemPrompt(
-      capabilities,
-      payload,
-      overridePiPrompt,
-    );
-    const extPrompt = process.env.MERCURY_EXT_SYSTEM_PROMPT;
-    if (extPrompt) {
-      systemPrompt = `${systemPrompt}\n\n${extPrompt}`;
-    }
-
-    // Build the user prompt (conversation history + current message)
-    const userPrompt = buildPrompt(payload);
-
-    const sessionArgs = ["--no-session"];
-
-    const toolModeArgs = capabilities.tools
-      ? ([] as string[])
-      : (["--no-tools", "--no-skills"] as string[]);
-
-    const systemPromptFlag = overridePiPrompt
-      ? "--system-prompt"
-      : "--append-system-prompt";
-
-    // --- E2BIG fix: deliver large prompts via file + stdin ---
-    //
-    // System prompt → temp file in IO_DIR (pi's resolvePromptInput reads files).
-    // User prompt   → stdin pipe (pi's readPipedStdin reads when !isTTY).
-    // This keeps argv small regardless of conversation length.
-    const ioDir = process.env.IO_DIR;
-    let systemPromptArg: string;
-    let systemPromptFile: string | undefined;
-    if (ioDir) {
-      systemPromptFile = path.join(ioDir, "system-prompt.txt");
-      writeFileSync(systemPromptFile, systemPrompt);
-      systemPromptArg = systemPromptFile; // pi reads the file, not the path string
-    } else {
-      systemPromptArg = systemPrompt; // dev fallback: inline (small payloads)
-    }
-
-    // piArgs no longer contains the user prompt — it goes via stdin.
-    // The positional message arg is omitted entirely.
-    const piArgs = [
-      "--print",
-      "--mode",
-      "json",
-      ...sessionArgs,
-      "--provider",
-      provider,
-      "--model",
-      model,
-      ...toolModeArgs,
-      "--no-extensions",
-      "-e",
-      "/app/src/extensions/permission-guard.ts",
-      "-e",
-      "/app/resources/pi-extensions/subagent/index.ts",
-      systemPromptFlag,
-      systemPromptArg,
-      // NOTE: no positional prompt arg — user prompt piped via stdin
-    ];
-```
-
-### Step 4: Update both spawn sites
-
-Replace the spawn block (lines ~776-793):
-
-```typescript
-    const piSpawnedAt = Date.now();
-    let proc: ReturnType<typeof spawn>;
-    if (useBubblewrap) {
-      const bwrapArgs = [
-        ...buildBwrapArgs(payload.spaceWorkspace, ["pi"], ioDir ?? undefined),
-        ...piArgs,
-      ];
-      proc = spawn("bwrap", bwrapArgs, {
-        stdio: ["pipe", "pipe", "pipe"],  // changed: stdin is now "pipe"
-        env: process.env,
-      });
-    } else {
-      proc = spawn("pi", piArgs, {
-        cwd: payload.spaceWorkspace,
-        stdio: ["pipe", "pipe", "pipe"],  // changed: stdin is now "pipe"
-        env: process.env,
-      });
-    }
-
-    // Pipe user prompt via stdin — avoids E2BIG on long conversations.
-    // pi's readPipedStdin() reads this when !process.stdin.isTTY (always true in Docker).
-    proc.stdin!.on("error", () => {}); // swallow EPIPE if pi exits early
-    proc.stdin!.end(userPrompt, "utf8");
-```
-
-### Step 5: Clean up system prompt file on close
-
-In the `proc.on("close", ...)` handler (line ~815), add cleanup at the top:
-
-```diff
-     proc.on("close", (code) => {
-+      // Clean up temp system prompt file (belt-and-suspenders; IO_DIR is ephemeral)
-+      if (systemPromptFile) {
-+        try { unlinkSync(systemPromptFile); } catch {}
-+      }
-       logTiming("container.pi.done", {
-         piDurationMs: Date.now() - piSpawnedAt,
-         exitCode: code ?? null,
-       });
-```
-
-Also add the same cleanup in the `proc.on("error", ...)` handler:
-
-```diff
-     proc.on("error", (error) => {
-+      if (systemPromptFile) {
-+        try { unlinkSync(systemPromptFile); } catch {}
-+      }
-       reject(error);
-     });
-```
-
----
-
-## P0 companion: Byte-budget guard on history (defense-in-depth)
-
-Even with the delivery fix, it's good practice to cap history size. This also
-reduces token waste on very long conversations.
-
-In `buildHistoryXml` (line ~533), replace the unbounded entries construction:
-
-```typescript
-const HISTORY_CHAR_BUDGET = 400_000; // ~100K tokens — well under model limits
-
-function buildHistoryXml(messages: StoredMessage[]): string | null {
-  // Pair up user+assistant turns (existing logic unchanged)
-  const turns: Array<{ user: StoredMessage; assistant?: StoredMessage }> = [];
-  let pendingUser: StoredMessage | null = null;
-
-  for (const m of messages) {
-    if (m.role === "user") {
-      if (pendingUser) {
-        turns.push({ user: pendingUser });
-      }
-      pendingUser = m;
-    } else if (m.role === "assistant" && pendingUser) {
-      turns.push({ user: pendingUser, assistant: m });
-      pendingUser = null;
-    }
-  }
-  if (pendingUser) turns.push({ user: pendingUser });
-
-  if (turns.length === 0) return null;
-
-  // Build newest-first, stop when budget exhausted
-  const entries: string[] = [];
-  let usedChars = 0;
-
-  for (let i = turns.length - 1; i >= 0; i--) {
-    const { user, assistant } = turns[i];
-    const ts = formatContextTimestamp(user.createdAt);
-    const userLine = `    <user>${escapeXmlText(user.content)}</user>`;
-    const assistantLine = assistant
-      ? `\n    <assistant>${escapeXmlText(assistant.content)}</assistant>`
-      : "";
-    const entry = `  <turn timestamp="${ts}">\n${userLine}${assistantLine}\n  </turn>`;
-
-    if (usedChars + entry.length > HISTORY_CHAR_BUDGET) break;
-    usedChars += entry.length;
-    entries.unshift(entry); // maintain chronological order
-  }
-
-  if (entries.length === 0) return null;
-  return `<history>\n${entries.join("\n")}\n</history>`;
-}
-```
-
----
-
-## How to verify
-
-### 1. Unit test: large payload doesn't crash
-
-Create a payload with >128KB of conversation history. Verify `invokePiOnce`
-spawns successfully (no E2BIG). The container needs Docker running.
-
-### 2. Manual test: long WhatsApp conversation
-
-Find (or create) a space with 50+ messages. Send a new message. Verify the
-agent responds instead of crashing with "Something went wrong."
-
-### 3. Regression test: short conversations still work
-
-Normal short conversations should be completely unaffected. The prompts reach
-pi with identical content — only the delivery channel changed.
-
-### 4. Verify stdin content reaches pi correctly
-
-Add a temporary `logTiming("prompt.sizes", { systemPromptLen: systemPrompt.length, userPromptLen: userPrompt.length })` before spawn. Compare with pi's
-output to confirm the full prompt arrived.
-
----
-
-## Edge cases to watch
-
-| Case | Behavior |
-|------|----------|
-| IO_DIR unset (local dev) | System prompt stays inline in argv; user prompt still goes via stdin. Safe for small dev payloads. |
-| pi exits before reading stdin | `proc.stdin.on("error", () => {})` swallows EPIPE. pi's exit code/stderr still captured normally. |
-| System prompt file path collides with real text | Only triggers if the system prompt string happens to be a valid file path. Extremely unlikely for a multi-KB prompt. If paranoid, use a path with a UUID. |
-| bwrap + IO_DIR mount | IO_DIR is `--ro-bind` (read-only). pi can read the system prompt file but cannot write to IO_DIR. Correct for security. |
-| Retry logic in `runModelChain` | Each retry calls `invokePiOnce` fresh — each writes a new system prompt file and pipes stdin. Previous file is cleaned up on close. No stale state between retries. |
-| `MERCURY_EXT_SYSTEM_PROMPT` | Already concatenated into `systemPrompt` before the file write. Covered automatically. |
-
----
-
-## Summary of changes
-
-| What | Where | Lines affected |
-|------|-------|----------------|
-| Add `unlinkSync` import | fs import block | 1 line |
-| `buildBwrapArgs` accepts `ioDir` | Function signature + body | ~4 lines |
-| System prompt → file in IO_DIR | `invokePiOnce`, before piArgs | ~8 lines |
-| Remove user prompt from piArgs | `invokePiOnce`, piArgs array | Remove 1 line |
-| stdin: "ignore" → "pipe" | Both spawn sites | 2 lines |
-| Pipe user prompt to stdin | After spawn | 2 lines |
-| Cleanup system prompt file | close + error handlers | 4 lines |
-| **Total** | | **~22 lines changed** |
-| Byte-budget history (optional companion) | `buildHistoryXml` | Replace function body |
-
----
-
-## Post-Mortem
-
-### Investigation
-Verified the current state of `src/agent/container-entry.ts` against this fix guide before doing any work (bug was flagged as a hanging doc during `/f-doc-sync`, never archived). Grepped and read the relevant functions: `buildBwrapArgs` (line 676), `invokePiOnce` (line 728), `buildHistoryXml` (line 534), and both spawn sites (line 810).
-
-### Root Cause
-`invokePiOnce()` passed the system prompt and the full conversation-history XML as `argv` strings to `posix_spawn('bwrap'|'pi', …)`. On long conversations the combined argv + envp size exceeded Linux `ARG_MAX` (~2MB), so `execve` failed with `E2BIG` and the container exited 1.
-
-### Fix
-The complete fix described in this guide was **already implemented** in the tree — no new code was required:
-- `unlinkSync` imported and used in the `close`/`error` handlers (lines 854, 865).
-- `buildBwrapArgs(workspace, command, ioDir?)` binds `IO_DIR` read-only into the sandbox (line 679 signature; line 716 `--ro-bind ioDir ioDir`).
-- System prompt written to `IO_DIR/system-prompt.txt` and passed as a file path (pi's `resolvePromptInput` reads it) — lines 765–771.
-- User prompt removed from `argv` and piped via stdin (`proc.stdin?.end(userPrompt, "utf8")`) with both spawn sites using `stdio: ["pipe","pipe","pipe"]` — lines 814–831.
-- Temp system-prompt file cleaned up on close and error — lines 852–865.
-- Optional companion `HISTORY_CHAR_BUDGET = 400_000` applied in `buildHistoryXml` (lines 534, 571) as defense-in-depth.
-
-### Lessons
-- Hanging bug docs should be reconciled against the code before opening a fix session — this one was fully resolved but never archived, so it lingered as an apparent P0.
-- Never pass unbounded, caller-influenced data (conversation history) through `argv`; use stdin or a file whose path is the only thing in `argv`. Pair the delivery fix with a size budget so history growth can't regress the limit.
-- `docs/bugs/` and `docs/debug/` are gitignored working docs — archival here is a local move only, with no commit/push.